Back in 2005 we launched a directory of IT security vendors as part of IT-Harvest. Three people in Salt Lake City worked 4 months to compile and categorize 1,200 vendors, their products, and their executives.
On the day of the launch an engineer at a Canadian vendor wrote a script and systematically began sucking down our entire database. We scrambled to counter the script by changing how the search field worked. Finally I just called the CEO and told him to rein in his guy.
Today web bots are a big business. Any organizations that has invested heavily in content creation, compiling a database, or presenting any proprietary data, has probably had to face this scourge of people trying to profit off of their hard work. Another example of web bot malfeasance is click fraud whereby an affiliate of an ad network clicks a lot on their ads to generate revenue.
A recent report released by the newly funded Distil Networks finds that bad bots nearly doubled in their percentage of all web traffic between the periods of Q1 2013 and Q4 2013, jumping from 12.25% to 23.6%. You can get a great feel for the types of bad bots out there by perusing Distil Networks’ directory here.
Screen scraping, and other automated web activity is just another example of how where there is a way to profit, someone will figure out how to automate it.
While most web application firewall vendors attempt to add an anti-bot feature to their WAF, Distil Networks is taking a best of breed approach that likely will appeal to online retailers, content sites, and directory services. By quickly fingerprinting the sources of attacks they propagate defenses to all of their 100 plus customers, which include Fortune 500 companies.