Unwelcome website visitors - Journal of Omnifarious
Jul. 24th, 2004
05:14 pm - Unwelcome website visitors
I was checking over log files for my website, and I noticed that my two biggest visitors were companies that spider the web looking for trademark violations. One followed all the rules. It read robots.txt and it wasn't overly aggressive about fetching pages. And the User-Agent string for the bot said it was a bot.
The other was from Cyveillance. It was very rude. It gave a User-Agent string that claimed it was a real web browser, and it fetched pages at a rate of 12/minute. And it never fetched the robots.txt file, so I must assume that it completely ignored it.
I sent a message to the second company telling them that I would out them and their practices to the world while also filing an abuse report with their ISP if they ever did that to my server again.