This morning, we received an email stating that our estimated bandwidth usage over the past 10 minutes was 35Mbit per second. This would be an ok thing if our average usage wasn't closer to 9Mbit per second. We're a small shop, and where we're located, bandwidth is expensive, so when I saw the email come through, I started scouring the logs.
We use (among other things) Munin to track what's going on with our machines. I confirmed that the traffic was "legit", in that it came out of our proxy servers, which only serve HTTP traffic. So I started fumbling through the logs. I noticed that the spike started at about 10:10am EST (1410 UTC), and ran for about 10 minutes. Grepping the logs revealed about 12k page loads to a single URL. Again, this would be fantastic, but is very unrealistic for us. That's when I noticed that all of the useragent's were the same. And old. Very old.
Every one of the page loads had
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:126.96.36.199) Gecko/2009042316 Firefox/3.0.10 (.NET CLR 3.5.30729) as their useragent. Firefox 3.0.10 was released April 27, 2009. This sent the red flags up for me. When I noticed that there were over 9k unique IPs, I nearly lost it. My first thought was a botnet test against our machine(s). Though, what our sites have done to garner a botnet attack is beyond me.
I've been looking at information about the IPs for several hours now. Using a script that I modified that looks at ipinfodb.com's API, I can found out the City and Country of the offending IPs. This allows me to filter out non-US IPs, as I've found that most ISPs outside the US have very lackluster responses to "A machine on your network may be compromised." Once I get all of the US entries, I can attempt to lump them together by ISP, and if there's enough from any one ISP, I'll be able to send them something.
My most interesting find so far? An IP with whois info pointing to Cupertino. Yes, that Cupertino.