WordPress “Brute Force” Attack and FatCow’s Response

By moosnews
April 12, 2013

wordpress password As you may know, Tuesday, a widespread “brute force” attack against WordPress sites started impacting sites across the internet. This attack is leveraging a botnet which looks to have more than one hundred thousand different computers at its disposal. Its intent is very simple: to find and compromise WordPress sites with simple passwords, to likely later use them to distribute malware (and further increase the size of the botnet).

On Tuesday, our admins discovered this attack as we investigated increased load and decreased performance on our hosting servers. We quickly identified this as a widespread attack on the WordPress login page. The attack was a large one (hundreds of hits a second to many WordPress sites spread across our infrastructure). It became quickly obvious we needed to act fast. At this point, the fastest solution was to drop all traffic to the WordPress login page (wp-login.php) while we worked on a better plan.

The downside to this, of course, is that we blocked legitimate access for customers who wanted to login to WordPress. We knew that was not an acceptable solution for very long, so we immediately went to work on a better solution. We truly apologize if we kept you from logging into your WordPress, but we felt that keeping your site up (but not allowing you to login), was the better option.

With the infrastructure stabilized, we dug in and started investigating better solutions. We reached out to some partners and other groups on the web, and collaborated on some security rules that would help mitigate the attack. These security rules are, in a sense, rules based on behavior: if a single IP address or browser used the wrong password on a WordPress site more than a handful of times in a few minutes, we would ban that IP address for a period of time. This rule would help us allow legitimate customers to login to WordPress, but would stop the attacker after a number of bad attempts.

We rolled these changes out Tuesday afternoon. It took a few tries to find the right balance to block the bad guy but not keep a legitimate user from logging into their WordPress site. The attack subsided overnight.

The attack returned in force on Wednesday as we reached peak business hours. This made it obvious that the attack was based off a botnet—likely using the computers of unsuspecting office workers coming in for a normal day of work! We spent Wednesday tweaking rules and working with other folks in the industry to share tips, tricks, and findings.

By this point, between ourselves and our partners, we were approaching having flagged nearly that hundred thousand IP addresses, and more new IP addresses were showing up every second. Even though we were stopping much of the attack, it was so large that simply handling the traffic was starting to impact our servers.

The team was able to keep things stable for most of Wednesday, working hard to tweak rules as we or our colleagues identified new trends.

By Thursday, it was clear that the attack was not subsiding. The first thing we did was to roll out a new heuristic-based set of rules, that would look historically at our growing set of log data, identify patterns, and block the attack based on that data, not just on current bad behavior, but combinations of bad behavior.

That put a big dent into the attack. But the attack was still big enough to be causing our servers to run at a higher than normal load.

Our breakthrough happened on Thursday, as our team looked through data on the web and data in our logs. We found a difference between the way the attack accesses WordPress and legitimate customers access WordPress. Thursday afternoon, we rolled that change out to our edge servers (before the traffic even reaches the web server that might be hosting your site) to drop any traffic that didn’t look legitimate.

Hundreds of hits a second dropped to nearly none.

We’ve been rolling this change out across our data centers and seeing much of the attack mitigated. This is allowing us to focus less on just keeping things running and more on the proactive work of heading off the next variant of this attack. The attack, as it usually does, has started to pick up again today during peak business hours, but thus far, we’re not feeling the effects.

We head into the weekend in good shape, but vigilant against a returning or altered attack. In the meantime, our support team is ready to help you if you are feeling any lingering effects (the most common one might be if your IP got marked as a possibly bad IP). If you’d like to help make your site stronger, we recommend changing your WordPress password to a secure one, if you haven’t already.

This entry was posted on Friday, April 12th, 2013 at 2:34 pm and is filed under Announcements. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.

21 Responses to “WordPress “Brute Force” Attack and FatCow’s Response”

  1. Paul Bourret Says:

    Thank you for the update. I changed my password shortly after receiving the tweet Fatcow sent about the attack. I appreciate the notification about the attack and the details about what was happening.

    Best regards,

    Paul Bourret

  2. john Says:

    Well done guys!

  3. Dave Says:

    Good work suppressing the attack. Hopefully that will be it for the time being, but I appreciate the hours you’ve been putting in on this.

  4. Enrico Says:

    A simple step any WordPress admin can take is to instal a free plugin called “Stealth Login Page” which will change the URL of the login page.
    This will basically prevent most of the automated login attacks.
    A strong password and limiting the number of failed attempts is also a good thing to have.

    Regards.

  5. fatcowuserme Says:

    Way to go fatcow! I’ve had a wordpress site hacked once, not fun. I’ve learned my lesson and a whole lot about making wordpress more secure thanks to it. carry on.

  6. Jeanne Says:

    so, how long is an IP blocked if you used the wrong password? I’d not logged in in a long time and couldn’t remember mine, now I cannot open the log in page to reset my password.

  7. Andrew Newey Says:

    Good response FatCow and good prompt for me to tighten up all my passwords. Thanks :)

  8. Patricia Says:

    This is where I get extremely annoyed. Check your logs. I contacted support more than once this week with some issues regarding my account and never once was it mentioned that this could be the issue. In the last call, I just hung up on the guy because it wasn’t until I’d gone on and on that he casually mentioned something had happened “about 20 minutes ago”. You can listen to the recording. Had I known that, I could have ascertained that it was likely the hiccup I’d had with my problem and that would have been that. You guys are comical. Not at all the FactCow of 6 or 7 years ago. Not by a longshot.

  9. Bridgette Says:

    Thanks for explaining what was going on… I changed my password and am trying to upload a new template, but keep getting an error message. Can you help with this?
    Thanks!

  10. Kushal Azza Says:

    Thanks for update and mail. I will consider changing password now. And also please make your server uptime as high as possible, I am getting website down alerts most frequently.

  11. Colleen Says:

    Last weekend I (coincidentally) removed my WordPress blog, before this happened. However, I do have a standalone bbpress v1.2 forum. Is that affected? I will change my passwords regardless.

  12. Aaron Rigby Says:

    If no one else has said so, or even if they have… While there are many choices in the web-world for hosting, I will always suggest fatcow.com to my customers. Prices aside. Fatcow.com and their engineers go well beyond any typical/normal level of wanting to help their customers. They will work through everything with you, until it meets the level that you want. And even then, they’ll still strive for more. :)
    Kudos guys!

  13. Larry Anuta Says:

    My wordpress site seems to still be intact, but I can no longer log in.
    It does not recognize my e-mail address, user name or password.

    Larry…

  14. Larry Anuta Says:

    I wordpress site is still there, but I cannot log in to it.

    It does not recognize my user name, password. it will accept my e-mail address “microkey@sbcglobal.net” and the username “microkey”. But these are for http://www.larobot.com. I had a separate username and password for WordPress. tried to reset my password and I get a reply e-mail, but it says that the link you send me is not valid.

    I have tried it twice…

    http://larobot.com/liberty/wp-login.php?action=rp&key=7q9Pfjq8ChSxAYvLSKsl&login=microkey
    http://larobot.com/liberty/wp-login.php?action=rp&key=Pl0OXWJ4KZj5JKcT4gP0&login=microkey

  15. Larry Anuta Says:

    My password had been reset and I have a new password and I can now log in…
    Thanks…
    Larry…

  16. Justin Orsborn Says:

    Thanks Fat Cow for such an effective and proactive response. I am not real knowledgable on these matters so wondering why the attack and what would they hope to achieve

  17. Jesse Joyner Says:

    So you found out what happened and have curbed the attack, any idea what the botnet was doing to wordpress sites and can we check to see if ours was affected. There was no mention of what the attack was designed for and the damage it may cause except a slowdown of the servers. Should we be worried that some code has been inserted into our sites that will be unleashed at a later time.

    Please continue to be vigilent on our behalf and let us know, if you can. I understand as an IT professional that sometimes you keep certain information from your customers out of a need to not reveal

  18. Jesse Joyner Says:

    everything that is going on, but you still need to inform your customer that something is going on. Please keep us informed

  19. inka Says:

    000 webhost is under DDOS attach..i think its better to go for this solution..thank you…i was so scared that i lost my wordpress on 00OWEBHOST…

  20. sean Says:

    while we NOW appreciate the information and all of your efforts during this issue, we feel very strongly that fatcows could have handled the customer support portion of this MUCH, MUCH better. even this news post is 3 DAYS AFTER the initial event occurred.. we had created a trouble ticket, emailed AND phoned fatcows support over the ENTIRE 3 DAY period only to receive NO REPLIES on the trouble ticket, NO REPLIES to our emails and NO PHONE CALL-BACKS (as promised on more than one occasion).. when we were able to get a support staff on the line, we weren’t even told a system wide issue existed, only that someone from a higher tier support level would contact us, which never happened.. being left in the dark for what we feel is a very, very long amount of time (over 3 days) was just simply unacceptable behavior from fatcows. while we appreciate the time you’ve put into fixing this issue, we’re hoping fatcows rises to become a MUCH better hosting proivider, addressing its customers MUCH sooner when an issue like this does arise and, more importantly, preventing these types of attacks from even being an issue.

  21. Matthew Jackson Says:

    The hosting company acted quickly to put new security rules in place that would help them allow legitimate customers to login to WordPress, but would stop the attacker after a number of bad attempts. During some periods of the week the site itself went down.

Leave a Reply