Surviving an amplified Wordpress pingback attack

On August 31, 2016 as I began turning off lights and heading to watch Mr. Robot, I received an interesting email.

High 5 minute load average alert - 20.82

The email additionally showed that my traffic was extremely high.

1 Min Load Avg:          77.74
5 Min Load Avg:          20.82
15 Min Load Avg:         7.87

This traffic is not normal for my little Linode (8GB plan), so I logged into the system to see what was happening.

This was not normal and not even a Reddit/Hacker News spike of traffic would cause this. Services were beginning to be killed and my SSH connection was so slow to diagnose anything.

After finally making it to my access logs, I found the following lines

54.84.166.245 - - [01/Sep/2016:02:31:51 +0000] "GET / HTTP/1.0" 301 - "-" "WordPress/4.0; http://blog.roundtown.com; verifying pingback from 185.81.158.233"
52.24.240.21 - - [01/Sep/2016:02:31:51 +0000] "GET / HTTP/1.0" 301 - "-" "WordPress/4.1.12; http://52.24.240.21; verifying pingback from 185.81.158.233"
52.24.44.55 - - [01/Sep/2016:02:31:51 +0000] "GET / HTTP/1.0" 301 - "-" "WordPress/4.3.5; http://poymi.me/nya; verifying pingback from 185.81.158.233"
23.238.21.90 - - [01/Sep/2016:02:31:51 +0000] "GET / HTTP/1.0" 301 - "-" "WordPress/4.1.12; http://carbonchat.com; verifying pingback from 185.81.158.233"

There were hundreds, if not thousands, of these similar lines all targeted towards one domain that I host. At this point, MySQL had been killed so all other sites on the box were either timing out from a lack of apache workers or dying from lack of a MySQL connection.

Pingdom alerts were flying in. Each and every site I host began reporting downtime, but at least I understood the attack was targeted to one domain.

My first course of action was to suspend the site being hit. In order to do this, I fed the public directory for apache to a template for suspended domains.

Now all these requests were returning a small index.html file vs the previous which was a bulky PHP forum software. Requests had no sign of stopping, but load wasn't through the roof anymore.

I started researching the "verifying pingback from" string in order to discover what this type of attack was. It didn't take long till I found this post from Sucuri and what an interesting attack this was.

To summarize the attack Wordpress has a "pingback" feature which allows blogs to say, "Hey I'm linking to X post" so the original post can acknowledge another site linking to it. In theory it seems like a good idea, but what happens is that malicious authors can collect as many Wordpress installs as they find and send a fake request that appears to be from any source.

Those Wordpress installs then reach out to that source to verify the pingback. The domain being attacked was not Wordpress so these GET requests were just going to /.

I can't write iptable rules from my head so a temporary bandage was to block all the offending IPs, so I began isolating attack traffic thanks to a tip from John Cuppi.

cat access-log | grep "verifying pingback from" > wp_attack.log

This left me with a file of all attack traffic. The next step was to get the IPs from those lines. A quick PHP script helped achieved this.

<?php
error_reporting(0);

$reqs = file("pingback_attack.log");
$sum = 0;
$ip_address = [];

foreach ($reqs as $req) {
	$ip = explode(" - - ", $req);
	$ip_address[$ip[0]]++;
	$sum++;
}

arsort($ip_address);

foreach ($ip_address as $ip => $attack_times) {
	print "Wordpress IP (" . $ip . ") attacked {$attack_times} times \n";
	system("iptables -A INPUT  -s {$ip} -j DROP");
}
echo "Total Attacks: " . number_format($sum) . "\n";

This dropped all traffic that participated in this attack and the numbers shown were quite alarming.

Wordpress IP (195.154.1x.x) attacked 17,267 times
Wordpress IP (37.187.1x.x) attacked 16,865 times
Wordpress IP (37.187.7x.x) attacked 15,903 times
Wordpress IP (178.79.13x.x) attacked 15,022 times
Wordpress IP (212.129.1x.x) attacked 14,424 times
Wordpress IP (91.146.10x.x) attacked 13,339 times
Wordpress IP (37.48.2x.x) attacked 12,753 times
Wordpress IP (148.251.9x.x) attacked 11,888 times
Wordpress IP (213.171.20x.x) attacked 10,478 times
Wordpress IP (149.202.19x.x) attacked 10,180 times
...
Total Attacks: 454,951

Immediately load dropped and IO returned to normal levels. My ghost blog (this one) and my Gitlab instance came back up and Pingdom began alerting myself to the end of downtime.

It was down for 27 minutes and 21 seconds.

For a single server that hosts nothing mission critical this was an okay response time. Luckily I hadn't fallen asleep yet or this may have been far worst.

So now it was time to dig into the graphs and see what traffic was heading my way.

As you can see, this attack is substantially higher than any traffic I normally obtain. The attack peeked at 479 requests a second with 179 unique IPs totaling 454,951 requests in roughly 28 minutes.

The blog post mentioned above had 162,000 different IPs in their attack. I had 179. That is roughly 0.11% the attack than Sucuri discussed here, but was enough to cripple my single server for a bit.

Now that it was the next morning I found an additional resource that put a block at the iptable level by reading the packet for a WordPress user agent and "verifying pingback" string. I unfortunately did not stumble upon this resource until after the fact.

As for the website that was targeted, it rarely passes 100 unique hits a day. I'm not sure if there was a purpose to this attack, but the application attacked was definitely the most resource intensive I host. It's almost like the attack was to one of the other domains I host, but the attack just looked for the most expensive application on the same server to attack.

I missed Mr Robot thanks to this attack, but learned a bit more about iptables.