Scum of the Web

One morning I woke up with a scary email in my inbox, it was from Linode (my host) with a Terms of Service violation due to spam. The countdown till my server would go offline was 4 hours. This post is that journey.

The rest of email included the base64 payload, tips on recovering from a hack and the decoded base64. This is a server that has been around since 2011 and hosts a lot of sites so my heart was racing to resolve this problem.

I alerted my office immediately that I would be late, even though it was only 6:30am in the morning - I did not know how long this would take.

Step 1 - Check for complete takeover.

My server is on a non-standard SSH port and only allows login via key based authentication. So I didn't think it would be possible for a complete shell takeover, so I signed in and checked w and no one else was signed in. That was good news, so I then pivoted to check the ssh logs.

Opening that file last login was mine and everything I could see in my visible screen height was also me dating back to February 3rd. So that was good, this attack must have been something else.

Step 2 - Identify the client.

Since the server itself looked okay, it had to be one of the clients on the machine. Since it was a cPanel box, it was pretty segregated with 20-30 different sites. Decoding the base64 of the spam email contained links with a domain that I recognized as a domain I host.

So now I was nervous.

Step 3 - Stop the bleeding.

cPanel has a neat feature known as suspension, which basically just redirects the public directory to a generic "suspended" page and a few other logical things you'd expect when suspending an account.

Once that was done, I knew anything malicious was no longer viewable. So it was time to investigate.

Step 4 - Identify the entry-point.

This was a Wordpress blog, so maybe it was out of date or outdated plugin with an automated exploit. I pivoted myself to the user's home directory and ran a command to dump all files edited in past 24 hours.

find . -mtime -1 -ls > changed_last_24hrs

A good tip - When investigating a potential hack. The less you do the better to not affect logs/modification times/etc, so all my commands will output to a file so I can maintain that state.

It dropped a bunch of files, but a few caught my eye.

• .well-known/acme-challenge
• ./ssl/keys
• ./ssl/certs
• ./public_html/amx
• ./public_html/amx/Online

With this investigation though, nothing in WordPress was touched. So could WordPress have been the entry point with no files modified? I'm not sure, but to be safe I database edited the only two users in that system to random characters for the password hash.

Now my focus was back at the modified SSL files. I took a trip to my favorite "crt.sh" domain and looked up the affected website. Sure enough a new SSL certificate was created for "*.domain.com", so someone generated a wildcard certificate.

This makes sense, because the domain was "american.express.####.domain.com" in the malicious spam email.

Immediately I connected the dots. WordPress cannot generate SSL certificates so the only thing capable of doing that is AutoSSL on my cPanel machine. This must have been a password leak.

I changed the password, forcing a password reset to obtain access. Before that, I confirmed no new users were created and the email attached to the main user remained the same.

Step 5 - Cleanup

I went through and deleted all the affected files:

• The bogus PHPMailer (for spamming) and phishing website
• The wildcard certificate after instructing LetsEncrypt I wanted to revoke it
• The entire "amx" directory

Additionally, I reinstalled WordPress deleting all files outside from uploads and themes. I double checked the remaining files were not modified in past 24 hours.

Step 6 - The Post Investigation

With the affected website clean and back online (12-18 hours later), the warning notices on Google had already gone away. Which just blows my mind how quickly Google/Firefox notice a phishing website, flag it, block it and then remove the warning once removed.

Now that I understood the entry, I went peeking in the logs and found all the actions that occurred, including the IP from where it occurred from. It was Russia, which was probably just some scummy automated script or some deranged individual set on illegal actions.

This page above blew my mind, since it claims nearly 40-50k websites are marked DAILY for phishing or malware. My guess is hosts are infected or hacked, much like this one. The legitimate history of them allows them to be used for abuse until some automated system catches on.

My logs show my server emitted 4 emails that were phishing attempts against American Express, so I guess it could have been worse.

The grammar in the phished email was so bad. Here are some snippets.

• For your protection, online banking has been lockn exceeded the number allowed.
• We are sorry to inform you that your online account has been temporarily locked after too many unsuccessful log
• Will review and vd take necessary steps to protect your account from fraud.

Any sane person could easily spot that phishing attempt a mile away. It was a bit of a wake-up call that I need to take security on a shared machine a bit more seriously.

I went overkill and forced password changes for all my hosted clients and strongly requested 2FA to be enabled. I could not find a way to enforce it at this time.

To end this post, I guess after 10 years of a server - this is probably bound to happen. Don't re-use passwords. Leverage 2FA (not SMS based). Stay safe.

The timeline (UTC):

• 2020-02-14 10:15:16 - Attacker login.
• 2020-02-14 10:16:35 - First phished Email sent.
• 2020-02-14 11:26:00 - Abuse Email from Linode detecting it.
• 2020-02-14 11:33:00 - I begin investigation.
• 2020-02-14 11:40:00 - Site Suspended.
• 2020-02-14 14:10:00 - Cleanup complete.
• 2020-02-15 00:27:00 - Site enabled. (Owner setup 2FA)
• 2020-02-16 00:45:00 - Blog post written.