Ramblings of a Tampa engineer
Damage of a foreign sitemap with 500k entries.

On January 15, 2023 I deleted an old Linode as I helped migrate an old website I hosted to a new provider and webmaster. This went as seamless as possible with zero downtime - I asked all WordPress contributors to hold on writing - did an rsync of all contents and one quick export/import of database and I was done.

The short TTL I had on the domain led folks to the new host extremely quickly and for those contributors who didn't respect TTL - I had a draft blog in the old site titled - "This is old - do not write here.". I kept the old site up for about a week for stragglers then deleted the Linode as mentioned on January 15.

Five days later on January 20, 2023 I obtained an alert that a new owner was verified in the Google Search Console. I was pretty busy at work on that Friday, so I only started piecing together this mistake on the weekend.

So this is the story.


The Linode Deletion

Logs in Linode of actions.

So maybe not exactly a week as mentioned, but on January 10th I turned that Linode off and on January 15, 2023 I deleted it. This action immediately freed that IP address (96.126.126.141) and unaware to me started the cascade of mistakes.

There must be a bot/spider/site/tool responsible for finding recently released IP addresses as this behavior occurred within days. So unbeknownst to me at the time I was now broadcasting an A record to an IP I no longer controlled.

➜  ~ dig fevgames.connortumbleson.com

; <<>> DiG 9.10.3-P4-Ubuntu <<>> fevgames.connortumbleson.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15495
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;fevgames.connortumbleson.com.	IN	A

;; ANSWER SECTION:
fevgames.connortumbleson.com. 14145 IN	A	96.126.126.141

;; Query time: 9 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Sun Jan 22 07:19:47 EST 2023
;; MSG SIZE  rcvd: 73
"fevgames.connortumbleson.com" -> "96.126.126.141"

This was obviously the vector and a big reminder to always take it seriously when you are releasing an IP back into the global space that you previously used.


The Verification

So at this point with a valid subdomain pointed to an IP I don't control - somehow this attacker obtained access to the IP that I recently released in a short period of time. There must be some scanning service that watches when IP addresses move around. BishopFox has a tool for this, so there must be a few more of these tools available for attackers to use.

Some seem to work isolated to AWS spinning up IPs as fast as they can - scanning and releasing when done. Something like that seems like something AWS would like to prevent, but I digress. Perhaps Linode has the same sort of mechanism for iterating through IPs in the pool.

So this attacker now uploads the basic HTML verification file to some hosting in order to have Google verify ownership of that "fevgames.connortumbleson.com" website and that works. The attacker is now a verified owner of my sub-domain.


The Missed Alarm

The giveaway email.

So while at work on 11am on a Friday - I'm busy with some meetings prior to lunch and spot an email about a new owner for "fevgames.connortumbleson.com".

Since I had just recently migrated the real website, I just improperly assumed that the testing domain was migrated/re-owned by the new owners. I failed to open and fully read the email to see the clearly bogus email attached to it.

Clearly I wasn't thinking if I thought it was all right for our testing site to be index/added to Google Search Console. I guess you lose track of things when you are passing it along to a new person.


The Luck of the Draw

With the weekend here now (January 21, 2023) I got another email about my Google Search Console performance suffering because I did not put an image placeholder on my recent blog (Callous Tampa) that utilized a video.

So with a bit more free time I jumped onto the console and reviewed that warning to fix it. When I went to re-process that warning I noticed something strange on the results where I saw 500,000+ pages that were declined from indexing.

Indexing results on Google Search Console

I know I blog weekly, but surely not 500,000 pages worth. So I clicked in and saw something alarming. It was pages on pages of clear redirect/ad spam links that simply responded to all these popular search terms.

Discovered - currently not indexed on Google Search Console

So now I pieced it together with the email I got the day prior about a new owner on the domain. I went back and saw the email was dmcareportmail@gmail.com which is a clever naming trick to hopefully cause site owners to ignore the addition of the owner.

In my case - a mix of a busy work day and bad assumption led to ignoring the initial sign.


The Attacker Removal

So right out of the gate - I realized that the testing domain for the recently migrated FevGames was no longer needed. I deleted those relevant DNS records and did some cleanup on that connortumbleson.com zone.

I figure that would break the HTML page verification, but I needed to work quicker and explicitly remove this person.

Thankfully the email had two options.

  • The first being "Manage Users" in Google Search Console which was useless to me because it showed no one, but me.
  • The second being the legacy "Google Webmaster Central" which had the malicious user and how they verified, but was incredibly difficult to visit.
Google Webmaster Central showing malicious added owner.

So from that page I could remove the malicious user and identify how they verified - then I went onward to cleaning up the mess.


The Cleanup

Asking Google to removal all of these domains.

So I went back to Google Search Console and requested Google to remove all these malicious links. This didn't take too long to submit, but I'm not sure how long it takes to enact itself.

I sadly still see results on Google when I explicitly look for that subdomain. Granted, it hasn't even been 24 hours so we will give this a few more days to reset itself.

Back on the DNS front - I double checked I had no more references to the no longer owned IP.


So all in all - I learned first hand to always be careful when releasing IP addresses and hopefully didn't damage my blog too much with that mistake.

I'll monitor every day for a bit and make sure everything is returning back to normal on the Google Search Console front.

You’ve successfully subscribed to Connor Tumbleson
Welcome back! You’ve successfully signed in.
Great! You’ve successfully signed up.
Success! Your email is updated.
Your link has expired
Success! Check your email for magic link to sign-in.