NGINX, Ghost Blog and top of Hacker News

On September 27, 2022 I'm sitting at the office helping prepare for the arrival of Hurricane Ian and I stumble upon a thread on Hacker News with people discussing impersonation of others. I think "maybe they will like my blog I wrote last week", so I try and share my link and instead after submitting am met with a self-imposed error screen.

I'm in Hacker News time out, so I go to lunch.

While eating I get an email that my blog is offline. I tried to view the blog on my phone and it indeed was not responding, so my lunch ended early and I jumped onto my computer.

SSH worked, but very slowly and I saw the load on the server being 7.86 on a single Linode Nanode so something was eating up resources. It appeared node and Ghost was consuming so many resources that things were timing out.

I couldn't understand why it was climbing so much, so I tailed my NGINX access logs and saw a huge amount of hits coming to my "Someone is pretending to be me" post. At this point I connected the dots - I jumped to Hacker News and my post was sitting at #1 already and I had just posted it like 30 minutes ago. I figured my submission was blocked so didn't think that as a possibility.

So I made a quick rash decision to upgrade my Linode to gain another CPU core. This incurred about 8 minutes of intentional downtime of the service, but gave me some time to breath.

I messaged a co-worker and asked how I could survive what seemed to be a viral load of hits without incurring more downtime in a short amount of time. He mentioned NGINX Cache would be my solution.

So a quick Google and I stumbled upon this blog, which gave me a good base to start from and adapt.

proxy_cache_path /tmp/nginx_ghost levels=1:2 keys_zone=ghostcache:16m max_size=32m inactive=24h;

...

location / {
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header Host $http_host;
    proxy_pass http://127.0.0.1:2369;

    proxy_ignore_headers Set-Cookie Cache-Control;
    proxy_hide_header Set-Cookie;

    add_header X-Cache-Status $upstream_cache_status;

    proxy_cache ghostcache;
    proxy_cache_valid 1d;
    proxy_cache_valid 404 1h;
    proxy_cache_revalidate on;
    proxy_buffering on;
    proxy_cache_background_update on;
    proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504;
}

location ~ ^/(ghost/|p/|members/) {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
    proxy_pass http://127.0.0.1:2369;
}

location ^~ /assets/ {
    root /var/www/connor/blog/content/themes/nurui-github;
}

location ^~ /content/images/ {
    root /var/www/connor/blog/content/images;
}

location ^~ /content/images/(!size) {
    root /var/www/connor/blog;
}

A pretty cool set of changes that setup the goal to:

• Introduce a 1 day cache for blog posts to skip the origin servers after cached.
• Allow a stale cache (expired) to be returned while a new one is generated in background.
• Skip the cache for any admin/preview/member functionality.
• Skip Ghost for any direct file asset from my theme or uploads.
• Allow ghost to be used if the auto-generating image resize is in play.

Tested it out and set it live somewhere around noon. The results were insanely quick - the server load over the past minute had dropped to under 1.

I could load the blog on my phone and desktop insanely fast than before. A bit sad I hadn't ever looked into these optimizations before, but my blog previous record of unique hits in a day was ~3,000. I was already passed that within two hours so this was a good change to make.

So now I could relax a bit and add a comment to the post as the server was handling well. However, I was running up against a meeting at work at 1 so while I did just want to see the impact of this post live - I had other events to attend.

For someone that doesn't get a huge amount of social presence coming back to an influx of notifications was a change of pace.

• A bunch of emails from a variety of people.
• Tweets, Retweets and more.
• LinkedIn reshares and private messages.

This just really taught me how many people visit the same sites as I do. A couple friends messaged saying "Did you know you are on the top of HN???", but I guess they forgot my username is iBotPeaches.

What was most interesting in all these communications is how helpful people want to be to investigate something. I had folks continuing the research I started and suggesting additional paths to take - for a subset I had:

• Someone in Madison volunteering to go to the business I called out in person at the registered Madison address to ask questions live.
• Someone suggesting I create a fake LinkedIn account and attempt to be recruited by the suspected business.
• Someone suggesting I remove all my information from online if I don't want this to happen.
• Someone suggesting I go to the FBI and report identity fraud.
• Someone telling me my blog design sucks and shouldn't use JavaScript.

It was quite the whirlwind of activity all at once, but I was just happy the server was still healthy.

I then got a message from a friend that he saw my blog on the top of a subreddit he visits. So I learned you can just search Reddit with "site:connortumbleson.com" and isolate your search to link submitted by this blog.

What I found interesting with this is how information appears to travel from Hacker News to Reddit relatively quickly. I posted my own blog on Hacker News and everything posted to Reddit occurred from others within hours of the submit.

So now I understood why a huge amount of hits were coming in to the website at once. I checked the comments again and I saw Andrew at the top of the comment thread and I was happy for him. He was the reason I had that blog at all and it appeared he was getting a positive outcome from the crowd of Hacker News, so I sent him an email and went back to work.

Towards the end of the day still surprised my post was still at the top place my mind was changing to how much traffic is a "hug" of traffic from the socials.

So I configured GoAccess against my NGINX logs and let it run.

So it looked like a normal day has barely 1,000 unique views and 10,000 hits. The day the post went to the top of Hacker News registered 102,514 unique readers with 3,664,639 hits.

This is a 11,006% percent increase in traffic so I was pretty happy that folks were stumbling upon my blog. Since it was my 2018 resolution to blog weekly and since January 2018 I have not missed a week.

Sure some posts are pretty short and bad, but others are the culmination of a few weeks or days of research. So in the end I'm happy I learned how to further optimize my blog and glad a few folks enjoyed the read.