Spotty Frontier - Aug 2022

On Thursday August 18, 2022 I got to the office and noticed something a bit odd after noticing some weirdness at home. Sites like AWS Console and Atlassian were either not loading or taking a solid minute. Other sites however worked perfectly fine - so I chalked it up to temporary problems.

However, around 3pm and I was still noticing this, so I polled our office and surprisingly almost everyone aligned with these issues - except for the folks not on Frontier - they were fine. So at this point I guessed it was Frontier, but now we had a workaround. The benefit of having two service providers at the office - we just swapped and all our problems went away.

Friday came and went and the original Frontier network was still shaky. Now it's the weekend where I sit here typing this blog. I use Frontier at home and my connection is experiencing roughly an average 20% packet loss when using a video game debugging feature.

So I started with a live chat with Frontier - just in case this was a known issue.

I currently ran a test on the line. I did not see any physical issue that may be affecting your service at this moment.

I'm seeing that you have Fiber Optic services so I need to transfer you to the appropriate department, ok?

However, this agent than transferred me because he could not handle the type of service I had.

Thank you for being patient, Connor. Upon carefully checking on my end, I can't see any reported outages on your area.

The newer agent could not help me either, but I don't blame them - I figure this is a larger issue or they are simply not equipped with the right info to answer me.

So I started an investigation.

I started with Down Detector and it confirmed my assumption since both work and home were affected. Something was going on with Frontier isolated to the Florida area.

However, I started thinking maybe I want to blame Frontier when the issue might be elsewhere. I started looking at the sites I was experiencing issues with and 60% were confirmed hosting on AWS. To the best of my ability with researching Fortnite - it appears it was also hosted on AWS as well.

So I figured I should check the outages mailing list - it normally has some good technical discussions when a wider scale issue is occurring.

Unfortunately, outside of some discussions about AWS US-West-2 nothing concrete, so I was getting worried that this was more localized to my zip code.

So it was time to check Twitter as it normally has an no shortage of people complaining.

Thankfully this user had the right idea and just pinged them on Twitter and the response told me way more than my two service support requests. This user said they were in Tampa (Guess what - so am I) and Frontier said:

As discussed via direct message, the outage is still ongoing and there's no ETR posted yet due to a fiber cut. Rest assured that this will be taken care of for appropriate resolution. ^Jay

I was hoping since this message was on Thursday and this was days later that it would have been resolved, but it had not.

So I kept reading Frontier's responses and saw:

Hi, Becky. We're sorry to hear that you're having issues with the service. The outage is due to possible fiber/cable issue. Our techs are still investigating and working to get this resolved as soon as possible.
https://twitter.com/AskFrontier/status/1560335925967732736

I'm sorry for the service disruption, Ellejanae. There's an outage in FL due to a fiber cut and techs have located fiber cut that is due to new bore site for cell tower. Repair is still in progress and there is no ETR yet. You may send us a DM to check for updates. -Rezza
https://twitter.com/AskFrontier/status/1560261113392295936

No ETR posted, but our crews found the damaged cable and they're working on moving a fiber reel. We are pumping a manhole to confirm any empty conduit we can use. They will need to pump the second manhole shortly. - Shimi
https://twitter.com/AskFrontier/status/1560311450203066368

Hello, Cinzia. Upon checking, the ETR would be around midnight ET 8/19 but this is still subject to change. -JD
https://twitter.com/AskFrontier/status/1560465829216526336

As per the latest updates, the area-wide outage appears to have already been restored. For us to check if there are still known or isolated issues that could impact your service, kindly send us a DM for us to gather the necessary information. -Ly
https://twitter.com/AskFrontier/status/1560307222856417281

So this is amazing and confusing at the same time - I get information that a new cell tower bore severed a fiber line causing issues and its bit difficult to resolve due to things I don't understand.

Why does the Twitter support staff have more information than my direct support chat on the website? I did my support request on August 20 and all these tweets were posted on the 18th or 19th - so Frontier support should have been able to confirm an issue in my area due to a fiber cut.

However, even so it seems there are a variety of answers given on Twitter as Ly might say the issue is resolved at noon, while JD confirms an outage that won't be resolved till the next day. It seems its just a mess to get info from the official source on Twitter or directly via the support/call option. It does bother me though that an issue that is seemingly affecting an entire state has no confirmation outside of private support requests or Twitter replies.

In the slew of Twitter replies I find a user linking to DSLReports, which I honestly forgot existed and there was a forum post!

This was a good forum thread that linked to the same evidence I discovered on Twitter. So I slipped a reply in with some of the information below I discovered.

So at this point I wanted to look into the BGP announcements as those can be quite heavy if a line is cut as new routes for peering are announced.

Cloudflare recently launched a very public service for BGP information on any ASN, so it was time to look up Frontier (AS5650 Frontier Communications of America, Inc.)

So this is interesting - we see a huge change from the norm with 134,000~ announcements on August 17. I don't think that is a normal amount which might suggest a misconfiguration or a huge amount of network changes after a fiber cut.

However, a day later we get 127,000~ announcements which would suggest the issue was resolved and correcting itself. Though, we see a huge amount of announcements right afterwards which does not look normal given the trend of the past 7 days.

So my Internet is still not working right during the weekend, but the trend line of Frontier has stabilized. Is there an unfixed fiber cut affecting my issues or is there a misconfiguration in the wonderful world of BGP? I know I can't get a straight answer from Frontier to tell.

Until then - I'll join the upset customers and tweet Frontier and see what happens.

EDIT - August 22, 2022

I woke up today expecting a good ole Monday morning to have resolved my issues, but alas - it has not.

I jumped into another 1/1 chat with Frontier and they once again for the 4th time told me there is no issue.

I have checked that there's no active outage in your area. We need to check your battery back up unit which powers your ONT. This is usually located near the power outlet. Please try to reboot it by unplugging the power cord for 3 minutes. -JD

After that I provided a huge amount of links to the staff of other users on Twitter/forums experiencing issues. They responded with:

The reported outage has been restored already. Since you're still having issues with service, we need to do some real time tests in order to isolate where the problem is originating. -JD

So I know this issue only affects portions of the Internet, so I did roughly 10 pings to ESPN and my own blog.

--- espn.com ping statistics ---
12 packets transmitted, 11 received, 8% packet loss, time 11016ms
rtt min/avg/max/mdev = 8.332/8.990/10.425/0.687 ms

--- connortumbleson.com ping statistics ---
11 packets transmitted, 11 received, 0% packet loss, time 10017ms
rtt min/avg/max/mdev = 20.586/21.112/22.652/0.636 ms

Unfortunately passing these details to entry level support staff is wielding no results.

EDIT2 - August 22, 2022

I got home from work and tried my ping tests - 0% packet loss! I played a game of Fortnite and 0% packet loss! I believe this issue is resolved, but boy what a mess of information and delays to get here.

Timeline

• August 17, 2022 - 5pm - 134,000~ BGP Announcements from ASN5650.
• August 18, 2022 - 6am - Notice issues at home.
• August 18, 2022 - 8am - Notice issues at work.
• August 18, 2022 - 1:03pm - Frontier mentions draining manhole (Twitter).
• August 18, 2022 - 2:49pm - Frontier mentions fiber cut (Twitter).
• August 18, 2022 - 3pm - Swap work to other ISP - issues go away.
• August 18, 2022 - 9 to 5pm - Huge amount of BGP Announcements.
• August 19, 2022 - Midnight - Frontier ETA passes with no fix (Twitter).
• August 19, 2022 - All Day - Issue Persists
• August 20, 2022 - All Day - Issue Persists
• August 21, 2022 - All Day - Issue Persists
• August 22, 2022 - 6am - Issue Persists
• August 22, 2022 - 5pm - Issue Resolved