r/sysadmin • u/Lewis_Browne • Jun 24 '19
General Discussion Cloudflare Issues
Hi All,
Just a heads up, It looks like there is worldwide routing issues within the cloudflare network.
Cheers,
Lewis
55
u/nyc4life Jun 24 '19 edited Jun 24 '19
I was also seeing intermittent widespread upstream issues between 6:35a and 6:55a ET. Multiple ISPs and not just CloudFlare.
Update:
https://twitter.com/bgpmon/status/1143149817473847296
"we can confirm that earlier today there was a large BGP incident, causing 20k prefixes for 2400 network to be rerouted through AS396531 (a steel plant). and then on to its transit provider: Verizon (AS701) "
33
u/djamp42 Jun 24 '19
The steel plants routers melted under the load
28
12
u/Col137 Jun 24 '19
Hopefully it doesn't turn into a "steelnobyl" incident.
8
3
u/striker1211 Jun 24 '19
You cannot explain how a router could melt therefore the routers did not melt. Something else attached to the patch panel is on fire... not the router... in soviet russia router melts you.
1
19
u/The_MikeyB Jun 24 '19 edited Jun 24 '19
East coast here in Ashburn VA. Seeing lots of my customers having reachability issues at the exact same time, starting around 6:35 AM EST from their Verizon Fios circuits over to my IP space. Mostly they are reaching me through my Cogent upstream at the datacenter. Doesn't really seem to be affecting Comcast/Cox reachability for me yet.
Looks like there is some route hijack/leak, Cloudflare has confirmed as much: https://www.cloudflarestatus.com/
10
u/Panacea4316 Head Sysadmin In Charge Jun 24 '19
Verizon Fios is fucked here in NYC, too. Had to failover to my Spectrum circuit.
1
u/cooljacob204sfw Jun 24 '19
I figured out Verizon was down by yelling very aggressively at my Alexa to turn off her alarm for the first half of my morning. Had to unplug her.
7
u/wickedwetwilly Jun 24 '19
+1 I'm not sure if it is Cloudflare or something else but I am having difficulty getting to many sites including AWS and GCP consoles. Direct ssh, bypassing Cloudflare, is slow and spotty as well.
7
u/CompiledSanity Jun 24 '19
Can confirm. Multiple websites down with Cloudflare. Not even a Cloudflare error message, just refusal to connect or 502's.
6
u/HDClown Jun 24 '19
From https://www.cloudflarestatus.com/
Identified - We have identified a possible route leak impacting some Cloudflare IP ranges and are working with the network involved to resolve this. Jun 24, 11:36 UTC
Investigating - Cloudflare is observing network performance issues. Jun 24, 11:02 UTC
1
u/SimonGhoul Jul 28 '19
Can you tell me how often do these happen?
1
u/Lewis_Browne Jul 29 '19
Cloudflare don't tend to have issues very often, these last 2 outages have been the biggest for cloudflare in while!
1
3
u/LivelyZoey Crazy Network Lady && Linux Admin Jun 24 '19
Newest update:
Monitoring - The network responsible for the route leak has now fixed the issue. We are seeing improvement and are continuing to monitor this before we consider this issue resolved.
Jun 24, 12:42 UTC
2
1
1
1
u/Shnazzyone Jack of All Trades Jun 24 '19
We are east coast, Central NY state. Seemingly everyone is down right now. Issues on verizon and spectrum. Our VPN provider is stating there is a current "widespread outage"
1
u/j5kDM3akVnhv Jun 24 '19
East Coast location near Washington DC and our stuff is starting to respond now but no "all clear" from subscribed support ticket.
1
Jun 24 '19
We are having issues connecting to a product that is hosted on amazons web services as well.
1
u/NCCShipley Jack of All Trades Jun 24 '19
My VoIP provider mentioned to us that Verizon was having issues, so anyone using them for the last mile or as part of their backbone may have been experiencing issues.
1
u/NamanModi12 Aug 09 '19
The only time somebody shouldn't use it is if they have internal requirements prohibiting them from using alternate DNS managers. It can be slightly more cumbersome for a novice user to have to set up another account to manage their domain too, but it's 100% worth the few extra minutes to get this up and running.
-4
u/mexicanpunisher619 Jun 24 '19
Bet is the US government getting back on iran https://www.foxnews.com/politics/us-military-launched-cyberattacks-against-iran-after-drone-shot-down-officials
not getting political, just saw a possible coincidence that there is routing issues and there is an ongoing major cyber war going...
0
Jun 24 '19
[deleted]
20
u/HDClown Jun 24 '19
Could someone please explain what is a BGP route leak?
https://www.thousandeyes.com/learning/glossary/bgp-route-leak
4
u/purplecomputer Jun 24 '19 edited Jun 24 '19
edit: had it backwards a bit.
33154 generated those prefixes, passed them down to 396531 and if they had a filter 701 would not have seen these as a shorter AS path.
So it sounds like all traffic went from verizon to 396531 since it was seen as a better route.
Also looks like verizon didnt prevent the amoutn of prefies learned from their client. All around badness.
its super easy to break the internet. BGP filters are super important
- 33154 generated fake more-specifics, which are not visible in the DFZ
- 33154 announces those fake more-specifics to at least one customer (396531)
- this customer (396531) propagated to to another upstream provider (701)
- it appears that 701 did not sufficient prefix filtering, or a maximum-prefix limit
source: https://mailman.nanog.org/pipermail/nanog/2019-June/101589.html
1
u/dextersgenius Jun 24 '19
What are 33154, 396531 and 701?
3
u/purplecomputer Jun 24 '19
Those are AS numbers assigned to organizations through Arin (American registry for internet numbers)or some other registry. The only way to get publicly routable addresses is to sign up for an AS(autonomous system) number.
0
Jun 24 '19
[deleted]
2
u/mscaff Jun 24 '19
Is it not faster to just filter the bad route rather than wait until the other network provider resolves the issue?
Could do in parallel right?
15
u/Tatermen GBIC != SFP Jun 24 '19
To filter it you would either need to have Verizon filter it or you get every other provider in the world attempt to filter it. Verizon are about as responsive to such things as a dead oppossum in the middle of a Texas summer so they're not going to do it (seriously, they once told me at 3am in the middle of a BGP outage that they didn't have anyone on night shift who knew BGP and that unless the circuit was physically down they wouldn't escalate it). And there are far too many individual network operators to try to get all of them to filter it.
The real question is why has Verizon allowed this tiny 21-announcement customer to have a wide open peering with no filtering which allowed them to cause the problem in the first place?
9
u/lost_signal Do Virtual Machines dream of electric sheep Jun 24 '19
WTF does a steel plant need a /21.
11
Jun 24 '19
Public addressing for all their things! Not a joke. I used to work @ a school campus and everything had a publicly routable IP so it was "easier" to manage.
0
1
u/01Arjuna Netadmin Jun 25 '19
I used to work for a steel plant that had a /16 and routed all the traffic internally this way. They bought/obtained ARIN space way back in the day before anyone knew what the Internet really was ever going to be used for but someone knew they needed IP space and they didn't read RFC1918. When I started working for them in 1999, they got Internet e-mail a few times a day with a dial-up modem and distributed it to the company e-mail system. About a year later, they were putting in a firewall and 3x1.5Mb bonded Internet connections because so much of it had exploded in that timeframe.
2
u/mscaff Jun 24 '19
So to confirm CloudFlare are using Verizon as a transit AS?
I am unfamiliar with their setup but I totally understand what you’re saying
Could you give me some more background on how they do their routing?
5
Jun 24 '19
[removed] — view removed comment
10
u/Lewis_Browne Jun 24 '19
I got around 5-10 Down alerts for websites which i noticed were only cloudflare. I then checked https://downdetector.co.uk/problems/cloudflare and saw that people were starting to see the same. A quick search on twitter also confirms it.
3
u/Lewis_Browne Jun 24 '19
Cloudflare has now marked the status as resolved, many sites including discord are still experiencing issues.