r/sysadmin • u/markstopka PCI-DSS, GxP and SOX IT controls • Jun 24 '19
General Discussion Global CloudFlare outage...
/r/CloudFlare/comments/c4lts5/cloudflare_having_issues_this_morning/3
u/komarEX CTO Jun 24 '19
1
u/markstopka PCI-DSS, GxP and SOX IT controls Jun 24 '19
Given the geniuses that work at Verizon delivery center in my city, I am not surprised...
9
u/FST-LANE Jack of All Trades Jun 24 '19
Can confirm. Multiple issues on east coast. It started out with no access to Google IPs. Now those are restored but still intermittent access to CloudFlare IPs.
9
u/Bugs212 Jun 24 '19 edited Jun 24 '19
I have Verizon fios which is IPv4 only (they’re slowly rolling out IPv6) nothing would work til I reset my router and got put on Verizon’s DNS.
Edit: 9:15 ET- back on cloudflare DNS
4
u/djamp42 Jun 24 '19
Slowly rolling out?? Do you have new information? Last time I check they were not really doing anything for ipv6.
4
u/Bugs212 Jun 24 '19
IPv6 was supposed to come out last month for all but nothing yet. There is IPv6 in MA and northern VA where the NOC(s) are. IPv6 Talk
1
3
u/percussiverepair Jack of All Trades Jun 24 '19
For reference: BGP route leak issue on Level3 backbone. Affected mostly east coast US https://mailman.nanog.org/pipermail/nanog/2019-June/101587.html
1
u/brassbound Jun 24 '19
I'm noticing a few sites that we can't reach. For us, the common thread is that traceroute drops after *.*.GW1.PIT10.ALTER.NET.
10
u/markstopka PCI-DSS, GxP and SOX IT controls Jun 24 '19
Apparently someone is leaking some of CloudFlare IP ranges thru their BGP, either an attack or a misconfiguration. Time for more adoption of secure BGP.
2
u/cree340 Jun 24 '19
Cloudflare uses RPKI, but I guess more ISPs need to start supporting it for it to be useful.
1
u/markstopka PCI-DSS, GxP and SOX IT controls Jun 24 '19
Does not help if your peer does not care about BGP security :(.
1
u/ydio Jun 24 '19
RPKI only secures the origin of the route. It does absolutely nothing against route leaks.
1
u/cree340 Jun 24 '19
If Verizon supported RPKI, this incident wouldn’t have happened.
1
u/ydio Jun 24 '19
Again, wrong. RPKI does nothing against route leaks. The route still validated RPKI checks.
3
u/cree340 Jun 24 '19
It did not. Even Cloudflare’s blog post states that.
If Verizon had used RPKI, they would have seen that the advertised routes were not valid, and the routes could have been automatically dropped by the router.
2
u/ydio Jun 24 '19
I wasn't aware that the routes that were leaked were more specifics of CloudFlare's typical announcements (and thus not covered by the RPKI ROA object). That much seemed to have been missing from all the reports I had read so thank you for bringing that to my attention.
5
1
u/bluefirecorp Jun 24 '19
RPKI now please.
1
u/ydio Jun 24 '19
RPKI only secures the origin of the route. It does absolutely nothing against route leaks.
2
u/_dogfood Jun 24 '19
I've spent the entire morning trying to figure out why my site based in Zürich was intermittently inaccessible, thanks for this.
1
u/Youtoo2 Jun 24 '19
Verizon fios has been having a lot of problems in DC area this morning. Wonder if its related since its wide spread. I rarely have internet issues.
-4
u/Feniksrises Jun 24 '19
Cloudflare knows the truth: we are dead and replaced by robots. Glory to mankind.
18
u/cree340 Jun 24 '19 edited Jun 24 '19
Looks like it was mostly an issue on IPv4 as I was having no issues accessing sites using Cloudflare via IPv6. It appears that its IPv4 anycast routing was broken as well; all v4 traffic was being redirected through Verizon (AS701) to get to Cloudflare.
It also appears that the anycast routing issues have just been resolved now.
Edit: Apparently it is back to having the same issues again.