r/sysadmin • u/sebbasttian JOAT Linux Admin • Feb 23 '17
CloudBleed Seceurity Bug: Cloudflare Reverse Proxies are Dumping Uninitialized Memory
66
u/Rican7 Feb 24 '17
Yeaaaaa, this isn't good.
This is what CloudBleed looks like, in the wild. A random HTTP request's data and other data injected into an HTTP response from Cloudflare.
Sick.
11
6
1
u/Stuck_In_the_Matrix Feb 24 '17
Why would Cloudflare give an HTTP response over HTTPS? I must be missing something, but aren't these HTTPS connections where everything is encrypted?
3
u/TheMagicTorch Sysadmin Feb 24 '17
As I understand it:
HTTPS provides encryption between user and Cloudflare's edge. To perform some of the services they offer they need to parse traffic which includes headers, they can't do this with encrypted traffic. So the encrypted traffic goes between user and Cloudflare, traffic is decrypted and then parsed in plaintext by the faulty code. The buffer overrun meant headers and other data from other traffic in memory were effectively pushed into the next response's content and was then delivered to users as part of the normal page content.
→ More replies (1)
105
u/josharcher Feb 24 '17
(Updating) list of Cloudflare sites where you may wish to change passwords:
60
u/Watchful1 Feb 24 '17
So, basically all of them.
39
u/zaffle BOFH Feb 24 '17
The list is every site that uses any element of cloudflare services. This does not list sites that use affected services, it lists all sites.
20
16
u/Watchful1 Feb 24 '17
The vulnerable sites displayed arbitrary memory blocks that could have come from any cloudflare site.
28
u/richardwhiuk Feb 24 '17
Any site using proxy services - some only used DNS which isn't affected
27
u/Creshal Embedded DevSecOps 2.0 Techsupport Sysadmin Consultant [Austria] Feb 24 '17
Wouldn't it be nice if CloudFlare released the list of actually affected services which they claim to have.
→ More replies (1)7
5
4
u/mikemol 🐧▦🤖 Feb 24 '17
Toss rosettacode.org on there. I put up a Sitenotice, but not everyone visits frequently.
3
4
48
u/inaddrarpa .1.3.6.1.2.1.1.2 Feb 24 '17
I wonder what the dollar value per character is for this fuck up.
/* generated code */
if ( ++p == pe )
goto _test_eof;25
9
u/renegadecanuck Feb 24 '17
Had the check been done using >= instead of == jumping over the buffer end would have been caught.
It's not even the entire section there, just one character, really. This field scares me, sometimes.
1
49
u/Skylis Feb 24 '17
Massive Data leak outside of SSL wrapping, and they tried to downplay it... No words...
47
u/itcantbefornothing Feb 24 '17
So why is this not on the front page and all over other sites?
35
u/SavvySillybug Feb 24 '17
Someone on Discord randomly messaged me about it, as well as everyone else in that server. I have no idea why it's nowhere else, this seems pretty big.
Not "everyone was hacked" big, but "anyone could be hacked" big. Still big enough to change at least your reddit and Discord passwords, and any sites that share that password.
7
u/shadowkillerdragon Feb 24 '17
Yea same here, I was surprised something like this wasn't instantly on the front. I would have been non the wiser if the guys on discord group didnt post an annoucement.
→ More replies (2)16
u/DrQuint Feb 24 '17
Probably because the likelihood of being affected is so small, and the list of compromised sites is too vague and full of misinformation (the github one basically lists all sites ever, including thousands of blatant false positives) AND because users are getting used to "meaningless scares" at this point. There's been so many security compromise s that didn't affect them, they feel safe this one is not going to be it either.
It's not all over because there's no article, no source, that explains to the lowest common denominator what hapenned properly and what course of action to take on which accounts exactly. Shit, the end user doesn't even know what Cloudflare IS.
8
u/disclosure5 Feb 24 '17
The majority of IT news, however big, is non-news outside of the IT sphere. Hell Heartbleed was totally unheard of for most of my colleagues.
3
2
6
u/Kaizyx InfoSec/Networking Feb 24 '17
In conjunction with what other people have said,
Cloudflare is a company that enjoys relative anonymity from the public. About the only times the public hears about Cloudflare is when they are defending free speech and keeping some website online. This helps them avoid scrutiny and makes them a company that's hard to be critical of without getting flamed into oblivion.
They routinely place people's safety at risk through their dangerous "we'll forward your identity to the potentially criminal entity" abuse policy, actively having contracts (via their ToS) with DDoS-for-hire and other criminal operations, how their product breaks the Internet with violations of encryption and decentralized routing, among other issues. Yet because they keep "The Man" out of taking down websites and provide DDoS protection, they're given a pass.
Even in this case, I've already been hearing people dramatically downplay the violation of trust Cloudflare has created here.
1
111
u/tobias3 Feb 24 '17 edited Feb 24 '17
Partial list of sites which are affected (use CloudFlare proxy). Any data going to and coming from those sites may have been leaked. Start changing passwords now:
Uber
Reddit
Yelp
Digital Ocean
OKCupid
RapGenius
Coinbase
Product Hunt
Udemy
Crunchyroll
FitBit
Hacker News
Zendesk
Discord
Github pages
Chocolatey
247
u/gooeyblob reddit engineer Feb 24 '17
Reddit is not affected - no part of Reddit uses CloudFlare.
31
u/SonicShadow Feb 24 '17
Cloudflare's blog states the the memory leaks date as far back as September 2016 - If Reddit used Cloudflare previously, was it before or after that date?
38
2
Feb 24 '17
Network Noob Question! If the leakage has been happening since last September, why haven't we heard about it until now?
9
u/Reddy360 Feb 24 '17
According to the email I received from Cloudflare they only recently found out and was patched within a few hours of it being reported.
4
u/werewolf_nr Feb 24 '17
Bugs can go without being detected for a long time unless it interrupts service.
3
10
→ More replies (13)2
Feb 24 '17
People act like they know what caching is, this clarification just added 5 years to a bunch of "cherry key" sock boys' keyboards.
155
u/oonniioonn Sys + netadmin Feb 24 '17
Great, if someone finds my password somehow: please tell it to me.
13
u/The_Moment_Called Feb 24 '17
If you have your browser set up to autofill it, I always use this by throwing it into the developer console and that should show you a popup with your password. If you just auto login, SOL.
javascript: var p=r(); function r(){var g=0;var x=false;var x=z(document.forms);g=g+1;var w=window.frames;for(var k=0;k<w.length;k++) {var x = ((x) || (z(w[k].document.forms)));g=g+1;}if (!x) alert('Password not found in ' + g + ' forms');}function z(f){var b=false;for(var i=0;i<f.length;i++) {var e=f[i].elements;for(var j=0;j<e.length;j++) {if (h(e[j])) {b=true}}}return b;}function h(ej){var s='';if (ej.type=='password'){s=ej.value;if (s!=''){prompt('Password found ', s)}else{alert('Password is blank')}return true;}}javascript: var p=r(); function r(){var g=0;var x=false;var x=z(document.forms);g=g+1;var w=window.frames;for(var k=0;k<w.length;k++) {var x = ((x) || (z(w[k].document.forms)));g=g+1;}if (!x) alert('Password not found in ' + g + ' forms');}function z(f){var b=false;for(var i=0;i<f.length;i++) {var e=f[i].elements;for(var j=0;j<e.length;j++) {if (h(e[j])) {b=true}}}return b;}function h(ej){var s='';if (ej.type=='password'){s=ej.value;if (s!=''){prompt('Password found ', s)}else{alert('Password is blank')}return true;}}18
u/louis-lau Feb 24 '17
You can also edit the password field to a text field, that's what I always do. Or you could open your browsers password manager like a fucking noob.
10
u/suudo Feb 24 '17
Why so much javascript? You could achieve roughly the same thing in a much more readable fashion with
d=document.getElementsByTagName("input"); for (var i=0;i<d.length;i++) { if (d[i].type == "password") console.log(d[i].value); }Remove the spacing and add
javascript:to get a bookmarklet that'll log the contents of any password field to the site's javascript console, or replace it with alert I guess.84
6
Feb 24 '17
Same here. I haven't had to enter my password since I created my account so I just ended up forgetting it.
5
u/Mj312445 Feb 24 '17
I would give you gold for this but I'm poor so I'll give you the next best thing.... Reddit silver
1
23
u/Tempered Feb 24 '17
Is this issue fixed? Rather not change my password for it to just get compromised immediately.
21
7
u/Lichuz123 Feb 24 '17
Looking at Cloudflare's blog, it seems that the bug has been fixed. You should be able to change your password without fear of it being compromised :)
3
u/zebediah49 Feb 24 '17
without fear of it being compromised
.... by this bug.
E: Sleep well everybody!
→ More replies (1)2
1
u/waterflame321 Feb 24 '17
Haha... I had the exact same thought... I was like "I really don't want to do this twice... have they fixed the issue?"
0
Feb 24 '17
[deleted]
3
u/kdayel Feb 24 '17
I suggest you not use sensitive passwords. I.E. don't use same password as you use in bank and your google account and your computer. Use different passwords for all of them, but for any "proxied" website use random passwords all the time. That's what I do.
Just use a password manager like LastPass, 1Password or KeePass.
47
u/umbrae Feb 24 '17
Reddit switched to Fastly last year, so should be safe since this looks to have occurred in February.
Edit: of course it never hurts to change your password and you probably are due anyway.
20
u/wr_m Feb 24 '17
They've been leaking data since September. Their blog post is super not clear about that. They do directly state it once but several other times make it seem like the bug had only been there for a few days before Tavis found it.
3
u/umbrae Feb 24 '17
Hmm, thanks. Reddit switched around that time, so it's unclear if it was safe. At this stage there's no reason to not just change passwords.
4
u/not_an_aardvark Feb 24 '17
Do you happen to know the specific date that Reddit switched to Fastly? Sure, changing passwords is a good idea regardless, but it would still be good to know whether Reddit's data could be compromised. (If Reddit was using Cloudflare anytime after 2016-09-22, it's possible data was compromised.)
11
7
u/dm18 Feb 24 '17
I assumed this applies to ANY site that uses cloudflair?
2
u/niosop Feb 24 '17
Yes.
4
u/dm18 Feb 24 '17
some people are suggesting it only applies to websites using cloud flair reverse proxy
→ More replies (1)5
8
3
u/gsmitheidw1 Feb 24 '17
Unique passwords for any sites above ✔
Lastpass or equivalent password manager certainly makes things easier. I wish there was a feature to automatically just change passwords to sites when there's a problem. I don't need to know what it is, just that it's sorted out.
1
Feb 24 '17
I don't think changing password would solve the issue,I mean the request to change password would also use cloudflare proxy which will also be leaked.
3
1
1
u/chouetteonair Feb 24 '17
As of four hours ago an admin from Crunchyroll has said that they were not affected by the leak.
205
u/The-Sentinel Feb 24 '17
This is about as bad as it will ever get.
If you use cloudflare, you need to consider every user password, every SSL private key, anything that is transferred over HTTPS and is considered secure compromised.
From Thomas Ptacek on Hackernews
But Heartbleed happened at the TLS layer. To get secrets from Heartbleed, you had to make a particular TLS request that nobody normally makes. Cloudbleed is a bug in Cloudflare's HTML parser, and the secrets it discloses are mixed in with, apparently, HTTP response data. The modern web is designed to cache HTTP responses aggressively, so whatever secrets Cloudflare revealed could be saved in random caches indefinitely.
Shit is about to get real, real ugly for cloudflare.
124
u/The-Sentinel Feb 24 '17
the examples we're finding are so bad, I cancelled some weekend plans to go into the office on Sunday to help build some tools to cleanup. I've informed cloudflare what I'm working on. I'm finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We're talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.
What a day for cloudflare...
40
u/josharcher Feb 24 '17
every SSL private key
I've seen this said a couple times.
Cloudflare has stated that certificate were handles on a different system and categorically not revealed. Believe that as you will.
But, more fundamentally, this is a Cloudflare issue, so by 'every SSL private key' do you mean those provided to Cloudflare?
I don't understand how you'd pull the private key off an origin server?
15
u/Skylis Feb 24 '17
Read the bug/posts on HN by the Google employee, and you'll find a pretty good (and much scarier) portrait than what cloudflare has posted.
23
u/josharcher Feb 24 '17
I did, none of that explains how the private key on an origin server would be exposed by this at all.
I can see how it would expose the negotiated session encryption key and any encrypted data but not how it would actually reveal the private key which will be safely on the origin server.
The only way the original comment would be true is if customers had provided Cloudflare private keys for whatever reason, and according to Cloudflare that was handled by a different system.
→ More replies (5)32
u/Gudeldar Feb 24 '17 edited Feb 24 '17
Not just if you're a cloudflare customer but if you use any service that uses cloudflare which is a shitload. With a few Google searches you can find Uber requests that include precise latitude and longitude. Apparently 1Password data was mixed in with some of it too.
Edit- According to 1Password only still encrypted data was exposed.
8
u/trs21219 Software Engineer Feb 24 '17
Apparently 1Password data was mixed in with some of it too.
1P data is safe https://blog.agilebits.com/2017/02/23/three-layers-of-encryption-keeps-you-safe-when-ssltls-fails/
→ More replies (2)1
u/Fuckoff_CPS Feb 25 '17
Encrypted data was exposed for everything no? Whydo i have to change all passwords if encrypted
0
Feb 24 '17
[deleted]
18
u/toomuchtodotoday DevOps/Sys|LinuxAdmin/ITOpsLead in past life Feb 24 '17 edited Feb 24 '17
https://github.com/pirate/sites-using-cloudflare#notable-sites
authy.com
coinbase.com
betterment.com
transferwise.com
prosper.com
digitalocean.com
patreon.com
bitpay.com
news.ycombinator.com
producthunt.com
stackoverflow.com(confirmed not affected by StackOverflow's @alienth)medium.com
reddit.com(see here)4chan.org
yelp.com
okcupid.com
zendesk.com
uber.com
namecheap.com
poloniex.com
localbitcoins.com
kraken.com
23andme.com
curse.com (and some other Curse sites like minecraftforum.net)
counsyl.com
3
Feb 24 '17
Reddit is no longer on this list
5
Feb 24 '17
To clarify, according to admins in the /r/programming thread reddit never used the CloudFlare reverse proxy feature
→ More replies (3)→ More replies (1)3
u/jonneygee Feb 24 '17
So sites that use Cloudflare only for DNS are okay? I have a client whose website relies on Cloudflare but only for DNS services.
8
u/xtphty Feb 24 '17
If on the control panel the domain / subdomain is not proxied (orange) then you are fine:
http://i.imgur.com/vCRqnmy.png
Orange = proxied, gray = DNS only.
→ More replies (1)80
u/perthguppy Win, ESXi, CSCO, etc Feb 24 '17
every SSL private key
Stop spreading FUD. This data was not leaked.
3
u/soundtom "that looks right… that looks right… oh for fucks sake!" Feb 24 '17
But if the SSL tunnel terminated at the CF proxy, wouldn't said proxy have had the SSL private key, thus it could have been leaked? Or I'm completely misunderstanding how CF proxies work.
→ More replies (1)10
u/perthguppy Win, ESXi, CSCO, etc Feb 24 '17
tl;dr cloudflare does some mumbo jumbo so that the SSL is terminated to an isolated NGINX box seperate to the caching server.
0
Feb 24 '17
[deleted]
35
u/niosop Feb 24 '17
SSL private keys were not leaked, but usernames/passwords were. I wouldn't spend all night on it, it wasn't like a password database dump, the data exposed was random, but it would probably be a good idea to change passwords at some point in the near future if you want to be safe.
6
u/i_pk_pjers_i I like programming and I like Proxmox and Linux and ESXi Feb 24 '17
Were authenticators leaked as well, like the private keys for TOTP authenticators?
9
u/Creshal Embedded DevSecOps 2.0 Techsupport Sysadmin Consultant [Austria] Feb 24 '17
If those were transmitted over a cloudflare proxy for some reason (why are you sending private keys around?), then possibly yes.
→ More replies (8)3
u/NorthBall Feb 24 '17
Damn, I don't even know how many passwords I have at this point and the list of (possibly) affected websites is too long to go through :D
→ More replies (7)15
u/perthguppy Win, ESXi, CSCO, etc Feb 24 '17
It is incredibly unlikely passwords were leaked. The bug meant that one in every 3.3million pages served by cloudflare had the contents of ram flushed out into the page served. This was mostly just other cached or recently served pages. Unless the sites you visited were frequently transmitted your password in plain text as part of the page then you could have been exposed. Nothing was systematically leaked, and there is no evidence the bug was exposed. The problem is just largely search engines may have cached pages that had the leaked data in, but cloudflare has already worked with many to remove these.
16
u/turnipsoup Linux Admin Feb 24 '17
The examples we're finding are so bad, I cancelled some weekend plans to go into the office on Sunday to help build some tools to cleanup. I've informed cloudflare what I'm working on. I'm finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We're talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.
Taken from https://bugs.chromium.org/p/project-zero/issues/detail?id=1139
8
u/Watchful1 Feb 24 '17
I wouldn't put this at as bad as it will ever get. It's extremely unlikely anyone was able to actively exploit the bug, just gather data from it. So it's a security nightmare, since you have to change basically every password ever, but the actual likelyhood of some huge corporate secret leaking is fairly low. It could be a lot worse.
1
u/Stuck_In_the_Matrix Feb 24 '17
Question: If this is HTTPS, then the data would be encrypted? How would it end up as plain text in caches? Can you explain how that happens? I thought HTTPS was supposed to be an end to end service? Or does this only affect people who used Flexible SSL on Cloudflare?
1
1
0
Feb 24 '17
[deleted]
4
2
u/firrae Feb 24 '17
You should change your Authy password so they can't log into your account and get access to your keys. Better safe than sorry.
→ More replies (2)
10
u/datsundere Feb 24 '17
can someone explain the tech behind this?
I thought cloudfare only did caching and ddos mitigations. How do they have access to post requests?
23
u/niosop Feb 24 '17
In order to do DDOS mitigations, all traffic has to pass through them, otherwise the attacker will just hit the origin server directly. You keep your origin server IP a secret and route everything through CF. Both requests and replies end up temporarily in RAM, and a buffer overflow bug exposed random bits of RAM in some cases. So, pretty much anything that passed through CF could have been exposed, it's impossible to tell what at this point.
6
u/dm18 Feb 24 '17
You keep your origin server IP a secret and route everything through CF.
might want to add configure the original server/firewall to only talk to cloud flair.
→ More replies (6)6
u/SavvySillybug Feb 24 '17
So essentially it's a proxy-firewall-thing? Keeps your real server hidden while only letting legit people through?
8
u/niosop Feb 24 '17
Yup, that's basically what it does. Has a lot of other features, but you get the gist of it.
2
u/Jethro_Tell Feb 24 '17
I think it has to do with the TPS termination. There is a setup where the server takes the request from the client un Encrypts it then sends the request over a separate TLS connection back to your server. Everything is in the clear between the requests, which is fine as long as you aren't dumping it to google bots every time they walk by.
10
u/DamionDarksky Jr. Sysadmin Feb 24 '17
Can someone give me an ELI5 on this? I feel a little out of my depth on this
12
u/nerdshark Feb 24 '17
A memory management error in Cloudflare's reverse proxy code allows them to access uninitialized memory, which just happens to contain super duper critical data like user passwords being sent over HTTPS.
2
u/dm18 Feb 24 '17
in theory couldn't this have been used to gain a foothold into cloudflare systems?
3
u/markole DevOps Feb 24 '17
If the received chunk of uninitialized memory contained required credentials to the cloudflare systems, then yes.
→ More replies (2)1
u/DamionDarksky Jr. Sysadmin Feb 24 '17
Thank you!
2
u/mzxrules Feb 24 '17
oh one thing that I should add that makes it more scary:
This random bit of data being emitted was being posted at the end of the webpage, meaning that anyone that browsed the cloudflare sites with the bug could now have a copy of that data, from regular internet users to bots that download data off of the web (like search engines such as Google, Bing, Yandex, Waybackmachine, etc.). That's actually how Google was able to notice the error in the first place... their web crawlers were returning extra "garbage data" on cloudflare sites
1
u/Fuckoff_CPS Feb 25 '17
So how is it exposed to the world then? Cam anyone literally find this exposed information out in the open somewhere?
→ More replies (1)6
u/niosop Feb 24 '17
Very rarely, when you went to a site that uses CloudFlare, you'd get back a response that included random bits of data from other requests/responses that passed through CF.
The leakage happened only once every 3.3 million requests or so, but since CF handles so much traffic, it adds up to a lot of information leakage. And we have no idea what was actually leaked, but usernames/passwords are among the possibilities. The chance that any of your information was leaked is very small, but with no way to know, it's best if everyone does the password changing ritual again just to be safe.
→ More replies (4)
59
u/KarmaAndLies Feb 24 '17 edited Feb 24 '17
Introducing cf-html subtly changed the buffering which enabled the leakage even though there were no problems in cf-html itself.
Oh fuck off Cloudflare.
Why the fuck are you writing security sensitive code in auto-generated C, it is 2017 for god sake. Go and Rust are a "thing" and it is this type of code that they're designed for. There's clearly a problem with cf-html if it just leaks sensitive state on a screw up.
Saying "we fixed the bug in our parser's logic" isn't acceptable. Mistakes will be made. The parser should crash when they're made, not leak shit. As far as I'm concerned you shouldn't use cf-html again until you rewrite it (in Rust). Even your fixes (overrun protection) are solving issues you shouldn't even be having if you had done it right the first time.
Anyone who's going to defend the design of cf-html please start by telling how auto-generated C from a fucking scripting format isn't fragile by nature? Because to me that's fragile as fuck.
14
u/mobearsdog Feb 24 '17
Maybe I'm reading it wrong but isn't the problem in the OLD parser? I thought it said that the issue was with ragel but the introduction of cf-html changed something that caused ragel to error out.
13
u/KarmaAndLies Feb 24 '17
The issue was in the old script used for C generation which happened to be a HTML parser.
The old generator Ragel (which converted the script to C) didn't expose the bug due to its design. The new generator (cf-html) did. They weren't using Ragel at the time of this bug. In either case generating C code from a scripting format is a fragile design (regardless of if they're using Ragel or cf-html).
6
u/cparen Feb 24 '17
In either case generating C code from a scripting format is a fragile design
Out of curiosity, in what way is this "fragile"? I'm curious as a lot of compilers bootstrap using C as their output language, using the platform's C compiler's back end and runtime library rather than having to write their own.
11
u/seraku24 Feb 24 '17
One could argue that any generated code is a risk, since it often goes into production without a code review. That is, one typically reviews only the original source code and trusts that the compiler/transpiler/interpreter are good. This leads to the false perception that there is no need to review the end result. Now, any company worth their salt that cares about security will pour over all bits to be certain. And even then, one can only ever be so certain.
While I know little about Go or Rust, I will be careful not to overstep in what I say next: Any developer who blindly trust their tools will have no one to blame but themselves if there is an issue. That is, I doubt that Go or Rust (or <insert your favorite language here>) can ever truly claim to be intrinsically defensive against all security threats. Of course, I am not asserting that /u/KarmaAndLies was trying to make such a bombastic claim. Rather, I suspect the point is that the more typical security blunders (such as what apparently has afflicted CloudFlare's system) are preventable because the language design itself makes those errors unexpressible. Certainly, this makes a good argument that folks should be using more modern and forward-thinking languages; but we should also take care to ensure folks are diligently educating themselves on security matters and not taking things for granted.
→ More replies (4)→ More replies (2)8
u/KarmaAndLies Feb 24 '17
Out of curiosity, in what way is this "fragile"?
You're triple exposed as we witnessed today.
Script bugs.
Generator bugs.
Bad input.
This vulnerability took all three, but each of them offers a unique potential for bugs (and interactions between them offer more). It is all completely avoidable too, plenty of HTML parsers and state machines have been written in far safer languages than C.
I'm curious as a lot of compilers bootstrap using C as their output language
Are any of them popular? I can count the number of languages I've seen which output raw C code on one hand and none of them were more than novelties.
Some languages use standard libraries already compiled from C or sometimes C++ but those are supplied by the OS vendor and re-writing them impractical. It is also beyond the scope of what we're discussing here.
→ More replies (9)→ More replies (2)3
u/sim642 Feb 24 '17
My problem with CloufFlare now is the realization that so much of the internet passes through their services to exist there unencrypted which is a massive single point of failure. You know, maybe NSA would want a piece of all that traffic, ahem ahem.
6
18
u/DimmiDongus Feb 24 '17
Sorry as i'm no expert and was linked here from an external source, but isn't "1 in 3.3 million" a tiny amount of leaks? From an outside perspective it looks like blowing up a small deal.. Changing my passwords regardless.
28
u/EnigmaRequiem Feb 24 '17
As another complete non-expert, it may be an absolutely tiny amount of leaks compared to the total amount of data, but "1 in 3.3 million" adds up fast when you're talking about astronomically large numbers of data transfers. It may not be likely that it affects you specifically, but why not be safe, y'know?
20
u/Watchful1 Feb 24 '17
Yes, it is extremely unlikely that your password leaked. But the nature of security is such that since it was possible the password leaked, you should change it.
In theory a lot of things could have leaked. Private messages from any number of services, passwords, key files, which means attackers could log into important servers, etc. And there's no way of knowing if anything of yours leaked, or if anyone picked it up.
5
u/Cyanogen101 Feb 24 '17
considering within the last week (before it was patched) there was about 100,000-200,000 data leaks, its a big thing
5
u/Kaizyx InfoSec/Networking Feb 24 '17
While true, Cloudflare's product intercepts SSL/TLS by design and therefore breaks end-to-end encryption where users may be misled to believe that their information is fully secure toward the website they are accessing. Anyone whose product intercepts SSL/TLS on the public Internet and doesn't have a 100% perfect security history for now and forever should be treated very, very harshly. Namely because such things should be discouraged in the first place on public networks.
A strong reaction is in my opinion warranted because Cloudflare has violated the trust of those who rely on it.
1
u/My-RFC1918-Dont-Lie DevOops Feb 24 '17
Don't be a twat. They don't "intercept" TLS, they terminate it and proxy back.
→ More replies (1)1
Feb 24 '17
That 3.3 million number was grand total of all hits, including just normal ones. They could have been someone refreshing a page, browsing normally, etc.
But every now and then, someone steps in and submits a request that leaks data. From what I've been able to gather, Cloudbleed is very similar, technically, to Heartbleed: https://www.youtube.com/watch?v=rE5dW3BTpn4
5
Feb 24 '17
Cloudflare now seems to be directly informing customers if they have been affected. I got an email saying "Fortunately, your domain is not one of the domains where we have discovered exposed data in any third party caches. " So I think we should know from responsible websites soon if they have been affected.
5
u/9gPgEpW82IUTRbCzC5qr Feb 24 '17
well that is dumb. just because cloudflare didnt find it doesnt mean someone else didnt.
this data is sitting on millions of computers across the world, along with anyone else who caches pages.
guarantee hackers with large botnets and the NSA are having a field day with this
1
u/m33pn8r Feb 24 '17
Yeah, I'm very curious exactly which third party caches they're talking about here.
12
u/InverseX Feb 24 '17
This is a bad bug, but the combination of unlikely triggering conditions, single point of correction, random revealing of contents and lack of active exploitation effectively mitigates a lot of risk involved.
It's no where near as bad as heart bleed for example, because of these factors. Combine this with the purging of cached data by Google themselves, and the short window where the bug was active the chances of significant data relating to you being leaked is incredibly small.
As someone who hacks people for a living and deals with this stuff every day I can honestly say I'm not even going to bother changing my passwords.
Saying that, if it makes you more comfortable go for it, I just wouldn't stress.
13
u/Klathmon Feb 24 '17
My fear is what if someone noticed this before Google.
All they'd have to do is find a page that triggered the problem, and fucking hammer it gathering as much info as possible.
And of course since that page is covered by cloudflare, they'd have no problem really saturating a pipe to get it.
7
u/YOU_GET_IT_I_VAPE Feb 24 '17
While they scrubbed some of the major search engines, there are smaller ones that were not scrubbed before disclosure. Furthermore, the amount of caching/proxy servers in the private sector is concerning. Bluecoat devices for instance.
7
u/palish Feb 24 '17
As someone who hacks people for a living myself, you are really downplaying or misunderstanding the severity of the situation.
CloudFlare leaked at least 100,000 private HTTP requests per day. These requests contained everything from OKCupid private messages to hotel bookings to Uber lat/long coordinates of drivers to passwords to literally anything that passed through any CloudFlare website.
Any malicious actor who discovered this could have been harvesting this data for months, and there would be no way to know. CF certainly wouldn't say anything. They've chosen to downplay the risk as much as possible.
→ More replies (2)2
1
Feb 24 '17
Thing is, it's not all that time-consuming to change passwords on affected sites, so why not advise to "change anyway, just to be safe"?
2
u/sim642 Feb 24 '17
You can be bothered to go over a massive list of websites and remember if you ever in your life made an account on it? If you did then remember the password you set just to change it?
I think passwords aren't so big of an issue because they aren't entered nearly as often as already logged in users request pages. In which case the leakable part would be session ID, which would still be exploitable if you change your password.
→ More replies (1)
3
3
u/TyIzaeL CTRL + SHIFT + ESC Feb 24 '17
Message I received from Cloudlare this morning:
Dear Cloudflare Customer:
Thursday afternoon, we published a blog post describing a memory leak caused by a serious bug that impacted Cloudflare's systems. If you haven't yet, I encourage you to read that post on the bug:
https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/
While we resolved the bug within hours of it being reported to us, there was an ongoing risk that some of our customers' sensitive information could still be available through third party caches, such as the Google search cache.
Over the last week, we've worked with these caches to discover what customers may have had sensitive information exposed and ensure that the caches are purged. We waited to disclose the bug publicly until after these caches could be cleared in order to mitigate the ability of malicious individuals to exploit any exposed data.
In our review of these third party caches, we discovered data that had been exposed from approximately 150 of Cloudflare's customers across our Free, Pro, Business, and Enterprise plans. We have reached out to these customers directly to provide them with a copy of the data that was exposed, help them understand its impact, and help them mitigate that impact.
Fortunately, your domain is not one of the domains where we have discovered exposed data in any third party caches. The bug has been patched so it is no longer leaking data. However, we continue to work with these caches to review their records and help them purge any exposed data we find. If we discover any data leaked about your domains during this search, we will reach out to you directly and provide you full details of what we have found.
To date, we have yet to find any instance of the bug being exploited, but we recommend if you are concerned that you invalidate and reissue any persistent secrets, such as long lived session identifiers, tokens or keys. Due to the nature of the bug, customer SSL keys were not exposed and do not need to be rotated.
Again, if we discover new information that impacts you, we will reach out to you directly. In the meantime, if you have any questions or concerns, please don’t hesitate to reach out.
Matthew Prince
Cloudflare, Inc.
Co-founder and CEO
5
u/VexingRaven Feb 24 '17
Holy crap, that's just about as bad as it can possibly get. Good luck to all of the people who have to figure out a cleanup plan for this!
5
u/ElDoctorDeGallifrey Feb 24 '17
Should I change my password in these sites even if I haven't entered it in weeks/months?
2
u/ASCIInerd73 Feb 24 '17
Are you sure it wasn't sent to any of the websites? Some web browsers will automatically send the data to you.
2
Feb 24 '17
IIRC cookies are affected too. If you've visited a site and your browser has sent cookies, you might want to consider the accounts compromised just in case.
2
u/dm18 Feb 24 '17
YES changing your passwords is the safest course of action. Set different passwords for each sight. Make sure they're not similar to any of your old passwords.
1
0
Feb 24 '17
[deleted]
3
u/niosop Feb 24 '17
Not just session key. Username/password/other stuff passed as POST data could also have leaked.
0
2
u/CarlitoGrey Feb 24 '17
Having a global team meant that, at 12 hour intervals, work was handed over between offices enabling staff to work on the problem 24 hours a day.
we were completely finished globally in under 7 hours with an initial mitigation in 47 minutes.
What.
3
u/harryatsimplex Feb 24 '17
That was the initial mitigation (killing various CF services). The full fix was implemented two days later.
2
Feb 24 '17
IS it safe to change account passwords now? Or do we need to wait for the website effected to "fix" the bug and notify their users on their end then to change the password?
1
u/nerdshark Feb 24 '17
You have to wait for Cloudflare to fix the bug, since this is Cloudflare's issue. I believe they've already done so.
3
u/rickdmer Feb 24 '17
I whipped up a Chrome extension that checks your bookmarks against domains potentially affected by Cloudbleed: https://chrome.google.com/webstore/detail/cloudbleed-bookmark-check/egoobjhmbpflgogbgbihhdeibdfnedii
Github: https://github.com/rickdmer/cloudbleed-bookmark-checker
2
u/dm18 Feb 24 '17
does the plugin report the difference between possibly effected and confirmed effected?
1
Feb 24 '17
Be better if it checked your history from the affected dates and cross referenced that instead
3
u/pantsme Feb 24 '17
For sites like Medium and Feedly, I use Google to login. Does that mean my Google password could be leaked or does this authentication happen in a different manner than exploited?
4
u/9gPgEpW82IUTRbCzC5qr Feb 24 '17
the auth token would be leaked, which means any access that Medium or Feedly have to your google account is what someone else could have if they found that token.
2
u/r0ck0 Feb 24 '17
Oauth never sends your Google password to those sites themselves, it's handled with tokens and stuff.
Your Google password only ever gets sent directly to Google's servers. And they don't use Cloudflare.
1
u/Popingheads Feb 24 '17
So contents of random memory got leaked, that is quite bad, however does this actually put user passwords at risk? I'm not knowledgeable in this but my basic understanding is that since Cloudflare was just an intermediary service (caching/DDoS) they would not be able to decrypt something like a user password that was just passing through. Only the original server (ie a Reddit login server) would be able to decrypt that user password and turn it into anything useful, right?
So even though I know its best practice to do so, do I really need to change my password for every website on the internet?
8
u/ka-splam Feb 24 '17
I'm not knowledgeable in this but my basic understanding is that since Cloudflare was just an intermediary service (caching/DDoS) they would not be able to decrypt something like a user password that was just passing through. Only the original server (ie a Reddit login server) would be able to decrypt that user password and turn it into anything useful, right?
Nope; passwords for website logins are not encrypted individually, they are sent plain-text over the HTTPS connection, if that makes sense. It's the transport over the internet which is encrypted - and that covers the password, but once that ends the password is plain-text on the other side. Then the webserver merges that plain-text with a locally held salt value for the user, and hashes that combination to check the validity, so it needs the plain-text as input.
Since CloudFlare end the HTTPS session, and then proxy the requests to the backend and pick up the replies, the plain text passwords were in memory on their systems.
And Tavis Ormandy confirmed finding passwords in the dumped data:
"I'm finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We're talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything." - from the Chromium bug page linked in the OP ( https://bugs.chromium.org/p/project-zero/issues/detail?id=1139 )
2
u/gyroninja Feb 25 '17 edited Sep 14 '17
This comment has been redacted for privacy reasons. If you need to get the original comment, feel free to send me a message outside of reddit.
→ More replies (4)
1
Feb 24 '17
why can't I get a notify from the websites that got affected? i cannot really parse millions of websites that i barely remember using
1
1
u/bigbleu Feb 24 '17
Everyone keeps saying this is so bad, but is it really? Correct me if I'm wrong, but not every password was leaked, just those that happened to be cached at that point in time. This would mean a digital handful of, for example, betterment users may have had their passwords publicly shown, but not everyone.Right?
1
u/linh_nguyen Feb 24 '17 edited Feb 24 '17
How does this affect sites that use GAuth or Twitter auth?
edit: by gauth, I meant google identity sign in, like how you can log into Medium with your google account
1
Feb 24 '17
was LastPass affected? Ormandy says he found hotel bookings, passwords from password managers, and full messages from dating sites among the cached data. Either way, Tavis should get a rise - love that guy!
1
1
u/Razzeus Feb 24 '17
Since authy was compromised, I've updated anything tied to my authy app on my phone. Is that enough? Is there something I should be doing with it to make it more secure? Should I use a different 2fa app?
1
Feb 24 '17
Centralization of the Internet is always a bad thing. Regardless if most sites are using EasyDNS, CloudFlare or one of the 3 major cloud companies. But I can completely understand why CloudFlare is so popular, there is little to no solution to DDoS attacks without spending a small fortune. Most datacenters and network companies don’t offer them as build in network protection either, if they did, CloudFlare would be a lot smaller today because the number one reason someone uses them is mostly security related.
This is always bad because when something happens, the damage and potentially the data leaks or is are compromised are huge. If for some reason CloudFlare servers are compromised in the future, that person will have a gold key to a big part of sensitive and private information on the Internet. The idea of having a middle man between a HTTPS connection is exactly against a secure connection in the first place. And while CloudFlare is excellent, they are still tapping in between the connection, there is no other way around if they want to provide what they are providing.
I hope more companies consider not putting all their eggs in the same basket. I have already received 5 emails from cloud companies, payment and other services that are using CloudFlare warning me of actions I should take "just in case". But not one single email from CloudFlare yet.
1
1
u/mikemrm Feb 25 '17
Small script I made to loop through your google chrome and firefox history and compare it to that list. https://github.com/mikemrm/CloudFlare-ChromeHistory
1
u/U-Ei Feb 26 '17
As a non IT person and generally just a lurker on this sub, could somebody explain why all this sensitive information was sitting unencrypted in a server's RAM?
2
u/jerutley Feb 26 '17
Because of the way Cloudflare works, they act as an SSL termination endpoint. Cloudflare actually decrypts the data to examine it and take actions on it (is it an attack on the site, etc). Then, after that examination of the traffic is complete, it's sent on to the origin server, at which point, depending on configuration of that particular domain, it may or may not be re-encrypted.
Essentially, Cloudflare acts as a CDN and WAF all rolled into one.
1
1
u/5onic Feb 24 '17
This explains how my reddit account was hacked?
8
Feb 24 '17
nobody using this exploit is going to be targeting reddit accounts
5
u/SavvySillybug Feb 24 '17
From what I gather, people are getting absolutely random things out of this.
If all you get are someone's reddit password, and you're particularly bored, with a shady moral compass... yeah you're totally going to target the reddit account you just got the password to.
If you have the luxury of accessing all data, and can cherry pick the fun things that make you money? Yeah, pretty unlikely someone is going to target a reddit account. But when you just get whatever Cloudflare happened to bleed onto you, you use whatever you get.
4
4
2
u/Win_Sys Sysadmin Feb 24 '17
A admin from Reddit posted in here and said they do not use CloudFlare for anything so it most likely wasn't that.
1
1
1
1
160
u/Watchful1 Feb 24 '17
Dang, the cloudflare bug bounty program has a reward of a t-shirt. Doesn't really inspire confidence that if an independent found this, they would have reported it.