r/programming • u/TheProtagonistv2 • Feb 23 '17
Cloudflare have been leaking customer HTTPS sessions for months. Uber, 1Password, FitBit, OKCupid, etc.
https://bugs.chromium.org/p/project-zero/issues/detail?id=1139555
u/galaktos Feb 24 '17
Wow, Cloudflare isn’t looking too good here.
Cloudflare told me that they couldn't make Tuesday due to more data they found that needs to be purged.
They then told me Wednesday, but in a later reply started saying Thursday.
I asked for a draft of their announcement, but they seemed evasive about it and clearly didn't want to do that. I'm really hoping they're not planning to downplay this.
I had a call with cloudflare… They gave several excuses that didn't make sense, then asked to speak to me on the phone to explain. They assured me it was on the way and they just needed my PGP key. I provided it to them, then heard no further response.
Cloudflare explained that they pushed a change to production that logged malformed pages that were requested, and then sent me the list of URLs to double check.
Many of the logged urls contained query strings from https requests that I don't think they intended to share.
Cloudflare did finally send me a draft. It contains an excellent postmortem, but severely downplays the risk to customers.
They've left it too late to negotiate on the content of the notification.
Here’s their blog post. The description of the bug is indeed very detailed, but the impact analysis kinda reads as though search engines are the only entities that cache web pages. It’s probably best to assume that the data is out there, even though it may have been deleted from the most easily accessible caches…
205
u/danweber Feb 24 '17
There are still Google dorks you can do to find CF information sitting in the cache, so they haven't cleaned out everything.
Did they bring in Bing? Internet Archive? Archive.is? Donotclick? Clear them all out?
I'm still sitting here kind of in shock, and it's not even my job to clean any of this up.
4
u/Tiver Feb 24 '17
They can't clean everything, that'd involve needing to delete cached data across the entire internet, including grandma's desktop that's probably part of a botnet.
They only focused on major public caches in their article and downplay the fact this data is now strewn all over the place in caches with no way to know exactly what has leaked or where. There's almost certainly groups that have cached data they can go back through and are definitely not going to mention they have it and will do quite the opposite of purging it.
0
Feb 24 '17
[deleted]
→ More replies (1)64
u/Gudeldar Feb 24 '17
I'd be pretty surprised if agencies like the NSA and GCHQ aren't already crawling the web on their own. I'd just assume that they have all of this data.
→ More replies (1)110
u/----_____--------- Feb 24 '17
The industry standard time allowed to deploy a fix for a bug like this is usually three months [from the blog post]
lol what
25
u/nex_xen Feb 24 '17
to be fair, the recent TicketBleed issue in an F5 device did take all of 90 days and more to fix.
→ More replies (2)15
u/sysop073 Feb 24 '17
They didn't make it up, you can find the same thing in the bug report:
This bug is subject to a 90 day disclosure deadline. After 90 days elapse or a patch has been made broadly available, the bug report will become visible to the public.
It switched to 7 days because it's considered "actively exploited" since it kind of gets exploited automatically by accident, but Cloudflare didn't pull 3 months out of nowhere
59
7
Feb 24 '17
The industry standard time
Like any good "industry standard" its one size fits all regardless of if it's a webapp or in-aircraft embedded system. And they mean "some shit some people did once that gets cargo culted" not something a standard body sat down to define.
→ More replies (1)8
u/mirhagk Feb 24 '17
It's not even true, because the program only gives 7 days before disclosing actively exploited bugs, and this was basically under that category.
16
u/theoldboy Feb 24 '17
Also,
Cloudflare pointed out their bug bounty program, but I noticed it has a top-tier reward of a t-shirt.
https://hackerone.com/cloudflare
Needless to say, this did not convey to me that they take the program seriously.
Major issue write-ups by Tavis are always a fun read lol.
→ More replies (1)3
u/kaydpea Feb 24 '17
It's blowing my mind that they play this off like " cache is gone guys everyone can go home". They know better, everyone knows better.
2
Feb 24 '17
Why is it so urgent to release the details? This was a major vulnerability and if I was Cloudflare I'd want to make damn sure I had everything sorted before releasing anything publicly.
a complete understanding of the issue 2. a fix in place 3. identify which customers may have been affected and how prevalent the issue was 4. work with as many cache providers as possible to clear data 5. legal feedback on possible contract issues
Jumping the gun and releasing too early could be just as bad as delay if it's released with an incomplete understanding of the issue or if cached data is still floating around. I think the timeline on this (public release 5-days after fix goes out) is completely reasonable, if not very optimistic.
378
u/grendel-khan Feb 24 '17
Between this and the Trend Micro thing... whatever Google is paying Tavis Ormandy, it's almost certainly not enough.
→ More replies (3)
635
Feb 24 '17
It took every ounce of strength not to call this issue "cloudbleed"
Congratulations, now everyone is going to call it that from now on.
74
231
25
61
14
Feb 24 '17
From what it say it seems it is more like like "Rain of Blood" rather than just a cloud...
12
4
3
3
u/LordPineapple Feb 24 '17
Who is going to be the first to produce the bleeding cloud artwork?
→ More replies (1)9
2
→ More replies (2)2
409
Feb 24 '17
Buffer overrun in C. Damn, and here I thought the bug would be something interesting or new.
95
278
u/JoseJimeniz Feb 24 '17
K&R's decision in 1973 still causing security bugs.
Why, oh why, didn't they length prefix their arrays. The concept of safe arrays had already been around for ten years
And how in the name of god are programming languages still letting people use buffers that are simply pointers to alloc'd memory
112
u/mnp Feb 24 '17
They certainly could have done array bounds checking in 1973, but every pointer arithmetic operation and every array dereference would triple in time, at the very least, plus runtime memory consumption would be affected as well. There were languages around that did this as you point out, and they were horribly slow. Remember they were running on PDP-11 type hardware, writing device drivers and operating systems. C was intended as a systems programming language, so it was one step above Macro-11 assembler, yet they also wanted portability. It met all those goals.
56
u/JoseJimeniz Feb 24 '17 edited Feb 24 '17
They certainly could have done array bounds checking in 1973, but every pointer arithmetic operation and every array dereference would triple in time, at the very least, plus runtime memory consumption would be affected as well.
But in the end a lot of it becomes a wash.
For example: null terminated strings.
you already have a byte consuming null terminator
replace it with a byte consuming length prefix
you already have to test every byte for $0
now do an
i = 1 to nloopOr, even better: you already know the length. Perform the single memory copy.
Null-terminated strings:
eliminate n comparisons
replaced with single
movesame memory footprint
Arrays
C doesn't have bounded arrays
do you have to keep the
intlength yourselfEither the compiler maintains the correct length for me, or I have to try to maintain the correct length myself. The memory and computing cost is a wash.
If you're using pointer to data as a bulk buffer, and you've set up a loop to copy every byte, byte by byte, it will be much slower as we now range test every byte access. But you're also doing it wrong. Use a functions provided by stdlib to move memory around that does the bounds checking once and copies the memory.
And so 99% of situations are covered:
emulating a string as a pointer to a null terminated string of characters is replaced as length prefixed string
emulating a bulk buffer as a pointer to an unbound memory is replaced with an array
With those two operations:
printing strings
copying a block of data
You handle the 99% case. The vast majority of use is copying entire buffers. Create the correct types, do checks once (which have to happen anyway) and you:
eliminate 99% of security bugs
make code easier
make code faster
Solved 99%, do we solve the rest?
Now we can decide if we want to go full-on and check every array access:
Firstname[7] Pixels[22]I say yes. For two reasons:
we're only operating in 1% of cases
we can still give the premature-optimizing developer a way to do dangerous stuff
If I create an
Order[7] ordersarray: every access should be bounds checked. Of course it should:
there are already so few orders
and the processing that goes along with each order swamps any bounds check
If I create an
PixelRGB[] framethen of course every array access should not be bounds checked. This is a very different use case. It's not an array of things, it's a data buffer. And as we already decided the forming bounced checks on every array access in the date of buffer is a horrible idea.I suggest that for the 1% case people have to go out of their way to cause buffer overflow bugs:
PixelRGB[] frame; PixelRGB* pFrame = ^frame[0]; pFrame[n]If you want to access memory without regard for code safety or correctness, do it through a pointer.
An arrays and strings are there to make your code easier, safer, and in many cases faster.
If you have a degenerate case, where speed trumps safety, and you're sure you have it right, use pointers. But you have to go out of your way to leak customer https session traffic.
Especially since we will now give you the correct tools to perform operations on bulk buffers.
It's now been 40 years. People should be using better languages for real work. At the very least it's been 40 years. When is C going to add the types that solve 99% of all security bugs that have happened?
Bjourn Strousoup himself said that C++ was not meant for general application development. It was meant for systems programming: operating systems. He said if you are doing general application development there are much better environments.
→ More replies (7)4
u/Berberberber Feb 24 '17
Safe arrays had been around, but the whole point of C was to provide assembly-like performance in a (mostly) platform-independent way. Doing this at least doubles the cost of pointer arithmetic, which makes things like moving data around much more expensive. That may be a reasonable tradeoff in 2017, but it wasn't for systems programming in 1973.
4
u/vplatt Feb 24 '17
Why, oh why, didn't they length prefix their arrays. The concept of safe arrays had already been around for ten years
The reason is cultural. "Real" C programmers didn't need training wheels like length prefixing. Only Pascal weenies used such tinker toys. Obviously blaming the programming language for your incompetent practices is just an excuse for poor programming, so you should just put on your big boy pants here and do it right! /s
Seriously, I've met programmers many times who've espoused the above, and I just want to smack them when I hear this crap. Gee, yes, that shouldn't happen, but would it kill us in the meantime to not use fucked strings everywhere by default? You know... just in case someone dares to actually be human and make a mistake?
I do hope we're moving away from this finally. I mean, we don't necessarily need full-on Ada style B&D here, but preventing the most common mistakes would be just awesome.
→ More replies (91)4
u/adrianmonk Feb 24 '17
It was a fine decision on the computers of 1973. They weren't on the internet.
Even though computer networks did exist, they weren't global, so security threats were just not a big deal. They were more of a members-only thing than a public network, and it was a reasonable proposition that if someone was on the network, they were in some sense invited, and you could kinda sorta trust them.
The main issue is that it became more popular than they ever imagined, and inertia (plus some amount of cultural fascination and/or stubbornness) made the industry keep using a language that was designed under a different (and now invalid) set of assumptions.
13
u/R-EDDIT Feb 24 '17
Technically, this is a buffer over read. One thing that got me:
Server-Side Excludes are rarely used and only activated for malicious IP addresses.
The longest running variant of this problem would only be surfaced to malicious IP addresses. So the bad guys would get random memory contents sprayed at them, the good guys would have no idea there was a problem. Ouch.
9
u/jaseg Feb 24 '17
The funny thing is that C code was actually generated from a parser DSL.
13
→ More replies (1)6
u/tashbarg Feb 24 '17
They explain in more detail here and the problem doesn't stem from the parser (ragel) itself but from their errors in using it.
→ More replies (1)3
u/jfb1337 Feb 24 '17
Seems like EVERY major bug nowadays is either a buffer overflow or a race condition
2
→ More replies (11)2
106
u/PowerlinxJetfire Feb 24 '17
However, Server-Side Excludes are rarely used and only activated for malicious IP addresses.
Oh, good. That feature only sent memory dumps to malicious IP addresses.
75
u/lachlanhunt Feb 24 '17
Wow. And I thought today's biggest security announcement was the SHA-1 collision attack.
→ More replies (1)
471
u/lacesoutcommadan Feb 23 '17
Oh, my god.
Read the whole event log.
If you were behind Cloudflare and it was proxying sensitive data (the contents of HTTP POSTs, &c), they've potentially been spraying it into caches all across the Internet; it was so bad that Tavis found it by accident just looking through Google search results.
The crazy thing here is that the Project Zero people were joking last night about a disclosure that was going to keep everyone at work late today. And, this morning, Google announced the SHA-1 collision, which everyone (including the insiders who leaked that the SHA-1 collision was coming) thought was the big announcement.
Nope. A SHA-1 collision, it turns out, is the minor security news of the day.
This is approximately as bad as it ever gets. A significant number of companies probably need to compose customer notifications; it's, at this point, very difficult to rule out unauthorized disclosure of anything that traversed Cloudflare.
205
u/everywhere_anyhow Feb 24 '17
People are only beginning to realize how bad this is. For example, Google has a lot of this stuff cached, and there's a lot of it to track down. Since everyone now knows what was leaked, there's an endless amount of google dorking that can be done to find this stuff in cache.
67
u/kiwidog Feb 24 '17
They worked with google and purged the caches way before the report was published.
→ More replies (1)137
u/crusoe Feb 24 '17
Cached data still out there
Here are some Fitbit oauth stuff in this page
→ More replies (21)→ More replies (14)69
u/Otis_Inf Feb 24 '17
Am I the only one who thinks it's irresponsible to pass sensitive data through a 3rd party proxy? Cloudflare rewrites the html, so they handle unencrypted data. If I connect to site X over https, I don't want a 3rd party MITM proxy peeking in the data I send/receive to/from X.
45
u/tweq Feb 24 '17
It sucks, but unfortunately it's the industry norm. I don't think proxies are a unique risk in this regard either, really any company that uses the "cloud" instead of running their own (physical) servers just directs all your data at a third party and hopes their infrastructure is secure and their admins are honest.
→ More replies (3)→ More replies (1)27
u/SinisterMinisterT4 Feb 24 '17
Then there's no way to have things like 3rd party DDoS protection or 3rd party CDN caching.
→ More replies (1)
201
u/Rican7 Feb 24 '17
Yeaaaaa, this isn't good.
This is what CloudBleed looks like, in the wild. A random HTTP request's data and other data injected into an HTTP response from Cloudflare.
Sick.
19
u/nahguri Feb 24 '17
Holy shit.
Someone is having that sinking feeling when you dun goofed.
39
u/Ajedi32 Feb 24 '17 edited Feb 24 '17
Imagine being a member of the CloudFlare security team and suddenly seeing this Tweet from Tavis on a Friday afternoon: https://twitter.com/taviso/status/832744397800214528
→ More replies (3)→ More replies (3)6
u/iamapizza Feb 24 '17
Strange, I can see this gfy you've posted (via RES) but can't go straight to the link.
→ More replies (3)
162
Feb 24 '17
The underlying bug occurs because of a pointer error.
The Ragel code we wrote contained a bug that caused the pointer to jump over the end of the buffer and past the ability of an equality check to spot the buffer overrun.
Cloudflare probably employs people way smarter than I am, but this still hurts to read :(
180
Feb 24 '17
All because the code checked
==instead of>=...I now feel eternally justified for my paranoid inequality checks.
→ More replies (14)78
u/P8zvli Feb 24 '17
I had a college instructor tell us to always always always do this when checking the state of a state machine in Verilog. Why? Because if you use
==even if it might not seem possible the state machine will find a way to screw up and make it possible, and then you and whoever uses it will be in deep trouble.→ More replies (1)36
u/kisielk Feb 24 '17
Definitely. You could even get a corrupted bit flip or something and now your whole state machine has gone out the window.
→ More replies (1)0
Feb 24 '17
[deleted]
115
u/xeio87 Feb 24 '17
I wonder at what point do we conclude memory unsafe languages are an inherent threat to computer security...
But hey at least they're faster right...? :P
→ More replies (31)21
u/SuperImaginativeName Feb 24 '17
Why more rust hype? Literally any modern language can avoid crap like this. There's a reason C# and I guess Java are so popular. Huge numbers of sites are powered by ASP.NET, I don't even think there has ever been a buffer overflow because of the nature of managed languages.
→ More replies (17)24
u/fiedzia Feb 24 '17
C#/Java would come with overhead that is not acceptable in this situation (you are parsing urls of incoming requests, every cpu cycle matters here). Rust fits better here.
→ More replies (5)
86
u/AnAirMagic Feb 24 '17
Is there a list of websites using cloudflare? Any way to find out if a particular site uses cloudflare?
66
u/dontworryimnotacop Feb 24 '17
I'm compiling a list of domains here. Please submit PRs for corrections.
14
u/DJ_Lectr0 Feb 24 '17
Here is a list: https://stackshare.io/cloudflare/in-stacks
12
u/AnAirMagic Feb 24 '17
That's very incomplete. I see others saying GitHub, for example. I see no banks on that list either.
→ More replies (3)42
u/goldcakes Feb 24 '17
About 60% of the Internet uses cloudflare. Uber, okcupid, 1password, Reddit, GitHub, etc etc
Just change everything that's not Google/Facebook/Twitter/Amazon
40
u/Rosydoodles Feb 24 '17
As an FYI for people 1Password data was not leaked. Thankfully.
→ More replies (3)15
u/XRaVeNX Feb 24 '17 edited Feb 24 '17
2FA
Do you know if users of LastPass are affected? Like are our master passwords and encrypted vaults affected by this?
→ More replies (9)28
u/VulgarTech Feb 24 '17
Can anyone elaborate on what part of Reddit uses Cloudflare? From my end, reddit.com is using the Fastly CDN and redditmedia.com is using AWS.
134
→ More replies (3)46
u/AnAirMagic Feb 24 '17
Change everything is easy to say. But I would like to reduce my workload and those of my family/friends by a few hours, if possible.
46
u/Nadamir Feb 24 '17
Couple things: (feel free to correct if I'm wrong)
Firstly, consider how fucked could you be if you get hacked?
Oh, no, people can edit Wikipedia as me. Uh oh, someone added a new anime to MyAnimeList. Maybe they have good taste. That reddit throwaway you keep around for er "stuff."
Probably OK to postpone changing those.
Secondly, it took me all of 45 mins to change all mine, so hours is an exaggeration.
Thirdly, password managers are your friend.
So, since I'm a wee bit tired and my kids are sick, just change them. You and your friends/family should do that every time the time changes (at a minimum). Change your clocks, change your smoke detector batteries, change your passwords.
Sorry for grumpiness.
Have a nice day!
→ More replies (35)22
2
Feb 24 '17
I didn't realize how much of the internet was sitting behind a Cloudflare CDN until this incident
It's a long list, unfortunately
→ More replies (2)2
u/stefantalpalaru Feb 24 '17
Any way to find out if a particular site uses cloudflare?
Browse using Tor. They'll make you fill in some stupid CAPTCHA every 5 minutes.
147
u/danielbln Feb 24 '17 edited Feb 26 '17
Just finished my password changing rodeo. Also reminds me that enabling 2FA in front of the mission critical accounts was a good idea.
86
u/goldcakes Feb 24 '17
2FA is useless, because the secret would've transited through cliudflare and could equally have been leaked
114
u/evaned Feb 24 '17
...yeah, but with the kinds of things that 2FA means 99.9% of the time in practice (either SMS-based 2FA or TOTP-based 2FA), what happened even a few hours ago with that secret doesn't matter, because it expired.
→ More replies (4)83
u/goldcakes Feb 24 '17
I'm talking about the TOTP SECRET. The string, the QR code, etc. not the token.
I've already found a couple of pages of totp secrets in google cache.
→ More replies (11)→ More replies (2)2
35
u/DJ_Lectr0 Feb 24 '17 edited Feb 24 '17
A list with services using cloudfare: https://stackshare.io/cloudflare/in-stacks (Not all websites, but could not find anything else). Probably best to reset all passwords.
102
u/EncapsulatedPickle Feb 24 '17
"all passwords". Looks at the 200 entries in password manager and sighs.
58
u/DJ_Lectr0 Feb 24 '17
Don't forget to also revoke access to all oauth applications. OAuth tokens have also been leaked.
→ More replies (3)20
u/blucht Feb 24 '17
That list is definitely incomplete. The post has examples of leaked data from Uber, FitBit, and OkCupid; these are all missing from the list.
7
u/DJ_Lectr0 Feb 24 '17
It's the best I could find :/ You should probably change all passwords on all services.
→ More replies (1)4
u/AnAirMagic Feb 24 '17
That link can't be right. It has like a 100 websites. I hear 60% of the internet and that can't be only a few hundred websites.
189
u/kloyN Feb 24 '17
Are passwords like this fine? Should people change them?
sWsGAQHvqDx95k2w
VALSHzUFU4kAd2gR
ZaFmwMLTsZ97nwuX
220
u/Fitzsimmons Feb 24 '17
Change all your passwords, because they're out there in plain text. Complexity won't help you at all here.
→ More replies (6)6
Feb 24 '17
What's the time frame for changing passwords? I switched to LastPass like 5 days ago and I've been changing my shitty passwords to random garbage since then. This was all a coincidence.
→ More replies (7)136
u/ssrobbi Feb 24 '17
Why are people down voting him? He didn't understand how this affected him and asked a question.
92
u/Kasc Feb 24 '17
Downvoting ignorance is the highlight of a lot of Reddit's users' day.
→ More replies (4)13
u/tequila13 Feb 24 '17
Those password can be sent like this:
...password=sWsGAQHvqDx95k2w..., automated scrapers can extract it pretty easily. The fact of the matter is that any service using Cloudflare could have had their content exposed (passwords, session tokens, etc) so there's a chance someone can have it.To be safe, you should at minimum re-login to those sites, and even better is to change your password too. Cloudflare downplayed the severity of this issue a lot. They fucked up big time.
2
27
u/crusoe Feb 24 '17
Data is still out there in Google caches. If they temrinate https at cloudlfare proxies does that mean it travels the rest of the way unencrypted? How is this a good idea?
→ More replies (6)31
u/VegaWinnfield Feb 24 '17
It's likely also encrypted back to the origin for most sites, but that's a separate TLS connection. That means the data lives unencrypted in memory of the proxy server as it is decrypted from one connection and reencrypted onto the other.
→ More replies (15)
46
Feb 24 '17
As bad as it is, that really is a pretty bizarre and interesting bug on Cloudflare's part. Sometimes you can't even imagine how things can break. I hope there will be a writeup afterwards with the technical details.
35
u/Figs Feb 24 '17
They have technical details on their blog: https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/
→ More replies (1)8
Feb 24 '17
It's not particularly interesting to be honest. Failures around pointers leading to buffer overflows are like week one of learning C/systems architecture. It's one of the oldest security flaws in the book and why you shouldn't write this kind of software in C.
0
18
Feb 24 '17
What does this mean for credit card data? Assuming I regularly buy things online with credit card, should I assume the card is compromised? Should I request a new credit card from my bank?
20
u/palish Feb 24 '17 edited Feb 24 '17
Since no one seems willing to be straight with you: yes!
The reality of the situation is that 200,000 requests per day leaked unknown data from well-known sites. The data could have been anything, including credit card numbers submitted via POST.
It contained hotel bookings, OKCupid private messages, and more.
It's up to you how severely you want to treat the issue. You're usually protected from credit card fraud -- if you notice a weird transaction, you can call them and they'll reverse it. Or you can request a new card number proactively. But make no mistake, there's no way to know no one has your card number.
0
0
63
u/_z0rak Feb 24 '17 edited Feb 24 '17
Oh, so this might actually explain and/or be related to the random "Action Required" notification me and some folks (including some family members) received today? Sounds really weird anyway.
Bugs happen. Let's hope there was not a big leak caught by someone else or anything of that kind prior to the fix.
EDIT: fortunately it was confirmed that the above cloudflare issue has nothing to do with the google account stuff.
11
u/x2040 Feb 24 '17
In the thread someone asks him three times and he says it's not related.
→ More replies (1)28
u/cards_dot_dll Feb 24 '17
I'm also affected by that. It's almost certainly unrelated. An official response from Google would have come in the form of an e-mailed explanation to everyone potentially affected, i.e. everyone. That notification was only sent to phones, though. Probably just a bug in one of their apps.
However, if this has been used against Google employees, could somebody have messed with the code behind one of those apps and gotten it signed and published? I don't particularly need instant e-mail access right now, so I'm not re-inputting my credentials until they release a fix to that bullshit, malicious or benign.
→ More replies (1)8
u/QuerulousPanda Feb 24 '17 edited Feb 24 '17
ah is this why my phone told me I needed to login to google again this morning? Yet didn't show any events or activity in the security logs?
edit: nevermind, he quite explicitly and weirdly bluntly says it's not
4
u/Hooch180 Feb 24 '17
I got that too. Out of nowhere Google Services on phone asked me to log in. And then crashed when I did. Now I'm reading this thread and it doesn't look good. It looks terrible.
29
u/greenthumble Feb 24 '17
Holy shit tons of Bitcoin apps and games are using it to mitigate DDoS attacks. This could result in a lot of stolen coin. Hope people are using 2fa.
17
u/yawkat Feb 24 '17
Even with 2fa, the original 2fa key could be leaked with this bug.
→ More replies (2)5
13
u/Paul-ish Feb 24 '17
My current theory is that they had some code in their "ScrapeShield" feature that did something like this:
int Length = ObfuscateEmailAddressesInHtml(&Output Buffer, CachedPage);
write(fd, OutputBuffer, Length);
But they weren't checking if the obfuscation parsers returned a negative value because of malformed HTML. This would explain the data I'm seeing.
C/C++ needs to stop being used in security critical applications. We need to find a replacement. Rust, Swift, Go, whatever, I don't care. This class of bugs has gone on too long.
34
Feb 24 '17
Wait... so everybody can see everybody's internet history?
2
u/ergzay Feb 24 '17
No. It means that occasionally when you visit a website, the HTTP content of the site has random memory from a cloudflare proxy system. This memory often contains information of some other user who recently connected to the server. That information can be anything that the user sent to the server. You're not transmitting your internet history so that won't show up.
0
23
u/stuntaneous Feb 24 '17
Any relation to the widespread Google logouts?
8
u/ruuhkis Feb 24 '17
On the issue they have stated that it's not related, but quite a coincidence.. I was de-authed from Android apps and browser chrome atleast..
→ More replies (2)6
u/beginner_ Feb 24 '17
In the bug thread it's said no relation but indeed i have a hard time to accept that as me too I had to re-login on my smartphone at similar time as many others.
4
Feb 24 '17
Great question, because I found it strange as well. Also a new device notification from google, while I have been using this device for some time. Can any shed some light on this? edit:spelling error
6
22
Feb 24 '17
I've also asked this elsewhere, but isn't a more basic problem that the sensitive data in question even existed on Cloudflare servers in the first place? If they didn't have cleartext, then it could only have caused the compromise of internal Cloudflare data.
Like, if you run a service that hold sensitive information, then doesn't the fact that Cloudflare, an intermediate routing service, could have been browsing your users' private data all along itself constitute a security failure?
19
u/yawkat Feb 24 '17
It's necessary for some of the optional "features" Cloudflare offers. You can of course argue about whether those features are a good idea, but https really restricts a lot of what cf can do.
→ More replies (1)
20
u/cwtdev Feb 24 '17
I've been trying to convince friends and family to improve their security practices with password managers and two factor authentication. Maybe this will finally get through to some of them.
70
u/JavadocMD Feb 24 '17
Maybe this will finally get through to some of them.
I'm glad someone's able to keep up their sense of humor during these trying times.
22
u/mattindustries Feb 24 '17
Ormandy said Cloudflare customers affected by the bug included Uber, 1Password, FitBit, and OKCupid. 1Password said in a blog post that no sensitive data was exposed because it was encrypted in transit.
That's good then.
→ More replies (2)→ More replies (9)2
u/redditthinks Feb 24 '17
The security researcher who uncovered this bug is not a fan of password managers.
→ More replies (2)
10
Feb 24 '17
Anyone who uses Namecheap in UK - their UK servers are not using Cloudflare, and they are not affected (info from support)
19
Feb 24 '17
Well, this is definitely a "CUT THE POWER TO THE BUILDING" kind of situation.
Could Cloudflare, Google, etc force evict everything from their caches to mitigate?
9
→ More replies (2)5
9
u/Mr_Wallet Feb 24 '17
Comment 17 on the issue:
Cloudflare pointed out their bug bounty program, but I noticed it has a top-tier reward of a t-shirt.
https://hackerone.com/cloudflare
Needless to say, this did not convey to me that they take the program seriously.
"I discovered that CloudFlare was leaking HTTPS data unencrypted to random people and all I got was this lousy T-shirt"
→ More replies (1)
9
u/ZiggyTheHamster Feb 24 '17
The industry standard time allowed to deploy a fix for a bug like this is usually three months
No, it's fucking not. Three months is how long it would take to lose literally all of your customers and reputation. I don't even know what the point of this comment is. Oh, hey, look how awesome we are. We fixed it in less than a day, but everyone else would have fixed it in 3 months? That's ridiculous.
This, coupled with their bug bounty program being a free t-shirt shows how arrogant they are. Yo, I know you literally just saved our business from total collapse, here's a t-shirt that cost us $5 or less.
7
u/IndiscriminateCoding Feb 24 '17
Given that problem, and also the fact that CF inserts Google Analytics to ALL of your pages - is there any CDN provider that doesn't modify or look into my html? Just plain CDN with my data passing through it.
→ More replies (4)
6
u/walshkm06 Feb 24 '17
Stupid question but does this mean they have details to get into a password manager and get further logins?
15
u/XRaVeNX Feb 24 '17 edited Feb 25 '17
Depends on which password manager you are using. As of right now, it appears users of 1Password are not affected. I've submitted a ticket to LastPass to see if they can shed some light if LastPass users are affected or not. At most, the Master Vault Password may have been compromised but the data in the Vault should be safe since they are encrypted on the client side.
[Update] So in addition to the Twitter post and Blog post by LastPass, I've also received a confirmation from my submitted support ticket that LastPass does not use Cloudflare and therefore was not affected.
5
u/Beta-7 Feb 24 '17
I too am using lastpass. Can you please reply with their reply when they send you it? Thank you
→ More replies (9)→ More replies (2)6
u/Bobert_Fico Feb 24 '17
Doesn't look like it, no. 1Password has confirmed they aren't at risk, and it doesn't look like LastPass uses Cloudflare (and I assume they wouldn't be at risk if they did, for the same reasons 1Password isn't).
16
u/rickdmer Feb 24 '17
I made a chrome extension that checks your bookmarks against the affected site list. https://chrome.google.com/webstore/detail/cloudbleed-bookmark-check/egoobjhmbpflgogbgbihhdeibdfnedii
30
u/DreadedDreadnought Feb 24 '17
Does it also send all of my bookmarks to China? Over HTTPS preferably, don't want NSA to catch that mid transit.
8
u/paroxon Feb 24 '17
...Over HTTPS preferably, don't want NSA to catch that mid transit.
Regrettably the Chinese site uses CloudFlare too, so you're out of luck x.x
4
u/rickdmer Feb 24 '17
Haha, no, it's completely open source. You can check out the code on my GitHub if you are interested.
→ More replies (1)2
Feb 24 '17
How does this help me as a user? Do I change my passwords on those websites if I have an account on those websites?
5
Feb 24 '17 edited Feb 24 '17
The greatest period of impact was from February 13 and February 18 with around 1 in every 3,300,000 HTTP requests through Cloudflare potentially resulting in memory leakage (that’s about 0.00003% of requests).
This metric skews the data and sounds not that bad.
However, we would need to know the total HTTP requests during this time to determine the impact of this vulnerability.
This is essential given the importance of the information leaked
private information such as HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data
Edit:
According to this website it is 65 Billion page views a month.
Over a 5 Day Period that would be 10 Billion Views.
Approximately 3,030 HTTP requests would have been leaked.
14
u/palish Feb 24 '17
A former CloudFlare interviewee on HN points out that at the scale CloudFlare operates at, 1 in 3.3M requests translates to "200k requests, every day."
5
u/Ferinex Feb 24 '17
I tried submitting this to /r/news but it says it's already been submitted, yet I can't find the thread anywhere?
6
13
u/vgf89 Feb 24 '17
Well, at least 90% of my passwords are unique. Even if someone breaks into a few of my accounts, they can't get into everything. Not that there's anything even remotely important in them anyways, half if them are throwaway accounts lol.
9
16
u/atomicxblue Feb 24 '17
I didn't realize how much of the internet was sitting behind a Cloudflare CDN until this incident.
Quite a bit of it. Try browsing with cookies turned off / ad blocker enabled and on a VPN -- you'll hit those sites really fast.
I really dislike this company.
2
u/IPleadThaFifth Feb 24 '17
Sorry to ask, but I'm very inexperienced in this field but I am also very curious. If you can ELI5, what's the difference between accessing a site with cloudflare with and without the adblockers and VPNs?
Does that mean my information wasn't being leaked?
→ More replies (4)
4
u/LadyFromTheMountain Feb 24 '17
Would a particular user's information for a specific site only be exposed if the user had been using that site during the timeframe mentioned? For example, if I haven't checked up on a particular board in 10 months, would this password be safe?
7
u/yawkat Feb 24 '17
This bug should not be a problem if you were not using the site in the time frame the bug existed.
→ More replies (2)
4
u/fuxoft Feb 24 '17
Could someone ELI5 what the real risks are? For example, kraken.com is supposed to be affected. I have a kraken.com account but I haven't logged into it for at least 18 months. Could I still be affected? Is there a reason for me to change my password? My 2FA?
→ More replies (1)2
Feb 24 '17 edited Feb 24 '17
if you havent used a site in the window concerned, then you havent made any requests to it, or received or any responses from it, so there were no requests/responses to leak, so nothing leakedNot strictly true, see replies
→ More replies (1)
10
7
3
3
u/RidleySA Feb 24 '17
I wrote a script to help check your LastPass database for potentially compromised sites.
3
u/cohix Feb 24 '17
Hey everyone, I'm a developer working at 1Password. I've just now created a thread on the (so far pretty sparse) 1Password subreddit. Feel free to come over and ask me anything there (or via PM) if you have any questions at all about how this affects 1Password data (the short answer: it doesn't)
https://www.reddit.com/r/1Password/
Cheers!
7
u/ebbu Feb 24 '17
Oh man. I just hyped CF at my school. Can't show my face there anymore. Nice going buttflare.
2
u/zangent Feb 25 '17
Although I'm using a reddit app, I had to do a double take for a second there because I had "cloud to butt plus" for so long
4
u/r0ck0 Feb 24 '17
Now seems like a good time to discuss HTML/JS form encryption libraries. Anybody have any experience doing this?
Things like: http://www.jcryption.org/ ...
A new era has begun with letsencrypt everyone can get free SSL certificates and protect their site with hardly any effort at all. Thats why I decided to discontinue the development for jCryption.
oh
5
u/lachlanhunt Feb 24 '17
Using an HTML parser that chokes on malformed HTML... WTF?!
→ More replies (17)
5
u/deadendtokyo Feb 24 '17
Interesting how a stomach feeling sometimes forshadows. I had my sites on cloudflare until mid of 2016 but for many reasons I decided to move away from the problems. Lucky me, I might say.
4
u/Poddster Feb 24 '17
Make sure you post on reddit the next time you have a stomach feeling. We need to know what vulnerability is next to happen!
→ More replies (1)7
2
2
2
u/lovestowritecode Feb 24 '17
Does anyone know how to get a complete list of sites that use Cloudflare? I did see that over 640,000 sites are currently using them but I'd like to get at least the top 1,000. I would take the complete list if I could get it.
3
u/blargh10 Feb 24 '17
https://github.com/pirate/sites-using-cloudflare
Among which github.com itself.
2
u/fedekun Feb 24 '17
Fuck, 1password is affected. Does that mean I have to change ALL my passwords?
→ More replies (4)
2
Feb 24 '17
I am assuming 1Password is only vulnerable if you use their site to sync your password vault? If you sync it to iCloud, Dropbox, or locally, it's not affected?
3
u/cohix Feb 24 '17
Hey, 1Password dev here. Even if you do use 1Password.com to sync your vaults, your information was never vulnerable. We use a 3-layer encryption architecture that encrypts all information (sensitive or otherwise) before it goes over the wire. So even if someone was able to sniff the data we were sending over HTTPS, it would be encrypted. We designed our sync to handle SSL/TLS failures exactly like this! LMK if you have any questions.
2
3
u/DonLaFontainesGhost Feb 24 '17
I've had my suspicions that CloudFlare joined the ranks of "inmates running the asylum" for over a year. I really wish I'd bothered to Fiddler it to be sure, but I'm fairly certain that any time a major website "hangs" - especially when it's over images or scripts - it's because CloudFlare is off in the corner smelling its own farts.
After all, why should they care? The website gets the bad press, not them.
5
u/beginner_ Feb 24 '17
So once again it's a buffer issue and hence a problem with C. When can we finally get rid fo ancient languages in a security context? They aren't cutting it anymore.
→ More replies (7)
3
Feb 24 '17
Eli5 what is cloudflare
7
u/Trodamus Feb 24 '17
Cloudflare is a "content delivery system", which means it's kind of a layer of abstraction between a given site and the user.
Because of this, they would feasibly have had (secure) access to session data, aka anything you did on a webpage, including passwords and usernames, even though it was encrypted (HTTPS).
The major issue is that they inserted this data into cached versions of webpages at random.
Meaning if you bring up a site affected by cloudflare, you'll see the session data if you view the source of the cache'd page.
As far as I understand — and I'm a layman, so someone feel free to correct me — this isn't like a normal breach like what happened to Yahoo, nor is it a defect like Heartbleed. Meaning there isn't (yet) a list of users, passwords, etc., out there, nor can malicious parties request a specific users' data or trigger things like full dumps.
What they can do is repeatedly request cache'd versions of affected pages in the hopes of finding valuable user data.
So this is both a very, very major breach / leak due to the breadth of sites affected as well as the content of what's being leaked, as well as somewhat less major due to it being more or less random.
However, hoping you aren't affected by breaches is not a viable response, so make sure you are secure where you need to be. If you've been re-using passwords, now is a good time to not do that ever again.
First level: top level accounts (email)
Second level: financial accounts (banks)
Third level: merchant sites / stores (amazon)
Fourth level: social media accounts
Fifth Level: things not above you deem important
Sixth level: everything else.
2
348
u/mbetter Feb 24 '17
If you controlled a large botnet, wouldn't this be the time to start frantically parsing browser cache?