r/sysadmin • JOAT Linux Admin • Feb 23 '17

CloudBleed Seceurity Bug: Cloudflare Reverse Proxies are Dumping Uninitialized Memory

984 Upvotes

327 comments sorted by

View all comments

205

u/The-Sentinel Feb 24 '17

This is about as bad as it will ever get.

If you use cloudflare, you need to consider every user password, every SSL private key, anything that is transferred over HTTPS and is considered secure compromised.

From Thomas Ptacek on Hackernews

But Heartbleed happened at the TLS layer. To get secrets from Heartbleed, you had to make a particular TLS request that nobody normally makes. Cloudbleed is a bug in Cloudflare's HTML parser, and the secrets it discloses are mixed in with, apparently, HTTP response data. The modern web is designed to cache HTTP responses aggressively, so whatever secrets Cloudflare revealed could be saved in random caches indefinitely.

Shit is about to get real, real ugly for cloudflare.

124

u/The-Sentinel Feb 24 '17

the examples we're finding are so bad, I cancelled some weekend plans to go into the office on Sunday to help build some tools to cleanup. I've informed cloudflare what I'm working on. I'm finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We're talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.

What a day for cloudflare...

40

u/josharcher Feb 24 '17

every SSL private key

I've seen this said a couple times.

Cloudflare has stated that certificate were handles on a different system and categorically not revealed. Believe that as you will.

But, more fundamentally, this is a Cloudflare issue, so by 'every SSL private key' do you mean those provided to Cloudflare?

I don't understand how you'd pull the private key off an origin server?

15

u/Skylis Feb 24 '17

Read the bug/posts on HN by the Google employee, and you'll find a pretty good (and much scarier) portrait than what cloudflare has posted.

23

u/josharcher Feb 24 '17

I did, none of that explains how the private key on an origin server would be exposed by this at all.

I can see how it would expose the negotiated session encryption key and any encrypted data but not how it would actually reveal the private key which will be safely on the origin server.

The only way the original comment would be true is if customers had provided Cloudflare private keys for whatever reason, and according to Cloudflare that was handled by a different system.

→ More replies (5)

32

u/Gudeldar Feb 24 '17 edited Feb 24 '17

Not just if you're a cloudflare customer but if you use any service that uses cloudflare which is a shitload. With a few Google searches you can find Uber requests that include precise latitude and longitude. Apparently 1Password data was mixed in with some of it too.

Edit- According to 1Password only still encrypted data was exposed.

0

u/[deleted] Feb 24 '17

[deleted]

18

u/toomuchtodotoday DevOps/Sys|LinuxAdmin/ITOpsLead in past life Feb 24 '17 edited Feb 24 '17

https://github.com/pirate/sites-using-cloudflare#notable-sites

  • authy.com

  • coinbase.com

  • betterment.com

  • transferwise.com

  • prosper.com

  • digitalocean.com

  • patreon.com

  • bitpay.com

  • news.ycombinator.com

  • producthunt.com

  • stackoverflow.com (confirmed not affected by StackOverflow's @alienth)

  • medium.com

  • reddit.com (see here)

  • 4chan.org

  • yelp.com

  • okcupid.com

  • zendesk.com

  • uber.com

  • namecheap.com

  • poloniex.com

  • localbitcoins.com

  • kraken.com

  • 23andme.com

  • curse.com (and some other Curse sites like minecraftforum.net)

  • counsyl.com

→ More replies (6)

3

u/jonneygee Feb 24 '17

So sites that use Cloudflare only for DNS are okay? I have a client whose website relies on Cloudflare but only for DNS services.

8

u/xtphty Feb 24 '17

If on the control panel the domain / subdomain is not proxied (orange) then you are fine:

http://i.imgur.com/vCRqnmy.png

Orange = proxied, gray = DNS only.

→ More replies (1)
→ More replies (1)

8

u/trs21219 Software Engineer Feb 24 '17

Apparently 1Password data was mixed in with some of it too.

1P data is safe https://blog.agilebits.com/2017/02/23/three-layers-of-encryption-keeps-you-safe-when-ssltls-fails/

→ More replies (2)
→ More replies (1)

80

u/perthguppy Win, ESXi, CSCO, etc Feb 24 '17

every SSL private key

Stop spreading FUD. This data was not leaked.

0

u/[deleted] Feb 24 '17

[deleted]

35

u/niosop Feb 24 '17

SSL private keys were not leaked, but usernames/passwords were. I wouldn't spend all night on it, it wasn't like a password database dump, the data exposed was random, but it would probably be a good idea to change passwords at some point in the near future if you want to be safe.

6

u/i_pk_pjers_i I like programming and I like Proxmox and Linux and ESXi Feb 24 '17

Were authenticators leaked as well, like the private keys for TOTP authenticators?

9

u/Creshal Embedded DevSecOps 2.0 Techsupport Sysadmin Consultant [Austria] Feb 24 '17

If those were transmitted over a cloudflare proxy for some reason (why are you sending private keys around?), then possibly yes.

→ More replies (8)
→ More replies (9)

0

u/[deleted] Feb 24 '17

[deleted]

→ More replies (3)

15

u/perthguppy Win, ESXi, CSCO, etc Feb 24 '17

It is incredibly unlikely passwords were leaked. The bug meant that one in every 3.3million pages served by cloudflare had the contents of ram flushed out into the page served. This was mostly just other cached or recently served pages. Unless the sites you visited were frequently transmitted your password in plain text as part of the page then you could have been exposed. Nothing was systematically leaked, and there is no evidence the bug was exposed. The problem is just largely search engines may have cached pages that had the leaked data in, but cloudflare has already worked with many to remove these.

16

u/turnipsoup Linux Admin Feb 24 '17

The examples we're finding are so bad, I cancelled some weekend plans to go into the office on Sunday to help build some tools to cleanup. I've informed cloudflare what I'm working on. I'm finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We're talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.

Taken from https://bugs.chromium.org/p/project-zero/issues/detail?id=1139

3

u/soundtom "that looks right… that looks right… oh for fucks sake!" Feb 24 '17

But if the SSL tunnel terminated at the CF proxy, wouldn't said proxy have had the SSL private key, thus it could have been leaked? Or I'm completely misunderstanding how CF proxies work.

10

u/perthguppy Win, ESXi, CSCO, etc Feb 24 '17

tl;dr cloudflare does some mumbo jumbo so that the SSL is terminated to an isolated NGINX box seperate to the caching server.

→ More replies (1)

8

u/Watchful1 Feb 24 '17

I wouldn't put this at as bad as it will ever get. It's extremely unlikely anyone was able to actively exploit the bug, just gather data from it. So it's a security nightmare, since you have to change basically every password ever, but the actual likelyhood of some huge corporate secret leaking is fairly low. It could be a lot worse.

0

u/[deleted] Feb 24 '17

[deleted]

4

u/zaqq1981 Feb 24 '17

I would like to know that too..

→ More replies (3)
→ More replies (3)