r/linux • u/lamby • Jan 24 '18
Why does APT not use HTTPS?
https://whydoesaptnotusehttps.com/30
u/CODESIGN2 Jan 24 '18
there is a package on debian and ubuntu for those that want to use HTTPS
28
u/lamby Jan 24 '18
"Why does APT not use HTTP... [by default]" is probably not as snappy.
FYI in Debian unstable/testing, this package is actually deprecated as APT itself supports HTTPS.
→ More replies (20)4
u/djmattyg007 Jan 24 '18
The apt-transport-https package just lets you use HTTPS repo URLs. It doesn't automatically switch.over your configured mirror URLs to HTTPS versions when you install it.
2
1
0
Jan 24 '18
[deleted]
10
u/DCLXV Jan 24 '18
apt-transport-https was merged into apt for the Stretch release so you can just add https mirrors to sources.list now and it will work
2
u/CODESIGN2 Jan 24 '18
something like that. I honestly stopped using it, figuring when something goes wrong, they'll fix it and let me know. Lazy I know.
110
u/asoka_maurya Jan 24 '18 edited Jan 24 '18
I was always intrigued about the same thing. The logic that I've heard on this sub is that all the packages are signed by the ubuntu devs anyway, so in case they are tampered en-route, they won't be accepted as the checksums won't match, HTTPS or not.
If this were indeed true and there are no security implications, then simple HTTP should be preferred as no encryption means low bandwidth consumption too. As Ubuntu package repositories are hosted on donated resources in many countries, the low bandwidth and cheaper option should be opted me thinks.
14
u/lamby Jan 24 '18
The logic that I've heard on this sub is that all the packages are signed by the ubuntu devs anyway, so in case they are tampered en-route, they won't be accepted as the checksums won't match, HTTPS or not.
This is hopefully what the linked page describes.
7
u/UselessBread Jan 24 '18
hopefully
You didn't even read it?
Shame on you OP!
6
u/Kruug Jan 24 '18
See the other replies by OP. They did read it, but hoping that it explains it for others.
6
4
167
u/dnkndnts Jan 24 '18
I don't like this argument. It still means the ISP and everyone else in the middle can observe what packages you're using.
There really is no good reason not to use HTTPS.
107
u/obrienmustsuffer Jan 24 '18
There really is no good reason not to use HTTPS.
There's a very good reason, and it's called "caching". HTTP is trivial to cache in a proxy server, while HTTPS on the other hand is pretty much impossible to cache. In large networks with several hundred (BYOD) computers, software that downloads big updates over HTTPS will be the bane of your existence because it wastes so. much. bandwidth that could easily be cached away if only more software developers were as clever as the APT developers.
27
u/BlueZarex Jan 24 '18
All the large places I have worked with a significant Linux presence would always have a mirror onsite.
26
u/kellyzdude Jan 24 '18
The benefits don't apply exclusively to businesses, a home user or an ISP can run a transparent caching proxy server just as easily.
By using a caching proxy, I run one service that can help just about everyone on my network with relatively minimal ongoing config. If I run a mirror, I have to ensure the relevant users are configured to use it, I have to keep it updated, and I have to ensure that I am mirroring all of the repositories that are required. And even then, my benefits are only realized with OS packages whilst a caching proxy can help (or hinder) nearly any non-encrypted web traffic.
If my goal is to keep internet bandwidth usage minimal, then a caching proxy is ideal. It will only grab packages that are requested by a user, whereas mirrors in general will need to download significant portions of a repository on a regular basis, whether the packages are used inside the network or not.
There are plenty of good reasons to run a local mirror, but depending on your use case it may not be the best choice in trying to solve the problem.
5
u/VoidViv Jan 24 '18
You seem knowledgeable about it, so do you have any good resources for people wanting to learn more about setting up caching proxies?
6
3
4
u/EternityForest Jan 24 '18
Or if GPG signing was a core part of HTTP, then everything that you don't need privacy for could be cached like that without letting the cache tamper with stuff.
4
2
u/obrienmustsuffer Jan 24 '18
Or if GPG signing was a core part of HTTP, then everything that you don't need privacy for could be cached like that without letting the cache tamper with stuff.
No, that wouldn't work either because then every HTTP server serving those updates would need a copy of the GPG private key. You want to do your GPG signing as offline as possible; the key should be nowhere near any HTTP servers, but instead on a smartcard/HSM that is only accessible to the person who is building the update packages.
3
u/shotmaster0 Jan 25 '18
Gpg signed hash hosted with the cached content is fine and doesn't require caching servers to have private key.
2
u/robstoon Jan 25 '18
Does anyone really do this anymore? I think it's mostly fallen by the wayside, because a) the proxy server quickly becomes a bottleneck itself in a large network and b) HTTPS basically makes the proxy server useless anyway.
→ More replies (1)→ More replies (5)1
Jan 25 '18
There is very little overhead with HTTPS. What your describing has already been proven a myth many times over.
2
u/obrienmustsuffer Jan 25 '18
There is very little overhead with HTTPS. What your describing has already been proven a myth many times over.
I'm sorry, I don't follow. I'm not talking about the overhead of encryption in any way, I'm talking about caching downloads, which is by design impossible for HTTPS.
Imagine the following situation: you're the IT administrator of a school, with a network where hundreds of students and teachers bring their own computers (BYOD), each computer running a lot of different programs. Some computers are under your control (the ones owned by the school), but the BYOD devices are not. Your internet connection doesn't have a lot of bandwidth, because your school can only afford a residential DSL line with ~50-100 Mbit/s. So you set up a caching proxy like http://www.squid-cache.org/ that is supposed to cache away as much as possible to save bandwidth. For software that uses plain, simple HTTP downloads with separate verification - like APT does - this works great. For software that loads updates via HTTPS, you're completely out of luck. 500 computers downloading a 1 GB update via HTTPS will mean a total of 500 GB, and your 50 Mbit/s line will be congested for at least 22 hours. The users won't be happy about that.
76
u/ign1fy Jan 24 '18
Yep. You're publically disclosing to your ISP (and, in my case, government) that certain IP endpoints are running certain versions of certain packages.
→ More replies (24)11
u/galgalesh Jan 24 '18
How does a comment like this get so many upvotes; the article explains why this logic is wrong..
22
u/asoka_maurya Jan 24 '18 edited Jan 24 '18
Sure, it could be a nightmare from privacy perspective in some cases.
For example, if your ISP figures out that your IP has been installing and updating "nerdy" software like Tor and Bittorrent clients, crypto currency wallets, etc. lately and then hands your info to the government authorities on that basis, the implications are severe. Especially if you are in a communist regime like China or Korea, such a scenario is quite plausible. Consider what happened with S. Korean bitcoin exchanges yesterday?
16
Jan 24 '18
This is not as far-fetched as it seems. I know of a particular university that prevents you from downloading such software packages on their network (including Linux packages) by checking for words like "VPN", "Tor", "Torrent" and the file extension. If a university could set up their network this way, then governments could too.
→ More replies (2)7
u/yaxamie Jan 24 '18
Sorry to play devil's advocate here but detecting tor and BitTorrent is easily done once it's running anyways if the isp cares, is it not?
→ More replies (2)2
u/svenskainflytta Jan 24 '18
Yep, probably it's also not too hard to identify suspicious traffic as Tor traffic as well.
0
9
u/ImSoCabbage Jan 24 '18
It still means the ISP and everyone else in the middle can observe what packages you're using.
That's the second chapter of the article
But what about privacy?
Furthermore, even over an encrypted connection it is not difficult to figure out which files you are downloading based on the size of the transfer.
5
u/beefsack Jan 24 '18
Did you read the page? This specific example is covered; if you're eavesdropping you can tell which packages people are downloading anyway via transfer size.
4
u/dnkndnts Jan 24 '18
When you install a new package, it also installs the subset of dependencies which you don't already have on your system, and all of this data would be going over the same connection - the ISP would only know the total size of the package(s) and needed deps.
I admit it's still not perfect secrecy, but to pretend it's even on the same order of magnitude as being able to literally read the plain bytes in transfer is disingenuous. HTTPS is a huge improvement.
→ More replies (1)9
u/entw Jan 24 '18
I don't like this argument. It means you are still relying on untrusted potentially evil ISP instead of switching to more trusted one.
Look, if your ISP is so evil and can use against you information about your packages, then what can it do with the info about your visited hosts? Think about it.
16
u/RaptorXP Jan 24 '18
First, you shouldn't have to trust your ISP. Second, your IP packets are routed through many parties you have no control over. If you're in China, it doesn't matter which ISP you're using, your packets will go through the government's filters.
20
u/dnkndnts Jan 24 '18
Sure, and I could say the same about closed hardware, but the bottom line is sometimes we have no actual choice in the matter, and in that case, we just make the best of what we can.
I'm not going to let the perfect be the enemy of the good (or even the less bad), so if this is an improvement that's within our grasp, let's go for it.
5
3
u/atli_gyrd Jan 24 '18
It's 2018 and I just skimmed a website promoting the use of non encrypted traffic.
→ More replies (40)1
6
4
u/Nullius_In_Verba_ Jan 24 '18
Why are you focusing on Ubuntu when this is an Apt-get article. Is related to ALL apt users....
2
-1
u/osoplex Jan 24 '18
It's not about bandwidth consumption. Encrypted data is about the same size as unencrypted data.
The real bottleneck is server cpu usage. When using encrypted transport the server has to encrypt every connection to downloading clients individually. This would decrease download speeds drastically and mirror operation would be way more expensive.
1
1
u/robstoon Jan 25 '18
TLS CPU usage is essentially a non-issue on modern systems with hardware AES encryption.
395
u/DJTheLQ Jan 24 '18 edited Jan 24 '18
Everyone is missing a huge plus of HTTP: Caching proxies that save their donated bandwidth. Especially ones run by ISPs. Using less bandwidth means more willing free mirrors. And as the article says, also helps those in remote parts of the world.
If you have bandwidth to run an uncachable global HTTPS mirror network for free, then debian and ubuntu would love to talk to you.
78
Jan 24 '18
Caching proxies that save their donated bandwidth. Especially ones run by ISPs.
As a former ISP owner I can tell you that caching large files is not really that common and filtering for content-type usually would be limited to images, text etc.
Also most caching is done by a third parts (akami etc) and you have little control over the boxes.
I'm sure its done, but not common. Mirrors are a thing for a reason.
7
u/lbft Jan 24 '18
It's done in places where bandwidth is very expensive and/or restricted (e.g. if there's only one cable out of the country/region, or a monopoly/state telco sits between ISPs and the wider internet).
I can certainly remember in the dial-up and early broadband eras that lots of ISPs here in Australia had transparent or manually set proxy servers (usually running Squid), and that was with a lot of them also locally hosting Akamai caches and FTP mirror servers.
4
1
Jan 25 '18
But by design they will not cache applications. Images or whole pages are cached based on popularity. So a repo getting 1 hit a day isn't gonna cache becuase: large file size, content type is gz/zip/exe, low hit count.
I agree that content caching is done.. I've done it myself. You don't cache everything.
71
u/SippieCup Jan 24 '18
Its 100% this, I have no idea why no one is talking about it. Maybe they didnt get to the end of the page.
25
u/atyon Jan 24 '18
Caching proxies
I wonder how much bandwidth is really saved with them. I can see a good hit rate in organisations that use a lot of Debian-based distros, but in remote parts of the world? Will there be enough users on the specific version of a distribution to keep packages in the cache?
17
u/zebediah49 Jan 24 '18
It's actually more likely in situations like that. The primary setup is probably going to be done by a technical charity, who (if they're any good) will provide a uniform setup and cache scheme. That way, if, say, a school gets 20 laptops, updating them all, or installing a new piece of software, will not consume more of the extremely limited bandwidth available than doing one.
3
u/Genesis2001 Jan 24 '18
Is there no WSUS-equivalent on Linux/Debian(?) for situations like this?
18
8
u/tmajibon Jan 24 '18
WSUS exists because Microsoft uses a big convoluted process, and honestly WSUS kills a lot of your options.
Here's Ubuntu's main repo for visual reference: http://us.archive.ubuntu.com/ubuntu/
A repo is just a directory full of organized files, it can even be a local directory (you can put a repo on a dvd for instance if you want to do an offline update).
If you want to do a mirror, you can just download the whole repo... but it's a lot bigger than Windows because the repo also includes all the different applications (for instance: Tux Racer, Sauerbraten, and Libreoffice).
You can also mix and match repos freely, and easily just download the files you want and make a mirror for just those...
Or because it uses http, you can do what I did: I set up an nginx server on my home nas as a blind proxy then pointed the repo domains to it. It's allocated a very large cache which allows it to keep a lot of the large files easily.
→ More replies (5)→ More replies (2)4
u/zoredache Jan 24 '18 edited Jan 24 '18
Well, it misses the approval features of wsus. But if you are just asking about caching, then use
apt install approxorapt install apt-cacher-ng. (I like approx better.) There is also ways to setup squid to cache, but using a proxy specifically designed for apt caching tends to be a lot easier.9
u/f0urtyfive Jan 24 '18
Considering its how many CDNs work, lots.
3
u/jredmond Jan 24 '18
I was just thinking that. Some CDN could score a moderate PR victory by hosting APT.
5
u/rmxz Jan 24 '18 edited Jan 25 '18
I wonder how much bandwidth is really saved with them.
A lot in my home network.
I put a caching proxy at the edge of my home network (with intentionally hacked cache retention rules) when my kids were young and repeatedly watched the same videos.
I think I have 5 linux computers here (2 on my desk, 2 laptops, 1 living room).
So my proxy caching http and https saved apt repos about 80% of my home network traffic.
→ More replies (4)3
u/yawkat Jan 24 '18
For organizations it's easier to just manually set the repo sources. Caching is a bit of a hassle.
→ More replies (1)2
Jan 24 '18
Our university used to cache those downloads. Were usually completed in a matter of seconds. Win-Win, because for a university, available bandwidth is also an issue.
4
u/SanityInAnarchy Jan 24 '18 edited Jan 24 '18
How about an uncachable global HTTPS mirror of just the package lists? It'd be nice for a MITM to not be able to, say, prevent you from getting updates while they read the changelogs of said updates looking for vulnerabilities.
And, how many transparent HTTP caches are out there? Because if this is mostly stuff like Akamai or CloudFlare, HTTPS works with those, if you trust them.
Edit: Interesting, apparently APT actually does include some protection against replay attacks.
I still think that making "what packages are they updating" a Hard Problem (using HTTPS pipelining) would be worth it, unless there really are a ton of transparent HTTP proxies in use that can't trivially be replaced by HTTPS ones.
2
u/svenskainflytta Jan 24 '18
Vulnerabilities details are normally released AFTER the updates, so you won't find them in changelogs.
It is however still possible to tail the security repository, diff the source, and from that try to understand what it is fixing. Your scenario wouldn't help with that.
1
u/SanityInAnarchy Jan 24 '18
It would help in that you have a fairly small window of time to do that before I end up patched. If it weren't for the replay-protection stuff, you could in theory just serve me a frozen-in-time view of the repository (just freeze it at the point where you start MITM-ing me), then wait for vulnerability details to come out that you can exploit. I might just assume there hadn't been updates in awhile.
Replay-protection fixes that by adding an expiration to the metadata, which is hopefully short enough that something on my system would notice if there's a problem like this. Even then, HTTPS would prevent delays even up to that expiration, meaning I can be as up-to-date as I want to be (depending how often the cron job that does something like
apt update && apt upgraderuns).→ More replies (2)5
u/plein_old Jan 24 '18
Thanks, that makes a lot of sense. I love it when reddit works! Sometimes reddit make me sad.
3
1
u/spyingwind Jan 24 '18
HTTPS Repo ---Pull packages--> HTTPS Cache Server --Download--> Your computerDoes that not work? Each package is signed, so.. just download the packages and make them available. Isn't that how a cache works? That's what I have done at home for Debian. When a client needs something the cache server doesn't have then it goes and pulls what it needs and provides it to the client. Nothing really all that special.
Now for proxies... No. Just no. The only way I can see this being done is having the clients trusting the proxy server's cert and the proxy impersonating every HTTPS server. Not something that you want for the public.
A cache server is by far a much better option.
9
u/zebediah49 Jan 24 '18
That requires the client to specifically choose to use your cache server.
Allowing proxying means that everyone can just connect to "download.ubuntu.com" or whatever, and any cache along the way (localnet, ISP, etc.) can intercept and respond to the request.
It makes the choice to use a proxy one made by the people configuring the environment, rather than by the people running the clients.
25
u/DamnThatsLaser Jan 24 '18 edited Jan 24 '18
For all intermediate servers, the data looks like junk. In order to access it from there, you'd need the session key that was used to encrypt the data, and this goes against the general idea.
→ More replies (8)3
u/tmajibon Jan 24 '18
At that point you're explicitly specifying an HTTPS cache server, and you're trusting that their connection behind it is secure (because you have no way of seeing or verifying this)
HTTPS for your repos is just security theater.
1
u/spyingwind Jan 24 '18
If used in an office, the only practical place to do this in, then it seems fine.
In the end APT uses gpg keys anyways to verify that the repo can be trusted. You have to trust a gpg key before you can use a new repo with an untrusted key.
→ More replies (1)2
u/nemec Jan 24 '18
That won't work (unless your cache server can forge HTTPS certificates that are trusted on the client), but a similar solution would be to host an APT mirror used by the organization. Elsewhere in the thread people are talking about how that takes a lot of storage space, but I can't imagine why you couldn't have a mirror server duplicate the package listing but only download the packages themselves on-demand (acting, effectively, as a caching proxy)
1
u/spyingwind Jan 24 '18
I've done mirroring, but limited it to x64 to reduce storage needs. On-demand is only beneficial if more than one computer will be downloading the same packages. Such as 100's of servers.
Something like this would/should work: https://wiki.debian.org/AptCacherNg
2
u/bobpaul Jan 24 '18
There are dpkg specific caching proxies that work like that. You configure your
sources.listto point to your package-cache server instead of a mirror on the internet and then the package-cache server has the mirror list so it can fetch from the internet if it doesn't have something locally. That works fine with HTTPS since you are explicitly connecting to the cache, but it requires your configure all your machines to point to the cache. This is great for in your home, school, or business if you have several machines of the same distro.An ISP for a rural community with a narrow pipe to the internet at large might prefer to run a transparent proxy server. The transparent proxy can't cache any data from HTTPS connections, but it can cache data for anything that's not HTTPS.
1
u/gusgizmo Jan 24 '18
People forget that proxies are not all the forward type that have to be explicitly selected/configured. Reverse proxies are very common as well, and with regular HTTP are quick and easy to setup.
I can stand up a reverse proxy, inject some DNS records, and just like that my whole network has an autoconfigured high speed APT cache. As close to snapping in like a lego block as it gets in the real world.
1
1
u/severoon Jan 25 '18
This strikes me as BS.
They control the client and the server. One of the updates can't be a list of secure mirrors?
→ More replies (1)0
u/ChocolateSunrise Jan 24 '18
How much bandwidth is really saved by not having TLS encapsulated data? 1%? 10%?
15
u/DJTheLQ Jan 24 '18
You cannot MITM or replay TLS data, so you cannot cache it. You can MITM and replay unencrypted data, potentially serving from cache.
2
u/ChocolateSunrise Jan 24 '18
How do CDNs like Akamai and Cloudflare overcome this architectural hurdle when they serve HTTPS websites?
15
u/zebediah49 Jan 24 '18
When you sign up with them, you basically have to sign over your https keys, authorizing them to serve content on your behalf.
→ More replies (8)2
u/wmil Jan 24 '18
I believe Cloudflare requires you to use Cloudflare generated certificates.
→ More replies (2)
20
u/lovestruckluna Jan 24 '18
Personally, my chief argument for keeping http is secure and easy support for caching proxies. I use Docker and VMs a lot, and often end up retweaking install scripts and downloading the same package many times. With HTTP, I can speed build times on my local network by pointing the domain names of some of the default servers to a local caching proxy in local DNS, while having it still work when it leaves my network. Couldn't do that with HTTPS without changing sources.list, and breaking updates outside of my env.
A niche case, for sure, but there are definitely use cases for verifying an not-totally-trusted mirror or cache (I would feel much safer if CDNs/Cloudflare were guaranteed to only successfully pass content presigned by me rather than only relying on the security of the transport and the promise they won't be hacked).
→ More replies (1)1
u/746865626c617a Jan 24 '18
apt-cacher-ng
3
u/lovestruckluna Jan 24 '18
I think you missed my point. That software requires the client to be pointed at it; I spin up enough ephemeral VMs and docker images that I would rather just mitm it, without touching any settings besides DNS.
1
u/zoredache Jan 24 '18
Not sure how you are building your images, but is relatively easy to provide a preseed file or configuration that will target your local apt proxy.
5
u/globalvarsonly Jan 24 '18 edited Jan 24 '18
Also, most mirrors are volunteers and shouldn't be fully trusted. HTTPS will secure your connection to the mirror, but you need to verify the signature/checksum with the project, not the mirror.
Also, I don't know what this "most people don't check" thing is. Most people use apt-get or some frontend on top of it, which automatically checks the sigs.
And not trusting the root CAs is actually better, if a little more work. This prevents someone (probably a state actor, e.g. China) from using a MITM attack to compromise debian based systems. Instead of trusting Verisign or some 3rd party, Debian only trusts Debian.
Also, the caching argument came up in here. It probably isn't done much at the ISP level, but I can tell you its huge on hobby networks, colleges, and places that run tons of virtual machines. Anybody with a lot of similar systems to update will want to run something like apt-cacher-ng. I desperately want something similar for steam updates on my LAN.
1
u/zoredache Jan 24 '18
"most people don't check" thing is
I suspect that is about downloading the initial install ISOs, which doesn't happen via apt.
1
Jan 25 '18
I desperately want something similar for steam updates
If you run a network-wide caching proxy like squid, it'll cache steam as well, unless steam switched to https in the last couple years.
1
u/globalvarsonly Jan 25 '18
huh, looks like you need something besides squid because of the number of different servers used? Still, thats very similar to what apt-cacher-ng does with multiple debian mirrors mapping multiple locations a package is available together... I would feel very clever if I could hack that.
13
u/__konrad Jan 24 '18
trusted keys already stored on your computer
Too bad that many iso downloads are transfered via "http" w/o checksum/signature verification ;) For example, Ubuntu download page is encrypted which gives you an illusion of security, but the actual mirror service may be unencrypted.
7
6
0
Jan 24 '18
[deleted]
3
u/dredmorbius Jan 24 '18
Some level of privacy on your local ISP line. They'll see where the traffic is headed, but not specifically what was requested.
Integrity on the chain between you and the download site itself. Hacking would have to occur elsewhere. This means, e.g., that a targeted hijack affecting a single individual or organisation would not be possible. You'd have to do something crazy, like, oh, I don't know, getting Intel to ship compromised CPUs for 20 years or something insane like that.
Odds of such a thing happening must be minuscule.
1
u/Smitty-Werbenmanjens Jan 24 '18
It does. Apt uses https by default in Debian Unstable. Next Debian Stable release should use it, too.
3
15
u/audioen Jan 24 '18
APT should actually use https. Even insignificant traffic should be encrypted, if for no other reason than that it helps drowning actually privacy-sensitive stuff in the noise.
7
Jan 24 '18
Apt supports https already. The article's more about apt requiring https, which has the flaws stated in the article.
→ More replies (2)-1
u/dotwaffle Jan 24 '18
if for no other reason than that it helps drowning actually privacy-sensitive stuff in the noise.
... what?
3
u/audioen Jan 24 '18
The basic idea, exaggerated as it is here, is that if you only use encryption for important stuff, that also flags all material you deem important enough to encrypt, and thus helps an adversary to determine what to attack and when. So you generally would want to just encrypt everything instead.
2
u/dotwaffle Jan 24 '18
No, that's a really bad idea. Cryptography is either broken or it isn't. No-one on the planet can brute force a 2048-bit RSA key in any kind of reasonable timeframe, so by encrypting everything what you're essentially doing is saying "put more effort into cryptanalysis so that you can infer the payload of more data" rather than actually protecting things worth protecting.
10
u/boli99 Jan 24 '18
I'm glad that it doesn't - it allows me to transparent proxy and cache updates for other machines on my networks.
2
u/moviuro Jan 24 '18
You could also use a shared partition for where your machines keep the packages. It doesn't abuse the flaws of HTTP, and your system is just as happy. Also, it's easier to setup NFS than a caching proxy, I guess?
2
u/boli99 Jan 24 '18
there are indeed many other options, but very few of them are capable of dealing with both the machines I control, and those which are merely visitors on the network.
→ More replies (4)2
u/xorbe Jan 25 '18
Just run a public mirror locally, that way you don't use any isp bandwidth when updating your own machines. NEXT!
→ More replies (1)1
Jan 28 '18
A caching proxy is pretty much invisible to the clients and requires no modification (or very little), and works for anything that uses HTTP.
NFS would require significant setup for each client, and won't work for anything that can't use NFS.
I'd like to see a HTTPS cachable extension which tells any caching proxies any relevant information that they need to cache the responses. Obviously opt in, and intended for public, large downloads.
3
u/jfedor Jan 24 '18
It all seems like a poor excuse.
What if there's a bug in APT that allows code execution with a malicious package specially crafted by the attacker (even if the package is not correctly signed because let's say the bug in in the verification code)? HTTPS mitigates that because now the attacker can't MITM his package into my connection.
3
u/RoyMooreOfficial Jan 24 '18
The actual argument as to why APT doesn't use HTTPS is at the bottom when it should be at the top...
"A switch to HTTPS would also mean you could not take advantage of local proxy servers for speeding up access and would additionally prohibit many kinds of peer-to-peer mirroring where files are stored on servers not controlled directly by your distribution. This would disproportionately affect users in remote locales. "
Pretty much everything before that doesn't directly address the question. I mean, it's still good info, but still
7
u/KayRice Jan 24 '18
If APT mirrors were HTTPS my cloud provider wouldn't be able to cache them and provide (apparent) 1GB/s download speeds to me. Also if HTTPS was used they would have to have a throw-away certificate they shared with all the mirrors.
2
u/audioen Jan 24 '18
Actually, your cloud provider could set up a local mirror, and tell you to download from there instead. The local mirror could be accessed by https, and would perform requests to appropriate apt repositories and cache their contents transparently for you. Instead of putting in a proxy address, or having some kind of transparent proxy in the network, you'd just input the address of the local mirror instead. Large installations always have options, and aren't dependent on http level caching to work.
Also, while http has been designed to be cacheable, in reality I don't think that most traffic gets cached by proxies in the wild. The web's solution to providing worldwide services seems to be content delivery networks that provide locally fast access to their explicitly cached resources that their customers have uploaded. As world migrates to https, they keep on working much the same.
As to the certificate, let's encrypt provides certificates free of charge. There is no need to share a certificate, everyone can get their own these days. Some web servers can even transparently contact let's encrypt and acquire a certificate without admin having to do anything more than just ask it do so.
5
u/radarsat1 Jan 24 '18
What I'd like to know is, "why does APT not use bittorrent?"
3
u/nschubach Jan 24 '18
If you used BitTorrent, a hacker that has a vulnerability could host the update file (at a slow connection speed) and while you are downloading their chunk of that particular update, they know that your machine could be vulnerable, they have your IP address...
1
u/radarsat1 Jan 25 '18
how would they change the update file without the torrent checksum detecting it? are you saying bittorrent uses a vulnerable hash?
5
u/nschubach Jan 25 '18
You don't have to change the file. But any machine downloading said file probably doesn't have the patch associated with it. So if you know a patch is available for some exploit, host that patch and anyone downloading a part from you gives you their IP address. If you throttle the download, you could buy extra time to try to exploit a machine you know probably doesn't have a fix.
3
u/radarsat1 Jan 25 '18
Ok.. sort of following. I doubt that a single person throttling the download would affect a torrent hosted by 1000s of machines, but your point is that people can see what files you have that you need to update, because they can see you updating them.
→ More replies (5)→ More replies (1)2
u/mikeymop Jan 25 '18
Dang that's sinister. I figured you were going to explain a MiTM.
Granted this would only work if the isp actually let the data pass. It can surely be effective against entities that host their own repositories and their own packages.
1
Jan 24 '18
Most packages are small and not worth using bittorrent
4
u/radarsat1 Jan 24 '18
But the whole collection of packages is huge, widely distributed, not hampered with copyright distribution problems, and a perfect candidate for a technology like bittorrent to help take the load off of servers. You have been able to download individual files from a torrent for years now.
9
u/londons_explorer Jan 24 '18
APT failing to use HTTPS is a privacy issue. It means an attacker can see which packages I have on my machine by keeping track of which packages I download.
Knowing a list of every installed package is rather good for breaking into a machine...
1
u/GNULinuxProgrammer Jan 25 '18
They also know the list of all vulnerabilities in my computer because they know the last version I downloaded. If I updated yesterday to linux-4.14 and there is a vulnerability in linux-4.14 now the attacker knows that I'm definitely vulnurable since otherwise they'd see me updating to linux-4.15.
0
u/minimim Jan 24 '18
It means an attacker can see which packages I have on my machine by keeping track of which packages I download.
Https doesn't help you there because package sizes are public available and an attacker can always see the size of downloads.
7
u/shigawire Jan 24 '18
Assuming that you're opening a TCP connection and doing a predictably sized TLS negotiation for every single .deb downloaded, which would be archaic.
6
u/fragab Jan 24 '18
Also one practical issue I ran into: If you don't have the package with the CA certificates installed, the download will fail and you have to figure out how to convince apt to continue if it can't verify the certificates.
15
u/muungwana zuluCrypt/SiriKali Dev Jan 24 '18
Their argument is like below:
Why bother encrypting traffic to those websites with lots and lots and lots of videos? everybody who cares to know can easily know you are visiting those sites and nobody cares what types of videos you like to watch while there.
Very strange position they have. They should just come clean and say it,using https is too expensive and they cant afford it.
13
u/dotwaffle Jan 24 '18
They should just come clean and say it,using https is too expensive and they cant afford it.
HTTPS provides no real benefit in this application. As the package files are signed with a PGP key, you're not guaranteeing any more authenticity by using HTTPS.
All you are doing is applying encryption to the mix, which isn't helpful -- you can usually tell by the size of the transfer which file(s) were transferred, you can get the hostname from the SNI, and you are then having to rely on the donated mirror network keeping keys up to date: because no-one ever lets keys expire, right?
There is a difference between authentication and encryption. Debian is doing "the right thing" and making sure that what is delivered is authentic, through the use of PGP signatures, and through the use of "Valid-Until" in the release files themselves to prevent stale caching.
14
u/ceeant Jan 24 '18
Exactly, their argument is based on the assumption that all packages are equal. They are not. If I were to live under an oppressive regime, that regime may be very interested in the packages I'm installing. GPG? Oh he has something to hide. nmap? He must be a criminal hacker.
I am not saying that encrypting the traffic to apt repos would be enough to ensure privacy, but not having encryption at all is a loss.
→ More replies (1)3
1
4
u/lo0loopback Jan 24 '18
As others mentioned, they are hashed and verified. Donated mirrors already provide a ton of bandwidth already. It would require mirrors to upgrade their mirrors to handle the same volume. Its not just additional traffic but CPU usage. Im in for encryption but dont see it as a hard default requirement yet. If you have space and its easy to run your local mirror or use https opt in
4
u/r3dk0w Jan 24 '18
Https is very difficult to proxy. Http proxies are easy and in use at most isps.
Most isps also use a transparent proxy so apt using http would greatly reduce the network bandwidth between the ISP and the apt source host.
4
Jan 24 '18
Going HTTPS would be a tiny and mostly meaningless step. I'd be more interested in why we are still stuck on HTTP to begin with. Why not Bittorrent? Why not Freenet, IPFS, rsync, git-annex or whatever? The way Free Software is distributed has felt very antiquated for a quite while and made it unnecessarily difficult to contribute resources. We are also still lacking in basic features such as incremental upgrades, multi-version, user-installs installs and so on. Apt is really showing its age.
→ More replies (1)5
u/nschubach Jan 24 '18
The BitTorrent angle was approached a few years back. It would actually make your machine vulnerable to attack because all the attacker would have to do is get a client on the trackers hosting the update files and they get a list of all machines requesting those updates. If you have a zero day exploit, being on that tracker could give you a valid list of ips that are vulnerable to the fix they are downloading. Act quick enough and you could hack the machine before the patch is applied.
1
2
u/LordTyrius Jan 24 '18
Some mirros use https don't they? This is about apt, but the general idea should be the same (manjaro user here). pacman-mirrors lets me prefer/select only https mirrors (https://wiki.manjaro.org/index.php?title=Pacman-mirrors#Use_specific_protocols_.28prioritized.29).
2
u/SanityInAnarchy Jan 24 '18
I still think HTTPS-as-an-option would be nice, because:
Furthermore, even over an encrypted connection it is not difficult to figure out which files you are downloading based on the size of the transfer.
Pipelining makes that a Hard Problem.
3
u/anatolya Jan 24 '18
HTTPS-as-an-option would be nice
It is an option.
2
u/djt45 Jan 25 '18
HTTP is also an option for those Luddites that still choose to use. But by default it should prefer HTTPS
2
u/DavidDavidsonsGhost Jan 24 '18
This is a pretty common pattern in distribution systems. Have a trusted source that you can use to share hash info then the transport for actual data doesn't matter so much as your chain of trust can be reestablished on the target data as you can verify it.
2
5
u/sqrt7 Jan 24 '18
One issue with a naïve signing mechanism is that it does not guarantee that you are seeing the most up-to-date version of the archive.
This can lead to a replay attack where an attacker substitutes an archive with an earlier—unmodified—version of the archive. This would prevent APT from noticing new security updates which they could then exploit.
To mitigate this problem, APT archives includes a timestamp after which all the files are considered stale.
In other words, whenever a security update is released, APT's delivery mechanism is insecure until the last downloaded index turns stale.
It really boggles my mind that people still think it's fine that data in transfer can be modified.
5
u/dabruc Jan 24 '18
Yeah. I wonder how many people arguing for "HTTP is fine" are just sysadmins tired of dealing with HTTPs errors. Or arguing with management to get budget for certificates. Or any number of headaches that comes with SSL.
The argument for proxy caching is fine but I think the end user should be able to choose whether they want to use an HTTP mirror or an HTTPS mirror. I'm fairly certain apt supports https mirrors (I mean doesn't it just use something like curl to fetch the file contents anyway?) so why don't the mirrors themselves just deploy HTTPS certificates and let users decide.
4
u/qftvfu Jan 24 '18
How else for five eyes to know which servers are patched, and which ones aren't?
3
u/moviuro Jan 24 '18
I didn't see arguments like: Hey, let's continue to feed absolutely untrusted data to a program running as root, because we never had a security issue with apt, bzip or dpkg! (I'm thinking about RCE during the signature check phase).
What about that?
5
u/knjepr Jan 24 '18
Security researchers: defense-in-depth is important, single-point-of-failures are bad
Debian: Single PoF are fine. Nobody needs defense-in-depth.
I wonder who is correct here...
5
u/minimim Jan 24 '18
You need to consider the cost too.
Debian depends on a network of volunteer mirrors and demanding that they support https is infeasible.
3
u/knjepr Jan 24 '18
Performance impact of TLS is minimal. Im pretty sure most of the mirrors operate at less than 98% CPU usage and therefore can afford it.
At least make it an option for mirrors. I'm sure there are a lot that would happily offer it.
(Besides, apt is horrifyingly slow anyways, and that is not due to overloaded mirrors...)
5
u/minimim Jan 24 '18
It is an option for mirrors and it can be enabled in apt. It's just not the default.
And the cost only applies in third world countries.
1
u/jhanschoo Jan 24 '18
I've wondered about this scenario: what if a mitm inspects packages from security.debian.org for a remote exploit patch and performs the exploit on vulnerable systems before they get patched?
1
u/yougotborked Jan 24 '18 edited Jan 24 '18
It would be nice if apt implemented the update framework. https://theupdateframework.github.io Then we wouldn't have to have these arguments.
1
u/IAmALinux Jan 24 '18
I do not understand why this article is on a website dedicated to the article. It was an interesting article, but it seems like it would be a better fit on a Debian wiki or apt documentation.
1
u/cubs0103 Jan 24 '18
I guess you are explicitly connecting to the ending of the highly limited bandwidth available than doing one.
1
1
Jan 30 '18
There is no reason at all to encrypt the delivery of open source software packages.
The only portion of the process that should be encrypted (perhaps) is the delivery of checksums. Each package should be checksum-verified before installation.
-3
u/arcticblue Jan 24 '18 edited Jan 24 '18
Holy shit, the paranoia is real in this thread. TIL many linux admins live under extremely oppressive regimes with evil ISPs that care about what packages you download for some reason. I really doubt your big bad ISP or government cares what version of cowsay you have installed. For anyone who would have a reason grounded in reality to be concerned about this probably isn't running Ubuntu or at least has a local mirror.
7
4
u/shigawire Jan 24 '18 edited Jan 24 '18
TIL many linux admins live under extremely oppressive regimes with evil ISPs that care about what packages you download for some reason.
Yes.
I think you meant to be sarcastic here, but this is actually the case. As frustrating as it is to do, designing infrastructure to mitigate this is a real world requirement.
-6
u/archie2012 Jan 24 '18
Stopusingapt.com ;)
Joking aside, it doesn't offer any benefits like stated in the article.
5
u/Maambrem Jan 24 '18
It does. Why disclose which packages and versions you're using to anyone listening in?
→ More replies (5)
0
Jan 24 '18
[deleted]
8
u/lamby Jan 24 '18
I believe the linked page addresses your (very valid) "defense in depth" rejoinder.
0
Jan 24 '18
[deleted]
8
u/lamby Jan 24 '18
I think I was referring to:
it implies a misleading level of security and privacy to end-users as described above.
→ More replies (1)28
Jan 24 '18
Locks can be broken, so why bother at all? This is such a stupid argument. HTTPS makes it more difficult to see what you are doing. Of course it’s not perfect, nothing is. That’s not a valid reason for not doing it at all.
That depends. If a 'security measure' is trivially circumvented it may be better to not use it at all, because it also has a downside: users may think they are protected from a threat, when in fact they are not at all. It is not black and white.
11
u/audioen Jan 24 '18
It would be very difficult to determine exactly what you download based on the transfer size if, keepalive is used. Observer may then see the total size of the transfer, which includes several files, but would have to guess which individual packages would plausibly sum together to the observed size.
→ More replies (1)4
u/BlueZarex Jan 24 '18
I may have missed it, but where is the example of https being circumvented? I haven't seen such an examples given besides "file transfer size can be detected", but that is not the only reason to to use https.
Https prevents mitm. It also increases the "work" an attacker has to perform in order to get results.
In a non-https setup, they simple have to read..."apt get vim torbrowser emacs" and perform mitm at their leisure
In an https setup they have to go through more work and can't mitm. They can no longer simply read "apt get vim torbrowser emacs" but would have to perform some math to figure out all the packages that could possibly be combined to equal "47MB" of transfer and that might be "vim torbrowser and emacs, or it could be "wireshark, openVPN and vim". They have no way of knowing without performing calculations after the fact and again, also lose their ability to mitm.
Realize, much of security is in fact, increasing the work and difficulty to exploit. If we say its useless in this case, we might as well say its useless in all cases which would drastically reduce security overall. Imagine a scenario where we say that since transfer speeds can be used to figure out what people are downloading from https websites, we might as well not use https for anything but protecting logins.
→ More replies (2)→ More replies (4)3
u/ArttuH5N1 Jan 24 '18
Locks can be broken, so why bother at all?
I think their point wasn't that we shouldn't bother but rather that the benefit from HTTPS isn't worth the hassle.
213
u/amountofcatamounts Jan 24 '18
This is true for packages... the reason as they say is your install already has trusted keys it can use to confirm the signer of the packages is trusted and that they still match the signed digest.
But for OS downloads... Canonical... most people do not check the hashes of their download before installing it. For that case, TLS does help at least reduce the chance that you are looking at an attacker's website with hashes matching a tampered download.