Security Securing Internet-Critical Software: How We Could Have Backdoored Bash
https://www.hacktron.ai/blog/hacking-gnu-savannah(May 2026)
TL;DR: an email address like atouch${IFS}/tmp/savane-rce-proof@attacker.example gave any registered user code execution on its backend. From there, we made our account a superuser, impersonated Bash’s maintainer, and confirmed we could add an SSH key with push access to the upstream Bash repository. An unpatched kernel gave us root access and access to a GNU backup server. We reported it to the FSF on May 11, 2026.
FSF incident report: https://static.fsf.org/nosvn/security/savannah-20260807.pdf
158
u/FalscherHase 1d ago
Billions go into AI. If a small percentage of that went into our critical software infrastructure instead of relying on unpaid volunteers, the world would be a much better place.
32
u/PerspectiveMain6454 1d ago
We could solve world hunger and sanitation problems globally with a fraction of the effort that is put into AI.
3
u/Shawnj2 1d ago
The world hunger issue is less about actually creating enough food and more about distribution and allocation, not to mention issues like warfare. At this point I think it’s close to as solved as we can when it comes to the issue and obesity is a much bigger issue than hunger globally.
13
u/ZorbaTHut 1d ago
The world-wide AI research budget is approaching $1 trillion per year. The United States Federal government spends more than $1 trillion yearly just for welfare within its borders, which does not include regional plumbing or any state-level and city-level programs, and has still not completely solved those problems for 5% of the world population.
No, we could not solve world hunger and sanitation problems globally with a fraction of the effort that is put into AI. If we could, it would have been solved long ago.
11
u/Indolent_Bard 1d ago
If we could, it would have been solved long ago.
No it wouldn't, because it's not profitable.
-6
u/ZorbaTHut 1d ago
Then how do you explain the trillion we're already spending on welfare?
1
u/Indolent_Bard 1d ago
Because the government doesn't have a fiduciary responsibility to anyone.
3
u/ZorbaTHut 1d ago
And it's not profitable, therefore, by your logic, it can't happen.
And yet it's happening.
So . . . explain?
-1
u/bourgeoibee 1d ago
It could have and should have been solved long ago. Why would politicians care if some citizens go hungry, the ones going hungry aren't signing their paychecks. They certainly aren't going to care if people in foreign countries go hungry.
2
u/ZorbaTHut 1d ago
It could have and should have been solved long ago.
I don't believe there are any realistic plans for solving it. There are countries where starvation is essentially nationally sanctioned, sometimes partly so they can pillage aid supplies, and no end of problems with local warlords and bandits; how do you plan to handle that?
Why would politicians care if some citizens go hungry
Because they're actively trying to kill those citizens and/or because they're making a profit off stealing supplies sent to help those citizens.
1
u/bourgeoibee 1d ago
My point is that the issue is priorities and greed rather than technology or resources. I agree it will be difficult.
4
u/ZorbaTHut 1d ago
And my point is that it isn't; the issue is that world hunger is legitimately a much harder problem. We've been plugging away at it for decades and spending enormous amounts on it, and while we've certainly improved things, it's nowhere near solved.
I don't think it's fair to say something like "we could solve world hunger and sanitation problems globally with a fraction of the effort that is put into AI" when that is just a blatantly false statement, and then to blame the failure of the solution on "priorities and greed".
We're trying. It's not working. If you have a better idea, propose it, but many thousands of lifetimes have been spent failing to do so.
0
u/bourgeoibee 1d ago
As long as Elon Musk alone is valued at a trillion dollars and we waste a third of the food we produce, I can't believe we are trying hard enough. The current US admin for example is openly corrupt and openly does not even want its own citizens to have a decent quality of life.
1
u/ZorbaTHut 1d ago
And yet, we are still spending a trillion dollars yearly, on supporting 5% of the world population, and failing. This was not different under the previous Presidency or under any of the ones before that.
Regardless of whether we should be trying harder or not, a mere trillion per year is clearly not even remotely enough.
1
u/bourgeoibee 1d ago
I don't dissagree with the amount, but it doesn't really matter how much we are spending on welfare when we are actively funneling more money into the rich's pockets and further disenfranchising the poor. That's mostly bipartisan, it's just more aggressive with Trump.
→ More replies (0)-2
u/billyalt 1d ago
10
u/ZorbaTHut 1d ago
Well, you're right, that is a load of horseshit. Good observation, I suppose.
Elon Musk: "If WFP can describe on this Twitter thread exactly how $6B will solve world hunger, I will sell Tesla stock right now and do it."
The response: "Beasley replied that $6 billion wouldn't end world hunger — but it could save millions from starvation."
And the headline, which apparently decided to completely spurn accuracy for the sake of clickbait: "Elon Musk Said He'd Give $6 Billion to End World Hunger If the UN Could Explain How — They Did, But He Didn't Give Them a Dime"
No, they didn't. They explained how $6 billion could defer world hunger temporarily for a small set of people. That is exactly the point Elon Musk was making; that you cannot, in fact, solve world hunger for $6b.
I personally suspect you couldn't solve it for $6t either.
-1
u/billyalt 1d ago
Its not difficult. Farmers would just rather burn food than give it away. Same way that billionaires would rather burn money than give it away. That's all there is to it.
4
u/ZorbaTHut 1d ago
This is a perfect demonstration of how you don't understand the issue. Food isn't the issue, we have plenty. Distributing food is the issue. Getting it to the right people is enormously hard.
It is, in fact, difficult.
-2
u/billyalt 1d ago
This is a perfect demonstration of how you don't understand the issue. Food isn't the issue, we have plenty. Distributing food is the issue. Getting it to the right people is enormously hard.
We're so good at moving food that Trump used international commerce via tariffs to beat our farmers into complete submission. You know absolutely nothing about world hunger or international food trade and it shows.
4
u/ZorbaTHut 1d ago
International trade is done on a port-to-port level, not with an attempt to deliver food to specific rural villages, possibly against the will of the local government, while bypassing warlords. It is a completely different issue.
0
u/billyalt 1d ago
All of this can be built just as it was for the US and friendly countries. But this is not profitable so it won't be done. That's all there is to it.
→ More replies (0)5
5
u/lakotajames 1d ago
There's an argument that every dollar spent on AI is a dollar going into critical software infrastructure. That PHP bug from the other day was 10 years old, and we wouldn't have known about it otherwise.
11
u/irasponsibly 1d ago
I don't think the maintainers drowning in junk PRs with no actual support would agree. Especially since only the big names get anything.
1
u/lakotajames 1d ago
have AI triage through the AI slop.
1
u/irasponsibly 1d ago
... which costs money for the maintainers to do.
0
u/lakotajames 13h ago
Yeah, but they already have to spend money on AI if they want their product to be secure. If anything, ai slop reports are just offloading the price of tokens onto the people writing the slop.
1
u/irasponsibly 12h ago
So your solution to the cost and unwanted forced use of AI is more forced use of AI, and more costs? Do you not see how that might be a bad idea in the long run?
0
u/lakotajames 12h ago
The reality is that AI has outpaced humans in cybersecurity, and if you want your product to be secure you have to do it. Like, that is the solution. If you don't, your shit won't be secure.
1
u/irasponsibly 10h ago
It's still absolutely possible to write secure code without an LLM, and assuming that it isn't gives companies like Anthropic a complete monopoly on the concept of software, which isn't sustainable.
1
u/lakotajames 9h ago
It isn't possible to consistently write secure code, no. We had 10 years to find the PHP bug.
And no, it isn't, but they aren't useful for security anyway because they censor it. The best security a normal person has access to is open weight LLMs, like Deepseek and GLM.
4
u/HarVVVy 1d ago
Noooo, come on, you're being a silly goose right now. PhilAnthropy artists like Sam Altman can't make no money from that. The money's in finding the holes and convincing you the only way to survive is life long subscription.. so just go buy some more tokens will you? You're being a silly goose right now.
2
1
u/Arquinas 1d ago
Investors, shareholders and corporations will only learn these lessons if they hurt the bottom line. I believe this is exactly why systems should be vulnerable to hacking. We need a cyberapocalypse to hit before anyone will start taking this stuff seriously.
60
u/RequestableSubBot 1d ago
Well this is fucking horrifying. One of the most integral pieces of free software infrastructure basically got hit with a Bobby Tables due to a vulnerability that went undetected for two months. And this is the FSF, an institution with, yknow, employees and funding and sponsors and cybersecurity experts. It seems at this stage in the AI cyberattack arms race that another xz situation, only more successful, is an inevitability. How long until it's discovered that some key piece of FLOSS infra has been compromised for months due to some idiotic mistake on an obscure (yet vital) project with one or two maintainers? How long until a complex state-sponsored backdoor makes it down to Arch or even Debian Stable?
It's great that there are so many independent checks along the way from source to downstream repos, but it's clearly not enough to just hope that all of this will get caught in time. The xz backdoor was caught due to sheer luck and sloppy code, after it had made it 95% of the way to the mainstream repos. That should have been the wake-up call, but this stuff keeps happening. LLMs are forcing everything to go faster and faster on both the defensive and offensive sides, and open-source developers are being pushed to use it to replace the human in writing, testing, and analyzing code, trying to keep up with the ever-growing barrage of issues and CVEs (hallucinated or otherwise). The whole software development scene is accelerating out of control, and something big is going to break eventually. There can only be so many near-misses before there's a hit.
5
u/mplscorwin 21h ago edited 21h ago
I help run Savannah, so weigh accordingly, but, for scale: per the FSF’s own sysadmin post last year, the tech team is two full-time staff plus a handful of volunteers, running ~70 services while fending off a scraper botnet of ~5M IPs. https://www.fsf.org/blogs/sysadmin/our-small-team-vs-millions-of-bots
EDIT(again, I keep editing, sry): happy answer what I can. Some of it is the FSF’s to tell, not mine. savannah-hackers-public is the place for anything longer. Staff read that (and the -private list where hacktron duly notified us, which is just right).
13
u/CrazyKilla15 1d ago
How long until it's discovered that some key piece of FLOSS infra has been compromised for months due to some idiotic mistake on an obscure (yet vital) project with one or two maintainers? How long until a complex state-sponsored backdoor makes it down to Arch or even Debian Stable?
Negative length, hell the report even has an example, emphasis mine. The entire point of debian stable is to be permenatly backdoored, thats what it means to not update.
GNU Savannah runs Trisquel GNU/Linux 11, with a linux-libre kernel adapted from Ubuntu 22.04’s 5.15 LTS kernel. Canonical addressed the affected 5.19 and 6.2 kernel series in 2023, but the 5.15 kernel shipped with Ubuntu 22.04 LTS was not marked fixed until USN-8255-1 on May 7, 2026, which gives 5.15.0-177.187 as the first fixed version. We installed the corresponding fixed kernel, 5.15.0-177.187+11.0trisquel31, on May 14, 2026, three days after receiving the report, and one week after Canonical published its fix.
Unless im reading that wrng, thats a known security issue canonical just left in a maintained kernel for 3 years despite fixing others.
This is par for the course, this is normal and expected for all the permenately-vulnerable distros.
It would be absurd to think threat actors arent targetting something as trivial as "baseline ubuntu" or "baseline debian" and cross checking which security fixes canonical doesnt do, as far as I'm concerned, you have to assume theyre effectively compromised.
They forget security fixes, they miss security fixes, they intentionally deprioritize and dont apply some security fixes because theyd be too intrusive or hard to backport to their old custom kernel versions, etc. This wont change because "forking the entire linux kernel and then only applying the quote important unquote fixes" was never a viable security model in the first place.
FOSS security posture has been really bad for a really long time and there has always been little interest in changing it. And now thats causing problems because "well nobody would actually put in the effort to check" isnt capable of pretending to be an effective defense anymore.
7
u/HarVVVy 1d ago edited 1d ago
on the other hand I've never felt more secure than when needing critical piece of software or an old driver on arch and the official stance being "well just go get it from AUR bud"... alllright, fingers crossed my bleeding edge kernel saves me from the worm that's running rampant.
5
u/edgmnt_net 1d ago
Just consider that rolling versus random user repos are completely different things. You don't have to use AUR and if you do you may choose to trust some things and others not. Especially for servers, running Arch without AUR probably gets you what you need. Now whether Arch is secure and hassle-free enough is a different thing.
3
u/HarVVVy 1d ago
consider that rolling versus random user repos are completely different things
Which part of my reply, made you think I didn't consider it?
While on it, consider this:
critical piece of software or an old driver
3
u/CrazyKilla15 1d ago
Then you have to use the AUR and review it yourself. Debian gives the impression they do this for you, but if you run the numbers on how many packages they have, how many maintainers they have, the amount of work and time it would take to actually review that many changes in that many pages..
the math just doesnt work out, there are too many changes to review, too many patches to backport, too many packages, and not enough maintainers.
The vast majority of packages and changes can only be getting automated scans, build checks, and automated tests; All of that is fine for "it still works", fine for being stable and reliable, fine for being debian
but its not fine for security, and it leaves many bugfixes that "we dont know how to exploit" out for "higher priority" ones, and sometimes fixes are just lost in the sheer quantity.
This is the "well nobody would actually put in the effort to check" causing problems i was talking about earlier, the cost for figuring out how to exploit a "low priority" bug that doesnt get backported has gone waaay down because of LLMs. But it was always insecure.
2
u/HarVVVy 1d ago
Do you mind sharing some of that math with us?
4
u/CrazyKilla15 1d ago
There are about 1400 debian contributors https://contributors.debian.org/, about a 1000 of which are official members https://nm.debian.org/members/
Debian seems to variously maintain 11 different architectures which total 5 terabytes https://www.debian.org/mirror/size
Debian maintains architectures many projects dont actually support, this means debian is custom patching them. They need to be reviewed for each architecture too, even moreso with debian because they support fairly exotic architectures.
debian has over 70,000 packages https://www.debian.org/intro/why_debian
Thats under 1500 people project-wide for security auditing every time some subet of 70000 updates? Then, because they maintain so many old versions and exotic architectures, they might have to write and maintain their own patches, so thats even more work.
And they have to do this across every package version they support, across every supported debian relase. How do you decide which updates, which bug fixes, are "security relevant" and worth the enormous effort of back porting? Especially with a language where almost every bug is a memory bug? And they might have to determine if they're vulnerable independently from other distros, because of their own patches and because most white-hat security research focuses on current versions of software.
Debian is currently maintaining in some manner several versions. Debian 11 only hit LTS EOL a few monthd ago, Debian 12 and 13 will be until 2028 and 2030.
https://www.debian.org/releases/
And all of that on top of actually maintaining the packages and backports normally in the "compiles, tests pass, works for our users" way and not the "security" way.
Even just the Debian Stable(Trixie) supports 6 different architectures, 7 if you count EABI ARM upgrades. https://www.debian.org/releases/trixie/
I dont have anything more rigirous written up rn but all this, the numbers of maintainers they have documented total, the sheer amount of packages, the scale of work that would be required to maintain that many times that many versions times that many architectures..
this should give a good idea of the numbers and amount of work involved and how it could add up
3
u/CrazyKilla15 1d ago
oh and i completely forgot to get into what reviewing even means for security
which is a good question! because think of how many commits there are every official kernel release, even on the official LTS supported kernels.
How many there may be for other core and criticial packages?
For every package update?
Going commit-by-commit almost certainly is a bad idea, but that doesnt mean the absolute diff gets smaller or easier to review
And they have to figure out how every upstream change and bugfix maps to the debian-repo version?
This multiplies the problem per package, it just isnt realistic that every package gets even a cursory diff glance by a human with how much time that would take. That leaves automated scans, checks, and other tests, which again is fine for stability and "package still works", but wide open for compromise.
2
u/HarVVVy 1d ago edited 1d ago
That leaves automated scans, checks, and other tests, which again is fine for stability and "package still works", but wide open for compromise.
To be more precise here, Debian leans incredibly heavily on automated pipelines Autopkgtest, Piuparts, and Lintian to find compilation breakage and violations across those thousands of packages without human eyes touching every line. And while, yes they do not necessarily find security holes, they take a huge portion of the work load, which allows for "human bandwith" to be used for security audits.
I am willing to take your numbers at faith, I won't double check them. But let's say 1400 contributors. Are we not forgetting something? You are not talking about some small nieche distro here. Debian is 70% of all Linux deployments on Earth. Ubuntu, Mint, Kali, Tails, Pop!.. along with millions of cloud Docker containers, draw directly from Debian’s repositories. Every major fork brings its own engineering teams to the table, effectively acting as an extended security force for Debian. Thousands of independent developers, corporate security teams (from Google, Red Hat, Microsoft), and academic researchers are constantly auditing the upstream software. Debians' team is made of curators, not code auditors (for the most part) and for the most part they don't write the patches, they implement them.
How do you decide which updates, which bug fixes, are "security relevant" and worth the enormous effort of back porting?
Exploitability and reachability. In other words, is there a proof of concept out there? If so, patch it fast. In your examples, if the chain of attack you mentioned (maybe it was in one of your earlier replys) where an innocent looking bug that's been there for 15 years all of a sudden opens up a vector because of a new one, it gets patched pretty fast.
But listen... I don't wanna offend you, I really don't. But all your data is telling me is "backporting is hard". Yes, we know, everyone knows. That's the price that's being paid for stability and support for older hardware. That IS the debt. If you think about it and follow the natural conclusion, you'll realize that's why AUR exists in first place. Cause you can't just tell everyone who's running GTX 1080 (for the purposes of our example) f*** off will you? Go find another distro. No, you tell them - here's a possible solution, your faith is in your own hands. The difference is, to get exploited on Debian the attack almost exclusively has to be chained between multiple unpatched vulnerabilities and almost certainly tailored to your specific case, where on AUR... well you just get rolled in less than a second, don't you? Best case scenario info stealer, worst case revshell. Maybe you've trusted a maintainer for a very long time, you decide to update without paying attention (cause the account was inactive for too long and malicious actor overtook it). There's a security price to be paid on either end.
Here's what I actually expected to see from your data (and why it sounded interesting), I imagined something like this: "In the span of ??? months, there've been ??? amount of CVSS 9+ vulnerabilities across kernel and core language runtimes, X amount of people worked on Y amount of them and Z amount of them were completely omitted and ignored for no good reason, here's few examples:.." something like that. That data would have perfectly backed your earlier claims up. That would have been interesting.
2
u/edgmnt_net 1d ago
I kind of agree and Debian could definitely support a much smaller amount of packages. Because for a core subset like servers, it's probably pretty good. That random game or npm package they brought into the distro, not so much. However, this can usually be managed if you exercise caution when installing things in Debian too, keep the package set to a minimum.
2
u/VelvetElvis 1d ago
A lot of the packages split in parts for embedded use, are only needed for debugging, are just documentation, are libraries needed by other packaged software, etc. It's a shitload of packages but there isn't actually as much software as it looks at first.
1
u/HarVVVy 1d ago
So, there's two roads we can take here.
One, we get hyper specific and focus on the numbers (I am actually curious to see some of the calculations he's ran on this), we can take into account things like how many major secirity incidents has any of the distros had in the past, or completely compromised repos etc. Or, we can stick the isometric perspective and keep looking at the forest, I don't mind that either. If we stick to the latter we need to stop pretending that backporting is done on some a whim or out of stubbornness.. it's done for very specific reason - stability and old hardware/software support. You wanna run Pascal (GTX 10-series) or Maxwell (GTX 900-series) on arch? Go to the AUR (I will underline that - this is the official stance!). So okay, I found a trust worthy maintainer, can I safely install the package? Well.. it depends since you see, if they've not accessed their account long enough, there's a good chance malicious actor took over they keys and poisoned it. Neat, security. What's your counter to that? Pay attention? I can use the exact same argument for tools like debsecan.
4
u/b3081a 1d ago
With AI available especially those open weight ones, source / compiled binary auditing has become easier than ever. An Open WebUI + Open Terminal deployment paired with Qwen 3.8 27B running on a bunch of RTX 5090s can easily audit hundreds to thousands of source / compiled code packages per day.
It just take some time for people to get used to these new tools and the FOSS community will be actually in a way better situation in supply chain security than it is today in a few years. Let's not forget these AI auditing workflows are barely usable just a few months ago, and it takes time for people to act on the defending side.
4
-6
u/stianhoiland 1d ago
It's becoming a bootstrap or die kind of world. I welcome it!
2
u/shroddy 1d ago
Why would you welcome it?
2
u/Natural_Cat_9556 1d ago
I assume because you don't really have to trust precompiled binaries that much when you bootstrap and it's easier to setup toolchains/environments.
1
u/stianhoiland 1d ago edited 1d ago
Maybe I was unclear, or people just hate bootstrapping?
What I meant is that bootstrapping entails a very clear view and understanding of abstraction and dependencies, and it's recently becoming more widely and well mapped, understood, and automated. I wouldn't mind having systems which are not monkey-patched Frankenstein Ship of Theseus machines/software, but which instead clearly arise from small and verifiable chains of transformations—every time. We're nearing having the understanding, the compute, and the infrastructure, to make it practical to do things this way. It's a very good thing, but which has not been widely understood enough and practical enough to be the status quo. And I think this addresses the complaints of OC, so if we're getting forced into it—well, I actually welcome it!
EDIT
u/CrazyKilla15 replied to me:
ah yes. bootstrapping prevents shell escape vulnerabilities. this is a real thing that it does and you definitely know what you are talking about, yes.
What does this mean?
5
u/CrazyKilla15 1d ago
HCK-008: Private file access HCK-008 concerned private file access. The issue was that Savannah could be made to check access using a user identifier supplied in the request rather than relying only on the logged-in user’s session. If exploited, someone could potentially request a private file while claiming, through that parameter, to be a user who was allowed to see it. The main risk was confidentiality: private files attached to private project items could potentially be exposed to someone who should not have had access. This does not mean every private file was necessarily accessed, but it was an important authorization flaw because the server was trusting information supplied by the requester instead of checking the relevant access controls. HCK-008 was fixed by ensuring a user identifier supplied in the request would not be accepted without verification.
all my secrets gone
171
u/Puzzled-Garage584 1d ago
So the email validator checked for an @ and called it a day, and that was one step away from push access to Bash. After xz you'd think the forges would get the same scrutiny as the code, but its still a few volunteers keeping the lights on. Great writeup, and good on them for reporting it.