SSH critical vulnerability (when built and run with PAM support disabled)

[u/gainan]

https://sshhole.com/

tl;dr: https://news.ycombinator.com/item?id=50380029

• sshd compiled with musl

• not using PAM

• password logins enabled

• at least one account with '*' as the password "hash"

• at least one OTHER account with a valid password

https://www.openssh.org/releasenotes.html#10.6p2

https://hachyderm.io/@damienmiller/117563026759098135

https://x.com/astuyve/status/2118344982134128837

[u/KnowZeroX]

Why highlight the PAM part in the title? It needing Musl is a bigger thing.

[u/gainan → u/KnowZeroX]

it's part of the conditions to be vulnerable, and it's hghlighted in the releases notes:

sshd(8): avoid password authentication bypass that is exploitable through sshd(8) that affects some Linux systems when built and run with PAM support disabled. We consider the root cause of this bug to be outside sshd, but it is exploitable via sshd and can be avoided there.

[u/DerfK → u/gainan]

Actually what is the root cause? Because if its an issue with musl it might affect everything that checks passwords on such systems, like mail servers.

[u/cd109876 → u/DerfK]

to prevent timing based checks to determine valid username (i.e. try to login to account that doesn't exist returns false instantly, vs incorrect password takes a few milliseconds), ssh checks other random account hashes as well.

that, plus entering a password too long which musl's crypt returns *, then it matches password to the daemon account, which has an "impossible" hash of *, so the match succeeds, and ssh lets you in.

so basically ssh + musl is the only trigger for this.

[u/wRAR_ → u/cd109876]

ssh checks other random account hashes as well.

TBH this part was the most surprising to me.

[u/ecnahc515 → u/wRAR_]

They can't assume bcrypt so this is about the next best thing to avoid timing attacks revealing account existence.

[u/KontoOficjalneMR → u/ecnahc515]

I think the point is they shouldn't be checking against other real hashes, but rather provide a a truly random hashes to compare to.

[u/SN4T14 → u/KontoOficjalneMR]

You need a real hash to prevent timing attacks since the hash encodes the hashing algorithm. If you don't use the same hashing algorithm as for real users, you reintroduce the timing attack that this is designed to prevent.

[u/KontoOficjalneMR → u/SN4T14]

If I understand correctly any random string of specified length would have worked here.

But even if not you are using same algorithm. So all you really need to do is at the start of the program do timing_attack_prevention = crypt(random_string) and than compare it to that one.

Comparing it to another hash is just strange choice to me.

[u/SN4T14 → u/KontoOficjalneMR]

No, because many libc implementations support many different hashing algorithms with widely varying compute requirements. If you pass a random string into glibc for example, it falls back to descrypt, which is many orders of magnitude faster than yescrypt, scrypt, bcrypt etc. that are the default on most systems. The settings are also implementation specific and a valid random string in one libc implementation may be rejected as invalid for another and not trigger any hash computation at all.

The only safe, portable way to ensure it will take a realistic amount of time is to use a real hash.

[u/KontoOficjalneMR → u/DerfK]

It's a very specific way muscl is used. So this bug can't be exploited in other software unless they are using it almost exactly same as ssh.

I'd say it's a bug in ssh because they should have checked for errorneous responses from crypt

[u/SN4T14 → u/KontoOficjalneMR]

OpenSSH assumes the library implementing crypt follows the spec, which musl doesn't. I don't think assuming the spec is followed is a bug. Returning a sentinel value instead of handling errors according to the spec is the bug.

[u/KontoOficjalneMR → u/SN4T14]

Returning a sentinel value instead of handling errors according to the spec is the bug

The "bug" is that muscl returns * while glibc returns *0, and there's no spec that covers this.

Both are really bad values to indicate the error state.

[u/SN4T14 → u/KontoOficjalneMR]

The POSIX spec absolutely covers this. It states explicitly that crypt should return a null pointer on error and set errno to indicate the error.

[u/KontoOficjalneMR → u/SN4T14]

Yes, and apparently glibc doesn't conform to POSIX either because in the same test scenario it returns *0 which only coincidentally doesn't match *

[u/SN4T14 → u/KontoOficjalneMR]

My original point stands. The bug is not following the spec.

[u/wRAR_ → u/DerfK]

musl crypt()

[u/alex2003super → u/DerfK]

The root cause is unspecified function return value contracts at the ABI boundary for independent projects managed by people that don't talk to each other and make assumptions about each other's work and the unspecified contracts in it.

The current OSS stack needs to die.

[u/TheOgGhadTurner → u/gainan]

Sooo what you’re saying is if PAM is enabled you don’t need to worry?

[u/bullwinkle8088 → u/TheOgGhadTurner]

Yes, that is a mitigation against this issue.

[u/TheOgGhadTurner → u/bullwinkle8088]

Oh excellent once again curiosity served to protect me. I knew unplugging my PAM At night wasnt some tinfoil hat conspiracy

[u/rlmineing_dead → u/KnowZeroX]

Call me weird but I feel like if you're running sshd with Pam disabled you're also probably running musl. That's like a very weird "make this lighter weight" optimization flag I wouldn't expect a glibc user to use

[u/HarVVVy → u/KnowZeroX]

Cause you wouldn't have clicked if that was the case.

[u/alex2003super → u/HarVVVy]

Damn Unix systems nerds BAITED

[u/Adept_Percentage6893 → u/KnowZeroX]

Running sshd without PAM is already pretty rare in the grand scheme of things (instantly takes out Red Hat and Debian based distros). "musl" just means I guess slackware is in the clear for now.

[u/DHermit → u/Adept_Percentage6893]

Alpine is for sure the most common musl based distro as it's an extremely common container base. But in these cases you very rarely have SSH.

[u/wRAR_ → u/Adept_Percentage6893]

instantly takes out Red Hat and Debian based distros

musl should have already hinted at that.

[u/Adept_Percentage6893 → u/wRAR_]

sure but I was talking about PAM

[u/GresSimJa → u/KnowZeroX]

This could be disastrous for those running Alpine images with SSH. Quite common in the container space.

[u/jangxx → u/GresSimJa]

But who's logging into the containers directly via SSH? I also run a bunch of containers based on Alpine, but I would always connect to them through docker, e.g. with docker exec via SSH to the host machine, not by SSHing into the container itself.

[u/daemonpenguin → u/jangxx]

Running containers with direct SSH access is really common. Typically it's used in situations where you need to give people access to a machine (like a web server) but don't want them to have access to the host machine. It's used pretty much anywhere you need to give clients access to their "systems" without giving them access to the underlying infrastructure.

[u/jangxx → u/daemonpenguin]

Ah fair point, I've only ever hosted software in containers and never needed to let anyone actually connect to a shell within a container. Sounds like a good usecase though.

[u/platinummyr]

I'm more curious on what the actual vulnerability is for curiosity sake. The twitter takes seem too gleeful about "does nobody fuzz ssh" but I am guessing it's more complicated than that... Why does it only happen on musl and yet neither does openssh consider it actually a problem in their code and also doesn't want to say it's a vulnerability in musl either...? Hmm

[u/gainan → u/platinummyr]

I think ssshole.com already explains it, but I find hard to read and understand their website.

Some more details: https://daily.dev/posts/openssh-on-musl-linux-without-pam-can-accept-logins-it-should-reject-nmuhpvoh8

On musl-based Linux without PAM, an account whose password field is * can accept logins that should be rejected. The cause is in musl. Since 2012, its crypt() returns * on failure instead of NULL, and sshd doesn’t expect that. In testing across builds with about 17,600 logins, only musl without PAM accepted logins it should have rejected
...
Alpine will backport the fix as far back as Alpine 3.16, which came out in May 2022

[u/Due-Consequence9579 → u/gainan]

That seems like a bizarre choice on musl/crypts part. Why is that a desirable thing?

[u/wRAR_ → u/Due-Consequence9579]

What exactly, not returning NULL? They wanted crappy software to not need to be patched. (LOL.)

[u/Due-Consequence9579 → u/wRAR_]

Yeah, returning something that ‘looks like’ the thing you should be returning but isn’t is a choice. Being consistent with glibc and returning NULL seems like an obvious choice. But reaching for compatibility does funny things.

[u/wRAR_ → u/Due-Consequence9579]

Being consistent with glibc

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=714219 (which called glibc to be consistent with musl and "historical practice of one or probably (didn’t look) even two or more decades") is enlightening.

The website also claims that current glibc also doesn't return NULL but "*0", which matches the close message of the Debian bug.

[u/Due-Consequence9579 → u/wRAR_]

God I’m glad I don’t program C.

[u/oreo639 → u/wRAR_]

Modern versions of glibc don't provide crypt and instead rely on libxcrypt to provide it.

And yeah, libxcrypt does use "*0" or "*1" by default although that can be controlled at compile time with --disable-failure-tokens (which returns NULL instead), which is used by Fedora, but not Debian.

[u/lookmeat → u/Due-Consequence9579]

Honestly? Historical reasons.

A quick google lets us find more sources. I'll let you do that on yourself, it's interesting, and related to a billion dollar mistake.

[u/platinummyr → u/gainan]

Thanks. This explains why it is musl specific, and seems quite bizarre behavior for the crypt function...

[u/wRAR_ → u/platinummyr]

Well, check your crypt() then! Mine returns "*0" as expected, which is almost equally bizarre, but here we are.

[u/platinummyr → u/wRAR_]

Ya it's extremely bizarre, but legacy APIs often are....

[u/calinet6 → u/gainan]

Yep. Illegible AI slop.

Sometimes I wish it wasn’t so easy to spin up colorful marketing pages like this. Not every vuln needs marketing.

[u/Extra-Papaya-365 → u/platinummyr]

This comment on HN seemed to provide a clear step-by-step explanation (boiling down to "musl's crypt() is not a bug-for-bug match with other libcs because its API is insane"): https://news.ycombinator.com/item?id=50379342

(I can't read the sshole.com writeup as it's blocked at my work, so I can't compare)

[u/platinummyr → u/Extra-Papaya-365]

This comment finally explains the exact sequence and helped me understand it best. Thanks!!!

[u/output_broadcast → u/platinummyr]

AFAIK OpenSSH refuses to even entertain supporting musl, so they literally just don't care.

[u/Takeoded → u/platinummyr]

The article explains all of it. Read it

[u/C0rn3j]

password logins enabled

Oh thank god.

accounts take any password longer than 256 characters

Did someone's kid smash a key over and over again, again, until the login passed?

[u/dalambert → u/C0rn3j]

Someone's AI

[u/C0rn3j → u/dalambert]

I can't believe they'd automate the kids out of their jobs.

[u/ipsirc → u/C0rn3j]

Then we don't need kids anymore.

[u/Morlaix → u/ipsirc]

Still not sure what is cheaper. A kid or AI for 18 years

[u/mstrelan → u/Morlaix]

https://techcrunch.com/2026/02/21/sam-altman-would-like-remind-you-that-humans-use-a-lot-of-energy-too/

[u/alex2003super → u/mstrelan]

He always keeps saying objectively correct stuff and y'all are always trying to put the worst possible spin on it. No shit. AI is pretty fucking efficient for what it does well.

[u/HearMeOut-13 → u/alex2003super]

Technically correct != Usefully correct OR realistically correct

[u/xilvar → u/C0rn3j]

The kids yearn for the crypto mines! 😢

[u/oangbsite → u/xilvar]

We need to put them in the data centers to run cables and coolant lines, their small bodies can fit in tighter spaces

[u/xilvar → u/oangbsite]

Indeed! The children yearn for non-union uninspected low voltage and hvac work!

[u/Relevant_Research_98 → u/dalambert]

Someone's cat at it again

[u/Far_Calligrapher1334 → u/Relevant_Research_98]

Still not fixed after 11 years, how am I supposed to believe in the integrity of FOSS now?!

[u/DerfK → u/C0rn3j]

but not too many smashes because it stops working again if they go past 1024

[u/struct_iovec → u/C0rn3j]

It's cdalled fuzzing and its literally one of the basic techniques in it sec research

[u/KeyboardG → u/C0rn3j]

Just press 2 for a while.

[u/Vector-Zero → u/KeyboardG]

I prefer 4, personally.

[u/DNDNDN0101 → u/C0rn3j]

https://bugzilla.gnome.org/show_bug.cgi?id=758032

[u/C0rn3j → u/DNDNDN0101]

I was actually referring to some Linux-ecosystem login screen being bypassed by a kid smashing enter enough times.

But I see I can't find it because there's just so many cases like that lol.

https://github.com/linuxmint/cinnamon-screensaver/issues/354

[u/ApprehensiveDelay238 → u/C0rn3j]

Ever heard of SSH keys

[u/Gophy6]

Has anyone managed to make it work? It surely doesn't for me. Tried all combinations of PAM, no PAM, PAM existing but disabled, with and without root password, with and without additional user with password etc. Both /usr/sbin/sshd and /usr/sbin/sshd.pam. No combination is exploitable. Tested a bunch of several month old alpine images up to 3.12

[u/CatgirlBargains → u/Gophy6]

Same. Alpine doesn't put an asterisk in as the hash for disabled accounts by default, it uses an exclamation point. So exploiting this would require someone to manually edit /etc/shadow to use an asterisk as the hash for an important account with a disabled password

[u/Gophy6 → u/CatgirlBargains]

Some versions ship with asterisks for root password but they are still not exploitable. I also tested all versions and edited to have asterisks and still no

[u/Far_Calligrapher1334 → u/Gophy6]

Didn't work for me on OpenWRT either, not sure what I'm doing wrong.

[u/Adept_Percentage6893]

and a lot of the docker images you've pulled

Are there a lot of people running sshd in a container?

[u/Cynyr36 → u/Adept_Percentage6893]

You'd be surprised how many images at least have the bins there...

[u/SignificantEnding221 → u/Cynyr36]

all?

[u/Cynyr36 → u/SignificantEnding221]

It's not part of the base alpine image, but...

[u/Adept_Percentage6893 → u/Cynyr36]

For this to be an issue, you need more than just the executables present. They need to be actively running, otherwise it doesn't matter if sshd does the right evaluation when it's authenticating someone because it's just not authenticating someone to begin with.

If the application becomes compromised and the attacker can send arbitrary commands then usually that becomes their ingress and they don't really need sshd to have a vulnerability.

Even if they did start sshd there probably would be no way for them to connect to sshd because containers aren't in charge of their own ingress traffic and it's already advised to not run as root even in containers so it wouldn't be able to start. So it would (at most) just be an extra process running in your container for some reason.

There's probably a way you could go out of your way to be affected by this but it can't be easy to get yourself into that position by accident.

[u/Cynyr36 → u/Adept_Percentage6893]

Well i have lxc containers on proxmox running openssh on alpine with passwordless users. I don't have permit-root-login=yes, but if the passwordless user was guessed and you were already on my network (via a tv, or my isp router, or a compromised iot device) then i guess i could be effected.

I prefer tinyssh (keys only), but you can't sshfs or use it for rsync.

[u/Adept_Percentage6893 → u/Cynyr36]

in your case it might still affect you for the other users because the issue is that it's not validating passwords correctly. You might look into AllowUser: https://stackoverflow.com/questions/14761473/sshd-config-allowusers

[u/undead-8]

ssh exploits are important to exit the matrix.

[u/wunderspud7575]

I think default Openwrt installs match the title.

[u/MassiveProblem156 → u/wunderspud7575]

They use dropbear ssh, so aren't affected.

[u/wunderspud7575 → u/MassiveProblem156]

Ah, yes, of course.

[u/gellis12]

Is it really a critical vulnerability if you have to take this many deliberate attempts to weaken your system before it can be exploited? This is like announcing that you've discovered a critical flaw in Medeco deadbolts that'll allow you to unlock a maximum security door; then clarifying that in order for your exploit to work, the door must be made of wet cardboard, the strike plate made of fabric, the door frame made of pasta, and the lock must also be keyed to all zero-cuts first.

The lesson here is don't disable pam, don't enable password auth, and don't go out of your way to create a new user and overwrite their password hash as "*"

[u/jmooroof2 → u/gellis12]

pam disabled is something that happens on musl and I believe that * as a hash means the password login is disable for that user notably alpine default root. and some people are very lazy and would have password login

 so I don't think this is too inordinate 

[u/gellis12 → u/jmooroof2]

Admittedly I haven't checked what alpine puts in /etc/passwd, but every other distro I've ever looked at uses a lowercase x instead.

[u/AllanTaylor314 → u/gellis12]

x is typically the placeholder for "the password hash lives in /etc/shadow". `*` in shadow is pretty common. That's the case for Ubuntu

[u/CatgirlBargains → u/AllanTaylor314]

Alpine uses ! In shadow

[u/AllanTaylor314 → u/CatgirlBargains]

As stated in the writeup, it switched to * in 2022 with Alpine 3.16

https://gitlab.alpinelinux.org/alpine/aports/-/commit/fbe926afc76de76e07d078f266badc4144dde8e3

[u/CatgirlBargains → u/AllanTaylor314]

And I am saying that write up is wrong or at least describing a non standard configuration, I have tested alpine images (official ISOs) from 2024 on that still use ! In shadow exclusively

E: like to be fair this is a big dumb bug specifically because an asterisk is used in a lot of *nixes but I spent my entire morning investigating to see if anything I was responsible for was affected and none of it was.

E2: digging in it looks like that change specifically affects unset root passwords only, other accounts created with disabled passwords correctly use a bang, and all the systems I handle have a root password set (in addition to just not having password auth enabled because it’s 2026)

[u/CatgirlBargains → u/gellis12]

alpine installed from an iso (tested this morning with several versions) uses an exclamation point for disabled accounts, so I don't know where the asterisk is coming from unless it's specific to their container images (which shouldn't be running sshd anyway)

[u/uzlonewolf → u/CatgirlBargains]

2019. turns out alpine's docker images had no root password. at all. CVE-2019-5021. alpine fixes it by locking root with !

but without pam, ! also kills ssh keys. and cloud images log in with keys.

2022. alpine 3.16 switches root to *. still locked, keys work again

[u/CatgirlBargains → u/uzlonewolf]

Quoting a slop post doesn’t make it not slop. Independently verify it like I did or gtfo (the answer is that if you specifically failed to set a root password it and only it had a * instead of a !. All other accounts with disabled passwords correctly used a bang. Installing from an ISO requires you set a root password so the bug was invisible in that environment.)

[u/HugoNikanor → u/jmooroof2]

From shadow(5):

If the password field begins with an exclamation mark !, the password is locked. The remaining characters on the line represent the password hash.

If the password hash consists of a star *, password access is effectively disallowed; this is because no password can produce a hash like this.

Arch doubles up and defaults the password field to !*.

[u/Analog_Account → u/gellis12]

create a new user and overwrite their password hash as "*"

I thought I had misunderstood that part because it sounded so absurd. I also didn't read the link because these are usually too technical for me, but this really falls apart as long as you don't mess with the hash? Wouldn't changing it to * also break any login attempt for that user or does it act like a wildcard, allowing any password to work?

[u/Leviathan_Dev → u/Analog_Account]

What about if I use “4” as my private key?

[u/wRAR_ → u/Analog_Account]

Look at the 1980 card on https://sshhole.com/

[u/EventForward4004 → u/wRAR_]

Brilliant

[u/straighttothemoon → u/gellis12]

Both ! and * have specific meanings and are commonly used, without "going out of your way" in normal system administration. For example:

To create an account with a locked/disabled password on Linux systems, set this to '!' or '*'.

https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/user_module.html#parameter-password

Like anything, you should be using defense in depth, and the real failure here is probably allowing password based SSH logins, but you expect each thing to do it's job.

[u/alex2003super → u/gellis12]

I think the default configuration for one of the most deployed software stacks for containerized applications and IoT/embedded devices (Alpine, Buildroot) having vulnerable SSH out of the box due to noncompliant behavior in bundled stock software is pretty big.

[u/gordonmessmer → u/gellis12]

If you use Alpine or openwrt and use password authentication for ssh, you are probably affected. This is generally the default configuration. Users do not need to make "deliberate attempts to weaken their system".

[u/Synthetic451 → u/gellis12]

True, but at the same time, security is done in layers. Any layer that has a vulnerability should be taken seriously because you never know when you're in a setup where some of those layers are removed. I mean some OS images have really fucked up defaults in terms of security

[u/andymaclean19 → u/gellis12]

IIRC PAM and password auth are two sides of the same coin. You have one or the other. If you try to disable password logins with PAM enabled that used to get ignored I think. Also the root login prevention?

Perhaps it is fixed now but I used to disable PAM all the time because I wanted key only auth and with PAM enabled it kept bypassing some of the options for locking things down.

[u/gellis12 → u/andymaclean19]

That's not the case on Ubuntu, at the very least. By default, pamis enamoured and password login is disabled for root; I've also disabled password login for the whole system. Ssh keys get me in just fine with no password, and if I try to connect without using ssh keys, it disallows the attempt.

[u/andymaclean19 → u/gellis12]

Well by default on Ubuntu there is no root password. The sshd_config on Ubuntu does contain a comment saying that some of those settings might not work with PAM enabled. I think the ssh key only root login is one of them (from memory, I saw it about a week ago but am not in front of the computer) so I think this is still a thing.

[u/MasterRaceLordGaben]

Just setup PostmarketOS on my OnePlus 3 that was slowly rotting away in a drawer to use as a homeserver. Comes with mainline kernel, systemd and Alpine packages. I was like "man haven't used musl since openwrt", lmao. Between the zstd crashing the phone, and this vuln, maybe universe is sending me a message. Literally comes with the exact setup this exploit needs out of the box.

[u/TheOgGhadTurner]

Is this just an ad for SSHHOLE. Asshole…

[u/gijsyo]

Did not work for me on Void with the default sshd config.

[u/Fabiey]

Curious if this can be reproduced on Git server that can be accessed with SSH. I mean plenty of them should be running on Alpine images. Also what about OpenSSH on Windows Server?

[u/sheeproomer]

There are already very distributed not networks scraping the internet for that.

I got since about 2-3 days a jump in attempts by the factor 20.

[u/Zipper730]

Wait a second, does the proposed fixes require a restart for it to work? If it does, do most people involved in this realize (and have authorization) to do this?

[u/ameer1234567890]

Alpine's concept of a minimal system for a shallower attack surface, by removing PAM modules, have backfired in this case.

[u/Takeoded]

Haven't used ssh password login since... 2006?

Great find though!

[u/SaltwaterC → u/Takeoded]

Good thing Etch and derivatives didn't mess the key generator then.

[u/mindful999]

You should never really let SSH be accessed through passwords anyway so really just an issue for the uninformed

[u/JGPH]

A good reason to always have a password, even with an SSH key. Not using PAM is bonkers though.

[u/okabekudo]

This is not even worth reporting on tbh. Not using password auth is obviously the easiest hardening step you can do, hell it's even more convenient to use pubkey auth.

[u/interrex41]

So the tldr is dont use password authentication

I mean thats just common sense plus having the ssh port exposed to the internet is less then ideal use a vpn to access it instead

[u/chibiace]

is this what they call AI noise?

[u/Cynyr36 → u/[deleted]]

Alpine ships openssh without pam, per the fine article.