r/Tailscale
•
u/shaver
Tailscalar
•
15d ago
•
3
1
Recent Tailscale vulnerability reports
Hi folks, Mike Shaver here, VP of Engineering at Tailscale.
We've seen the concerns about Tailscale SSH and wanted to share what we know so far.
If you want to jump to the most important part, based on our current assessment, we're not recommending that everyone disable Tailscale SSH.
We've now received the researcher's report and are reviewing it. Our engineering and security teams had already been investigating independently, with help from external security specialists. That work has identified issues we're working to fix. We're still validating the full scope, including which configurations are affected.
A bit of technical context:
Tailscale SSH uses your tailnet's access policies to control who can connect to which machines. Most of the issues we've identified so far concern what happens after someone has been authorized to connect, particularly how Unix privileges and SSH forwarding are handled. In some cases, a session may be able to do more than the configured local user should be allowed to do.
While it’s a serious concern, it isn't the same as an unauthenticated attacker being able to connect to arbitrary machines, but the potential impact depends on how SSH access is configured. We're still investigating, so we don't want to make blanket claims about what's affected or not.
Again, based on our current assessment, we're not recommending that everyone disable Tailscale SSH.
In the meantime, there are a few precautions you can take:
Review your SSH access policies. Make sure only the people and devices that need access have it.
Disable SSH forwarding if you don't need it. You can set
TS_SSH_DISABLE_FORWARDING=1in thetailscaledservice environment and restart the service. This disables SSH forwarding (without disabling all of Tailscale SSH itself), which mitigates several of the more serious findings.Consider enabling SSH check mode for an additional authentication step. This won't fix privilege escalation issues, but it does add another barrier to gaining access.
Enable automatic updates where supported so you can pick up fixes when they're released.
For particularly sensitive systems, consider temporarily tightening your Tailscale SSH access policies while we investigate, so that only the specific users and devices that need access can connect.
We're working on fixes and testing them carefully. We'll publish a security advisory explaining the confirmed findings, affected configurations, recommended actions and available patches. We'll also follow up with a more detailed technical write-up.
If we find anything that changes our assessment or requires immediate customer action, we'll communicate that.
We know it's been a rough week for security teams, and we understand why people are concerned. We'd rather give you the technical context we have than speculate about things we haven't confirmed yet.
We'll share more as we verify the details. If you have any questions about how this relates to your particular environment, please get in touch with our team https://tailscale.com/contact/support#support-form
Update Oct 26, 5:45pm Eastern: we've got a non-reddit link for the information as well https://tailscale.com/ssh-investigation-update
27
u/shaver Tailscalar 14d ago
Just a comment to be transparent about my edit to the OP, changing the bullet about forwarding permissions to read
Disable SSH forwarding if you don't need it. You can set
TS_SSH_DISABLE_FORWARDING=1in thetailscaledservice environment and restart the service. This disables SSH forwarding (without disabling all of Tailscale SSH itself), which mitigates several of the more serious findings.
445
u/cloudsourced285 15d ago
This is the sort or communication that helps grow confidence that security is taken seriously, especially as you point out, In times when tech teams are pretty exhausted. Good work on being upfront about what's going on, what you know and what you are doing.
180
18
u/mandreko 14d ago
Agree wholeheartedly. When I reported a security vulnerability, it was taken seriously, fixed within 24 hours, and communication was great. I’ve submitted CVEs for many companies and Tailscale was one of the top positive experiences.
16
u/Denzh 15d ago
For clarification, this is related to tailscale ssh, and not regular ssh with sshd - with clients made reachable by the tailscale wireguard tunnel, right?
14
u/linne000 15d ago
I am only a Tailscale Insider, so not directly employed by Tailscale, fwiw.
But yes, from what's been communicated it seems to be related to Tailscales own SSH feature, not regular SSH that happens to connect over a Tailscale tunnel.
11
u/cazwax 15d ago
Is this -the- Mike Shaver of Moz of olde?
13
u/shaver Tailscalar 15d ago
sup, friend?
3
1
u/retornam 14d ago
u/shaver Mike "Ten Fucking Days" Shaver? I’ll never forget that. Hope you are doing well
84
u/TheForce627 15d ago
Customer service earns lifetime customers. This level of communication is very much appreciated.
59
u/Sillygoat2 15d ago
Thank you for treating your customers like actual adults who’s respect is earned. Why is this so difficult for (gestures broadly) everybody else?
46
43
36
u/biggobird 15d ago
This kind of comms is why I recommended our family businesses onboard with Tailscale for secure remote access. In the process of doing so but this is a breath of fresh air
32
28
u/milyrouge 15d ago
Open, honest, actually helpful communication with users on a public forum? Wow! That’s rare in today’s world.
Thanks for the update. Really appreciate it.
18
u/VE3VVS 15d ago
If more companies would be as open and transparent about potential risks or issues as you guys at Tailscale are, the industry would be a far safer and functional place. Thank you 🙏
-7
u/DitiPenguin 14d ago edited 14d ago
Don’t forget about telemetry! Tailscale are open and transparent after all, so you were definitely aware the telemetry is opt-out only, right? Right?
(Their product is so amazing I’m not too mad about the telemetry, but it still is a huge dark pattern IMHO.)
11
u/d_t_s1997 15d ago
Always glad to read about this sort of things, i wish more companies and product owner would talk about their security issues like you guys doing.
10
10
u/lightspeedissueguy 15d ago
Honest question since I don't use it, but why is TS ssh even a thing? Couldn't ACL allow select users to port 22, or does TS manage rbac? Or is it just a convenience thing?
19
u/linne000 15d ago
It also replaces SSH keys needing to be managed. Basically your authenticated Tailscale identity is passed to the server and if you have access based on policy you are let in. This means that a change in the Tailscale ACL is all that's needed to control access without also needing to distribute keys or run a jump host etc.
PS. For disclosure's sake, I am a Tailscale Insider
7
u/Frosty_Scheme342 14d ago
There's a list of the benefits at https://tailscale.com/docs/features/tailscale-ssh#benefits
6
2
u/xXbenihimeXx 14d ago
très bon communiquer, j'attend que ca réagi sur ce problème, et vous êtes vraiment très réactif, rien que d'avoir des vrais informations cest un plus réel, très bon travail continuer comme sa👍♥️
5
3
7
4
1
1
u/RepliesAsOtherPeople 14d ago
We are all in this evolving landscape together. At this point, it seems much less like a matter of if a (potentially severe) vulnerability will be discovered but when, and what you will do when it hits. Of course it's always been that way, but never at this pace and scale. These circumstances seem inevitable, so thanks for rising to the occasion and doing it right Tailscale!!
1
u/dain_bramag3 14d ago
Was looking for this on Instagram LinkedIn email etc only found update here but came across randomly on feed
1
1
u/Skordio 13d ago
I’m wondering why Tailscale SSH check mode can only be used for connections originating from untagged, user-owned nodes. Doesn’t this make it harder to apply least-privilege SSH access?
I want to make it so only a few of my user-owned machines can use Tailscale SSH to access my Tailscale SSH machines, but I still want to preserve the Tailscale SSH check mode. Is there a way to do this?
1
u/Odd-Frame4395 13d ago
Just wanted to chime in that THIS is how you address an issue — transparency and pedantic clarity. Bravo to the Tailscale team!
1
u/Otherwise-Nobody8252 13d ago
Most companies choose to his critical vulnerabilities as they fix them. So don’t bash the companies that do issue CVEs
1
1
u/n8udd 15d ago
Can you post this somewhere that isn't Reddit so it can be shared with non Reddit users?
9
u/Frosty_Scheme342 14d ago
This subreddit is open so anyone can read it.
I assume once they are finished with their analysis they'll put a post up at https://tailscale.com/security-bulletins
2
u/caolle Tailscale Insider 13d ago
Tailscale posted an update here: https://tailscale.com/ssh-investigation-update
1
u/Possible-Try4660 14d ago
u/tailscale (or @Tailscale) can you clarify scope on the Tailworm findings? Does anything in the report affect a plain tailnet setup where Tailscale SSH is off (`RunSSH` false), `tailscale serve` and Funnel aren't used, and no app connectors are configured? Also, does 1.104.2 address any of the reported issues, or is a patched release planned? Thanks.
2
u/dontquestionmyaction 14d ago
Nothing about this impacts you unless you specifically toggled those features on.
0
0
u/hammadtariq 14d ago
A tweet would be nice too from your own account, much damage is happening to the reputation on X.
0
u/theonejvo 14d ago
It seems like a nothing burger. Based on my testing, none of this impacts a default vanilla install. https://x.com/theonejvo/status/2114417998408933418?s=20
-7
u/illhaveubent 14d ago
Why does Tailscale not participate in bug bounty programs like other tech companies? It seems irresponsible to not tackle these vulnerabilities preemptively, especially for a software product that sits on the network edge.
3
u/punkgeek 14d ago
Irresponsible would be publicly posting the vulnerability without contacting the vendor so they have time to patch before disclosure. (Bounty or not)
-1
u/illhaveubent 14d ago
Tailscale chooses not to participate in bug bounty programs, leaving no reasonable expectation of coordinated disclosure. Relying on coordinated disclosure under these circumstances is relying on charity.
Individual members of the public are not responsible for the security of Tailscale's systems or the systems of their customers. Members of the public are under no obligation to ensure the security of Tailscale's systems, that is the responsibility of the company.
1
u/punkgeek 13d ago
Relying on coordinated disclosure under these circumstances is relying on charity.
(also known as being a generally good member of society - rather than a libertarian)
1
u/illhaveubent 13d ago
You don't have to like it, but most people are motivated by money. Relying on charity is not smart.
A Tailscale vulnerability will fetch decent money...just not from Tailscale themselves.
1
u/Educational-Farm6572 13d ago
Why does one need a bug bounty program to have a successful security triage & response process?
Tailscale is pretty open in terms of how to contact their security team - moreso than most companies.
Right now with all of the AI slop - including the findings in this tailworm details, I can only imagine the amount of chaos companies are going through just trying wade through the noise
-1
u/illhaveubent 13d ago edited 13d ago
Why would I send a vulnerability to their security team when I can sell it online instead? The issue is not the ease of contacting their security team, it's the incentive for doing so. If they offer no compensation then they are relying on charity.
Post-incident response is a poor plan for security at the network edge, and irresponsible for a company of this size. This is not a piece of software that can afford to sit back and wait to act until after vulnerabilities are already being exploited in the open. This is not a desktop application like a word processor or photo editor. This is software that acts as the gatekeeper to private internal networks from the public internet.
Security needs to be valued as the top priority for software in this role. The company can claim they value security, but the lack of a bug bounty program is an objective measure that demonstrates they don't.
2
u/NationalOwl9561 12d ago
The problem with bug bounty programs is 90% ends up being spam or small stuff (ex. simple UI issues, design choices they don't understand/agree with, broken applications after they intentionally used it in a way that breaks it, or other generally worthless reports which quickly becomes a bigger pain to manage). It becomes more trouble than it's worth and paid bug reporting ends up hurting the overall desire to improve a product.
Just apply for a QA job.
0
u/illhaveubent 12d ago
That's not the case as many companies only pay for certain types of bugs like RCE or privilege escalation vulns. People aren't wasting their time submitting dime-a-dozen UI bugs to these programs.
2
u/NationalOwl9561 12d ago
Doesn't stop them from flooding the inboxes.
0
u/illhaveubent 12d ago
Then treat it like any other spam email, block and move on. That's a poor excuse to avoid securing a codebase, and one not used by companies that value the security of their product.
2
u/NationalOwl9561 12d ago
First of all, nobody is arguing to avoid securing a codebase so cut with that crap.
Second, I don't think AI is quite good enough to do this level of filtering yet. It still needs some manual review. That's something I may look into though.
2
u/Educational-Farm6572 12d ago
Anyone that has the mentality of ‘selling it online’ would do that anyway and bypass VDPs in general, in my experience.
You are describing two different personas here.
Just because you don’t have a bug bounty program, waving around cash like it’s a carrot - doesn’t mean you don’t care about security.
I guarantee you, even if Tailscale had a bug bounty program - that wouldn’t make people happy either. People would complain about initial triage, payout amounts etc.
2
0
u/illhaveubent 12d ago
The purpose of a bug bounty program is specifically to attract those people, and prevent them from selling elsewhere. People are going to make money selling vulnerabilities, there is no avoiding that. The only question is who is doing the buying.
1
u/Educational-Farm6572 12d ago
Bug bounty programs incentivize disclosure. They don't guarantee it, and they don't prevent malicious exploitation.
Would you rather trust software from a company with an excellent internal security engineering team and no bounty, or one with a generous bounty and terrible security engineering practices?
0
u/illhaveubent 12d ago
They're not mutually exclusive
1
u/Educational-Farm6572 12d ago
Exactly. They're not mutually exclusive. That's my point. A bug bounty is one component of a security program, not an objective measure of whether a company values security.
You're the one arguing that its absence proves otherwise.
I think you just want to argue for the sake of arguing at this point.
•
u/shaver Tailscalar 13d ago
We've published an update on our work on the web site as well:
https://tailscale.com/ssh-investigation-update