Remix.run Logo
▲ Tailscale SSH: Investigation and remediation update(tailscale.com)
103 points by figmert 2 hours ago | 54 comments
▲minimaltom 14 minutes ago | parent | next [-]

Former tailscalar here and now someone managing tailscale at an org.

Having done my own review of the vulns and the fixes as they are landing in github, its pretty clear to me that this only affects situations where you use ACLs to permit ssh into a node (via tailscale ssh, not via opensshd), but dont give a path to sudoers/root.

If a user had no SSH access and/or connectivity to a vulnerable node, none of these vulns give them access.

However, if you have a vulnerable tailscaled and you give ssh access as some non-root account/privs, then yes, these vulns are relevant as theres now known privilege escalations that give access beyond some specific user account.

Not trying to play down the vuln (its important), but its not a free-for-all RCE that some ppl are acting like it is.

▲rietta 15 minutes ago | parent | prev | next [-]

I am fairly glad I have stuck with stock OpenSSH that ships with Linux. This incident proves I am glad with that choice.

Over the decades, I have maintained remote access through port forwarding on my router - at one point in 2008 I put SSH on port 80 so I could access it from the restricted wifi network at the hospital my grandmother was at. I have also used openvpn. I have also used cloudflare tunnels. But in the end, I always connect to hardened openssh with key only authentication.

▲hollow-moe 10 minutes ago | parent | prev | next [-]

Recently NetBird also had a huge privesc through their SSH component too... I'm using both at $DAYJOB and my personal infra, damn you guys ought to be more careful with such a sensible and important component...

▲wrxd an hour ago | parent | prev | next [-]

I like Tailascale as a product but I decided to move my homelab to plain WireGuard.

I wasn't using any of the additional features, I can get away with exposing a port on my router and I am changing devices rarely.

News like this make me happy I removed a lot of complexity and attack surface

▲femiagbabiaka an hour ago | parent | next [-]

> I can get away with exposing a port on my router

using wireguard is awesome, but I wouldn't say deploying and maintaining it lacks complexity exactly. and you didn't get rid of attack surface, you just made it your responsibility ;)

▲ectospheno 33 minutes ago | parent | next [-]

Always is your responsibility even with tailscale. Now it is his responsibility without tailscale.

▲LoganDark 43 minutes ago | parent | prev | next [-]

Attack surface almost entirely goes away if you just require a good port knock first! I do that with my protection SSH servers and it works just fine. Also prevents anyone from knowing there even is an SSH server there. Or any server if it doesn't respond to pings either, and only uses reverse tunnels to accept public connections

▲neilalexander 21 minutes ago | parent | next [-]

Wireguard doesn’t respond to malformed packets or handshakes from unknown devices by design.

▲LoganDark 9 minutes ago | parent [-]

Oh, awesome!

▲mbreese 38 minutes ago | parent | prev | next [-]

Is there a way to automate port knocks in your .ssh/config? I think that’s the main reason why I haven’t added it to my home server.

▲LoganDark 23 minutes ago | parent [-]

Technically you could use a ProxyCommand that does the knock and then runs nc to your target.

▲lakeviewterrace 37 minutes ago | parent | prev [-]

Port knocking is idiotic and amounts to Biff Tannen's "only I know how to start this car".

▲Gigachad 24 minutes ago | parent [-]

Dunno, in combination with having an up to date system following best practices, port knocking is probably going to save you when bots start scanning the internet with the next SSH 0day. Any amount of friction will cause the bots to move to the next IP address.

▲icedchai 18 minutes ago | parent | next [-]

Just moving SSH off a default port gets you 98% there.

▲LoganDark 16 minutes ago | parent [-]

An attacker only has to get lucky once, so while I do put SSH on a non-default port too, it's not enough to avoid attacks on its own.

▲LoganDark 17 minutes ago | parent | prev | next [-]

It can also save you if, theoretically, there's ever a public exploit going around that can achieve RCE on nearly all versions of SSH, and some moron decides that nearly a day of embargo on the patch is more important. And said moron seemingly wasn't even trying to make sure Linux distros had security updates ready either, they were just sitting on it while the exploit was at large. This is apparently a risk now.

▲esseph 20 minutes ago | parent | prev [-]

LLMs port knock for redteam scans. Have for a long time.

▲LoganDark 7 minutes ago | parent [-]

My knock for SSH was 20 ports within 2 seconds. knockd resets the sequence if you hit any of its ports out of order.

▲swingandamiss an hour ago | parent | prev [-]

With AI, the complexity doesn't really matter anymore. Codex or Claude can easily configure complex site-to-site setups and peer configurations on mobile devices. I did this by hand before and it was hard and difficult to get right. Now I just tell AI and it does so perfectly, checks and verifies things. AI made life so easy.

▲drum55 39 minutes ago | parent | next [-]

No amount of claude can handle the key thing that Tailscale fixes about wireguard, all of the connection to connection logic is handled be it direct, via all sorts of weird hole punching or via relays. Wireguard itself is horrible to do this with, half the time you're just looking at `wg status` wondering why the packet counter isn't incrementing today. It can never achieve the magic that is a direct connection between plane wifi and a host that's behind NAT on the ground.

▲swingandamiss 32 minutes ago | parent [-]

[flagged]

▲arcanemachiner 41 minutes ago | parent | prev | next [-]

This is true, but I still wouldn't discount the possibility of mistakes. It's good to know how your home network works. Even Opus 5.5 can make some pretty dumb mistakes, particularly if it doesn't have the full picture.

▲swingandamiss 30 minutes ago | parent [-]

Humans also make mistakes. At this point, I trust AI more than humans when it comes to casually configuring VPNs. I've read many who exposed things by mistake on reddit. AI has nearly solved this, probably is a solved issue at this point.

▲femiagbabiaka 41 minutes ago | parent | prev [-]

I would not personally expose my devices over a VPN that I self-hosted and set up purely through AI unless I felt like I understood how it worked completely. But that's probably just personal paranoia.

▲swingandamiss 29 minutes ago | parent [-]

Just go read reddit threads on people who do this by hand following half baked guides. There's a huge population doing this, now everyone is switching to AI. It's a big win and safer. Not everyone is a trained network engineer, but now they are with AI

▲fooqux an hour ago | parent | prev | next [-]

From the sounds of what info they felt was safe to provide, this wouldn't have impacted you anyway.

▲giwook an hour ago | parent [-]

Don’t think that refutes their point.

▲F_r_k 29 minutes ago | parent | prev [-]

BTW, netbird is a cool alternative also

▲slowin an hour ago | parent | prev | next [-]

I self-host a tailnet with Headscale and could not be happier: https://headscale.net/stable/

▲minimaltom 13 minutes ago | parent | next [-]

If you use Tailscale SSH with headscale you are also affected. The vuln is in tailscaled, not the control plane / netmap.

▲dist-epoch an hour ago | parent | prev [-]

Sure, that doesn't mean it's not vulnerable.

> Organizations using headscale v0.29.5 have no mechanism to disable the forwarding behavior at the center of the finding.

https://techjacksolutions.com/scc-vendor-rollup/tailscale-he...

▲6thbit an hour ago | parent | prev | next [-]

  > Most of the Tailscale SSH issues we've identified concern privilege escalation. In some circumstances, someone already authorized to connect over Tailscale SSH may be able to perform actions beyond the permissions of their configured local user, including potentially gaining elevated privileges.

"Most"? "In some circumstances"? Makes me think that they could also be investigating a full on unauthorized connection issue, or am I reading too much into it?
▲remus an hour ago | parent | next [-]

I think they're just writing cautiously because they're midway through the investigation and don't want to give false confidence. Doesn't seem worth reading too much into it until they provide a more thorough write up.

▲awildfivreld an hour ago | parent | prev | next [-]

The underlying writeup can be found at https://tailworm.com/

▲drum55 an hour ago | parent [-]

10,000 words of claudish, it's not worth reading.

▲pizzafeelsright an hour ago | parent | prev | next [-]

Obviously tailscale has some great devs, security is complex.

There is some weaselly language.

In some circumstances default config? with password? this is a condition leading to the next

someone already authorized to connect over Tailscale SSH they have access, but it may be limited or restricted, although scope could be beyond the intended

may be able to perform actions beyond the permissions If there is a full moon and your brother in law's second cousin is wearing a hat, he may be able to use your upstair's thermostat provided you have a second story.

▲fooqux 32 minutes ago | parent | next [-]

Given what just happened with PHP a few days so, it's not surprising to me that they're holding their cards close to their chest. People will already be doing their damnedest to reverse engineer the exploit with this meager info.

▲traceroute66 20 minutes ago | parent | prev [-]

> Obviously tailscale has some great devs, security is complex.

Quick reminder that in recent months Tailscale has had issues with age-old classic vulnerabilities such as input validation.

The problem with Tailscale is twofold.

First their attack surface is too big. They continuously shoehorn new features into the product.

Second they don't appear to do regular deep-dive code audits and instead they appear to rely mostly on the goodwill of third-parties to poke at select areas of their code as and when those third-parties feel like it. Hence simple vulnerabilities such as input validation keep on popping up.

Fancy SOC2 certificates are not the same thing. That's just dogfood for the corporate auditors.

At $work we are busy working on removing Tailscale. The bells and whistles of Tailscale sure are attractive candy, but we simply can't overlook the security risk of what has become a bloated product.

▲devmor an hour ago | parent | prev [-]

A security researcher colleague of mine was/is exploring an unauthenticated avenue along this line. I don’t have any update, she told me she didn’t feel comfortable exposing information to anyone other than Tailscale staff until she’s confident in what she found or didn’t.

▲raggi 5 minutes ago | parent [-]

we'd love to hear about it at security@tailscale.com, we're happy to work from & with and credit early ideas as well.

▲aborsy an hour ago | parent | prev | next [-]

There are too many features, and lines of code (probably more than OpenVPN). I'm pretty sure there are all kinds of zero days, with this attack surface and feature set.

I use it extensively, and I would rather pay them a fee if they could review thr code.

Another possibility is as lock down mode, disabling fancy and risky features.

▲drdexebtjl 20 minutes ago | parent [-]

This is also how I feel about Tailscale and its popular alternatives. They do too much.

Does anyone know of a Tailscale alternative that sticks to the basics?

Something that’s just WireGuard but without the manual key exchange, and with Tailscale-like NAT hole-punching that falls back to using an oblivious relay?

▲MisterTea 44 minutes ago | parent | prev | next [-]

Not a tailscale user so I am a bit lost here: where does SSH fit into a Wireguard based VPN service?

▲jubilanti 42 minutes ago | parent | next [-]

As added candy to their daemon to get you into their ecosystem

▲justinsaccount 42 minutes ago | parent | prev [-]

https://tailscale.com/docs/features/tailscale-ssh

▲Gigachad 19 minutes ago | parent | prev | next [-]

Everyone is twitchy over security right now, but at first glance this seems not that bad. If the attacker has to be manually added to your tailnet that makes this not very exploitable. You should be more worried about all the iot slop on your local network.

Good idea to set up automatic updates for the tailscale client though.

▲wslh 2 hours ago | parent | prev | next [-]

With the success of Tailscale I would like to be sure that their repositories and scenarios are continuously reviewed. I understand that there is always a next hop in cybersecurity though.

▲_def 2 hours ago | parent | prev [-]

I think it's so invasive that they ship this per default. But thats the business model i guess.

▲seemaze an hour ago | parent | next [-]

I've always had to manually enable this feature. I've never experienced it on by default across multiple platforms and deployments.

▲RobMurray 5 minutes ago | parent [-]

I've never used it once. don't even see the point. normal ssh works fine over Tailscale.

What is the advantage? Why would I want to swap the well tested auth in ssh for auth managed by tailscale?

▲femiagbabiaka 2 hours ago | parent | prev | next [-]

Do they ship this by default? I thought it had to be enabled: https://tailscale.com/docs/features/tailscale-ssh#configure-...

▲forbiddenlake 2 hours ago | parent | prev | next [-]

How do you mean? You have to run a command to enable tailscale ssh, it isn't on by default.

▲LeoPanthera an hour ago | parent | prev [-]

Tailscale SSH is not enabled by default.

▲zer0zzz 35 minutes ago | parent [-]

It’s also not even available on some platforms afaik