Tailscale SSH: Investigation and remediation update

Posted by figmert 1 day ago

Counter382Comment190OpenOriginal

Comments

Comment by minimaltom 1 day ago

Former tailscalar here and now someone managing tailscale at an org.

Having done my own review of the vulns and the fixes as they are landing in github, its pretty clear to me that this only affects situations where you use ACLs to permit ssh into a node (via tailscale ssh, not via opensshd), but dont give a path to sudoers/root.

If a user had no SSH access and/or connectivity to a vulnerable node, none of these vulns give them access.

However, if you have a vulnerable tailscaled and you give ssh access as some non-root account/privs, then yes, these vulns are relevant as theres now known privilege escalations that give access beyond some specific user account.

Not trying to play down the vuln (its important), but its not a free-for-all RCE that some ppl are acting like it is.

Comment by whywhywhywhy 1 day ago

Calling it “tailworm” feels like alarmism to promote their find more.

Would only show worm behavior if machines within the tailnet had the right settings and also were members of other tailnets with those settings but then it would require someone in that tailnet to have another tailnet with the right settings…

It’s far fetched.

Comment by dennysora-main 1 day ago

But a large portion of Tailscale users should be software engineering professionals, so a lot of the time, people use Tailscale just for SSH. I personally feel the impact is still quite big, especially since SSH leads directly to control of the server.

Comment by minimaltom 1 day ago

Do you use tailscale ssh but either dont have sudo, or don't permit login as root?

If all your tailscale-ssh ACLs allow you into an account with sudo, or allow you into root, theres no impact (you cant do anything with these vulns you couldnt do a month ago). Thats the nuance I'm trying to get across (admittedly my english is bad).

Comment by lambda 1 day ago

On the majority of nodes I provide Tailscale SSH access to, it's as root or a user with passwordless sudo. It's being used for admin access. So yeah, on those it doesn't matter as much.

There are a handful which restrict users to their own user accounts. But it's on dev server with friendly devs, so even for that use case it's not really the end of the world.

I'm sure our infosec folks are going to demand that we upgrade everywhere, however.

Comment by 19 hours ago

Comment by stingraycharles 1 day ago

Engineers use vanilla SSH for SSH, Tailscale SSH is more focused on enterprise stuff where you can have an audit trail of all commands executed and enforce ACLs.

Comment by scoodah 1 day ago

I’ve used tailscale in quite a few jobs now and not one have we used tailscale ssh. Call us old school or whatever but at my current place of employment we provision ssh keys on a box with ansible and just use regular old ssh over the tailnet, not tailscale ssh

Comment by randomblock1 1 day ago

Tailscale SSH is different than doing SSH with Tailscale

Comment by septune 1 day ago

[dead]

Comment by hmartin 1 day ago

[flagged]

Comment by zaptheimpaler 1 day ago

Basically you can use Tailscale for authnz instead of plain old keys and they add some extra logging and stuff [1]

[1] https://tailscale.com/docs/features/tailscale-ssh

Comment by gnabgib 1 day ago

[flagged]

Comment by waterTanuki 1 day ago

The vast majority of tailnets are just hobbyists on the generous free tier and probably don't use tailscale ssh, but rather ssh over a tailnet. Usually it's either just one person or a family sharing access to the tailnet. But for corporate accounts where ACLs are used for SSH yeah this is big.

Comment by jon-wood 18 hours ago

I'd be fascinated to see numbers on whether your assumptions here are correct in terms of the ratio between hobbyists using Tailscale on their home network and actual paying customers using it for more serious purposes. Does that change if you look at user counts rather than networks?

My gut feel is that you're correct but presumably they do have a decent number of proper corporate customers or none of these features would exist in the first place.

Comment by embedding-shape 11 hours ago

> so a lot of the time, people use Tailscale just for SSH. I personally feel the impact is still quite big, especially since SSH leads directly to control of the server

I too use ssh with Tailscale, but as I understand things, this is only impact "tailscale ssh", which is different from doing "ssh over tailscale" (over tailnet), at least how I understand other comments here, someone please correct me if I'm wrong.

Comment by lifeisloving 1 day ago

I use tailnet as a vpn and ssh into my server via the tailnet, but I dont use tailscale ssh the product. I think most people fall into this bucket.

Of course it should be taken seriously, but this topic is also piggy backing off of 7 days straight of non stop viral engagement farming on X after php2shell, where people we're farming likes by just posting stuff like

"YoU ThInK PhP2ShELl iS BaD JuST WaIt unTiL TaIlScAlE get HaCkED, ThIS JuSt ThE BeGiNNiNG oF a CybeR ReCKoNiNG"

To farm FUD from normies. Which of course led researchers who were working on tailscale vulns to take advantage of that algorithmic attention as well and talk about their research.

Like Im pretty sure we knew about this vuln for 3 days now, or atleast that it was responsibly disclosed (and not being used in the wild).

The attention this is getting is definately the product of some cringe viral snowball effect of the also rather benign p2pshell vuln.

It feels like there's a ton of AI influencer astroturfing going on, because its wildly easy to get engagement when you add the words "agent/ai" with "vulnerbility" and [insert popular service].

Comment by eru 19 hours ago

On a tangent: are you really using tailscalar as the name for employees, or is it a typo and you meant tailscaler?

Comment by minimaltom 15 hours ago

Tailscalar!

Another fun fact: there’s a list of things with tails.txt and scales.txt (see GitHub) and that list is used to generate the random tailnet name :)

(Or at least it used to, it’s been years)

Comment by ramblurr 1 day ago

Why does your comment here contain more information than the blog post from tailscale itself?

Comment by quantumwoke 1 day ago

I think this does downplay the vulnerability, since it assumes that teams using tailscale SSH have a lax security posture. I would hope that a security conscious team has accounts with least privilege available i.e. no root access. Please don't discount this critical vulnerability.

Comment by Alex13192 1 day ago

[flagged]

Comment by manyzens 1 day ago

[dead]

Comment by aborsy 1 day ago

I backup my different devices to my NAS. The user is a restricted backup user seeing only their own directory. I obviously don't want my torrent computer gets root on my NAS.

I personally use openssh, but someone else may use tailscale SSH.

Comment by rietta 1 day ago

I am fairly glad I have stuck with stock OpenSSH that ships with Linux. This incident proves I am glad with that choice.

Over the decades, I have maintained remote access through port forwarding on my router - at one point in 2008 I put SSH on port 80 so I could access it from the restricted wifi network at the hospital my grandmother was at. I have also used openvpn. I have also used cloudflare tunnels. But in the end, I always connect to hardened openssh with key only authentication.

Comment by selfmodruntime 18 hours ago

Your use of OpenSSH in a single private network does not equate in any way or form using and distributing a non-ACL managed SSH client.

Comment by rietta 7 hours ago

Clearly, but of all the ways to manage authorized keys on a fleet of endpoints was this particular methodology the only or best? Were there other ways to activate and deactivate keys without introducing the same complexity and brittleness. Clearly the ACL methodology provided a false sense of security in this instance in the presence of this bug.

Comment by InsidiousWombat 18 hours ago

sounds like just adding complexity over traditional method of OpenSSH + key distribution (either via configuration management or ldap)

and now, bugs

Comment by selfmodruntime 11 hours ago

No, it doesn't sound like it. Tailscale reduces complexity. It literally is key distribution with an ACL.

I work at a finance org with hundreds of engineers and thousands of technical users. You cannot be compliant with finance compliance frameworks using OpenSSH without audits and traceability and some source of identity management. That is exactly what Tailscale (or an alternative product) provides.

> and now, bugs

If you do it yourself, you now own that software entirely. Your comment lacks understanding of the complexity of real world scenarios.

Comment by ferngodfather 10 hours ago

You just need to implement an SSH CA. You can roll your own. Vault has an enterprise one. StepCA is suitable for smaller orgs.

All the benefit of the core SSH protocol as it's just key signing. It's honestly underrated.

Comment by nisegami 17 hours ago

Adding complexity would imply you're doing both. In this case it would be more correct to say you're trading the complexity of key distribution and access control via keys for the complexity introduced by tailscale ssh.

Comment by waynesonfire 13 hours ago

yep, this was obvious from day one.

Comment by speedstyle 1 day ago

I gather the bug allows people you've explicitly authorized to ssh into some account on your machine, to access more than that account can. So if you only use ssh yourself, or only give ssh keys for users with sudo, or only to people you trust not to exploit this, or if other vulnerabilities allow local shell users to escalate, or if you only use tailscale for port forwarding, then you're unaffected. Arguably even less exposed than a public openssh instance, though I can't say I have any issue doing that

Comment by LtWorf 23 hours ago

Privilege escalation is now nothing to worry about?

Comment by speedstyle 13 hours ago

Of course it is, if you rely on that boundary. The 'consumer' use of tailscale is just as a reverse proxy, and that's how I figured GGP sees it. I don't have anyone on my tailnet, and when I did they didn't have ssh keys, and if they did I imagine there'd be many paths to privilege escalation, certainly it's a bigger surface than sshd. So I was clarifying it seems like an escalation from authorized execution to privileged execution, and not e.g. from unauthorized users to RCE

Comment by sroussey 23 hours ago

Privilege escalation is nothing to worry about on root accounts

Comment by eternauta3k 22 hours ago

I guess if you have an industrial arm connected then they could escalate to physical access

Comment by LtWorf 22 hours ago

The "root account" is your assumption now no?

Comment by minimaltom 1 day ago

I'm with you- the attack surface of openssh (and my trust in damien) is a better match imo

Comment by LeBit 1 day ago

Doing ssh through port 80 or 53 or 443 is immediately flagged.

I understand port 443 is always opened but nowadays you can’t just run ssh traffic on 443 without it being detected as not being https

Comment by lmm 20 hours ago

A lot of networks still just do dumb port blocking. Yes, any network that is making a serious effort to prevent ssh will succeed. But a lot aren't.

Comment by octoberfranklin 1 day ago

Flagged by who? The Internet Police? This comment is absurd.

Comment by novafunc 1 day ago

By restricted networks, like hospitals, workplaces, schools, etc.

If their firewalls can detect it not being typical http/s traffic, they can block it.

Comment by QuantumNomad_ 1 day ago

I run WireGuard VPN on port 443 UDP. Many times I can connect to that from various public WiFi networks like coffee shops and public libraries. And then I ssh into my machines at home over my self-hosted VPN.

I’ve been hoping that HTTP3 adoption would make 443 UDP accessible from even more places. But maybe they already do the kind of thing you said for that too, looking at the traffic flow to determine that it is not HTTPS.

Weirdly sometimes there are WiFi networks where I am not able to connect to my WireGuard VPN on 443 UDP, but I can connect to servers over ssh on standard port 22 TCP. Perhaps they sometimes block all UDP traffic because they don’t want people to use BitTorrent etc, but one would think that with all the online meeting platforms that do video streaming there would be a lot of problems for people if all UDP traffic was blanket blocked.

I might eventually try out AmneziaWG instead of vanilla WireGuard, but at the moment I am fine not being able to Connect to my VPN sometimes. I only use it to connect to my other machines anyway, not as a VPN tunnel for my web browsing etc.

Comment by teravor 1 day ago

it's possible commercial routers such as the ones utilized by restaurants allow wireguard through these days? wireguard makes no effort to be stealthy when emitting.

some VPN apps bypass wireguard blocks automatically by using https tunnels and whatnot.

Comment by rietta 1 day ago

It worked in 2008 - 18 years ago. I have other options now like said cloudflare tunnel.

Comment by Imustaskforhelp 1 day ago

Does mosh help with it? Seems to be based on udp, and it looks good. I am not entirely sure if it helps the issue that you are talking about so hoping for someone more experienced to chime in

https://mosh.org/

Comment by jen20 19 hours ago

I think you're likely attributing too much competence to the kind of place that restricts outbound traffic to 80 and 443 in the first place.

Comment by megous 1 day ago

Naive. There's not much that prevents you from running ssh over https. Not even traffic patterns will give you up in the age of ubiquitous websockets and/or http2 streams.

Comment by toast0 9 hours ago

> Not even traffic patterns will give you up in the age of ubiquitous websockets and/or http2 streams.

SSH is dead simple to detect, because the server sends an outbound greeting in ASCII text; and the client doesn't send anything until it gets the greeting. The binary protocol does not look like a TLS stream at all either.

Comment by InsidiousWombat 18 hours ago

It will absolutely and trivially be detected by any decent IDS/IPS

SSH does nothing to hide it is SSH (tho "I'm just https" mode would certainly be welcome addition)

Comment by sandos 21 hours ago

Even with corporate MITM setups? I somehow doubt it..

Comment by megous 19 hours ago

MITM only works if you allow trust in corporate CA on the outbound HTTPS connections. And why would you?

Comment by jon-wood 18 hours ago

Its not always the case but many of MITM-ing corporate firewalls are going to immediately drop any TLS connection they can't decrypt on the basis they're coming from an unauthorised device.

Comment by LeBit 20 hours ago

No need for MITM.

DPI immediately can tell it is ssh.

Comment by megous 19 hours ago

Yeah, sure. By magic I guess. :)

Comment by p_l 18 hours ago

No, by DPI. SSH has recognizable signature as does web sockets etc.

Takes flipping one switch for 443 to be made HTTPS + HTTP2 + websocket only by egress firewall if you paid for one with DPI

Comment by LeBit 1 day ago

"Naive”. How ironic.

Comment by andix 1 day ago

Somehow I don't think it's a good idea to let Tailscale handle service authentication at all. It's supposed to be an additional security layer around existing infrastructure and services.

Letting Tailscale handle VPN and SSH at the same time is basically once again a single point of failure. Similar to exposing SSH directly to the internet.

Comment by computerfriend 19 hours ago

This was a decision I made in my old job, seems to be proved correct.

Comment by JJJollyjim 23 hours ago

I agree that this is how you should use it, but it is not "supposed" to be that in the sense that Tailscale strongly encourages you to use it in lieu of auth...

Comment by teekert 23 hours ago

The amount of people claiming to be happy to have never gotten on the Tailscale (or other virtual network layer like Nebula) boat is too damn high. Remember people, it's just a layer. I.e. GitLab seems to have a new set of critical CVEs every week now, and I have been glad to have Tailscale (plus a physical firewall) as a layer in front of it. Total stress level would have been higher without Tailscale.

And before people start coming up with alternatives, I know, but they either have the same (theoretical level of) attack surface (ie [managed] Nebula), or they require much more manual work (ie Wireguard) and are therefore more sensitive to misconfiguration.

Comment by mzhaase 21 hours ago

Always comes back to fundamentals: defense in depth, principle of least privilege, patch your shit. Almost every security issue comes from violating one of those.

Comment by waynesonfire 13 hours ago

> they require much more manual work

Yes, needing to RTFM is much more manual work. Better to deploy bugs.

Comment by aborsy 1 day ago

There are too many features, and lines of code (probably more than OpenVPN). I'm pretty sure there are all kinds of zero days in it, with this attack surface and feature set.

I use it extensively, and I would rather pay them a fee if they could review thr code.

A possibility is a lock down mode, disabling risky features.

I just want a VPN for connrctiviy, no taildrop, taildrive, SSH, funnel, services, DNS, node sharing, ...They are disabled in my admin console, but the code runs on my devices and they can be enabled by mistake.

A question. Taildrop enables unauthenticated input and doesn't even obey ACLs. I'm not quite sure about this, but couldn't malware in a low trust device send a file to my Synology NAS, which will then automatically process it for indexing things like thumbnail, leading to potential code execution? Why doesn't it obey ACLs, and why isn't the recipient approval needed before the file exits the sender's device? I use tags, but I'm the owner not the tag.

Comment by drdexebtjl 1 day ago

This is also how I feel about Tailscale and its popular alternatives. They do too much.

Does anyone know of a Tailscale alternative that sticks to the basics?

Something that’s just WireGuard but without the manual key exchange, and with Tailscale-like NAT hole-punching that falls back to using an oblivious relay?

Comment by sigbottle 1 day ago

I've been toying around with nebula, self host your lighthouse on a cheap VPS (which even if it gets hacked, the most it can do is lie about traffic, not lie about node certs). Basically you're just paying for the punchthrough (you may even be able to do some duckdns shenanigans?) The main issue is that updating nodes is a pain in the butt, which is the main advantage of tailscale IMO - centralized control plane. Keeps it simple.

I was trying to do some kind of insane workflow where you as the operator have a privileged hardware yubikey that admins the nodes when you plug it in, and the point is the nodes would self-propogate the changes outwards. I immidiately ran into classic issues with distributed security in distributed systems (even with only one central operator trying to admin everything). Tailscale is also a bit more secure in its oauth layer - with this architecture, you need to set proper time limits for expiring so one node compromise doesn't get access to the whole fleet. Couldn't really come up with better ideas than forcing the operator to reuse some kind of hardware key (the idea trying to be avoiding maintaining a cloud control plane as much as possible)

Fun foray but creating such a system requires a lot of expertise that I unfortunately lack

Comment by LeBit 1 day ago

I used Nebula extensively.

I really liked it. I like the idea of embedding firewall rules in a certificate.

But , like you observed, it does not scale as well as tailscale.

Comment by Yasuraka 1 day ago

https://github.com/netbirdio/netbird

Bit more than the basics, but not as much as tailscale.

Comment by huslage 1 day ago

You can run Tailscale in this mode just fine, or use Tailcat if you want.

Comment by drdexebtjl 1 day ago

The point, in the context of this thread, is that the attack surface is huge.

Comment by LeBit 1 day ago

No.

{tail,head}scale is the best there is for wireguard at more than 3 nodes.

Comment by chasd00 1 day ago

You could just use IPsec. It’s been probably 20 years but look into SWAN. It’s just a vpn in the vpn sense.

Comment by InsidiousWombat 18 hours ago

IPSec is horrid. Please, don't

It is massively more complex than say Wireguard, because it was developed with "everything and kitchen sink approach", and even the standard de facto authors, Cisco, took good 20 years to stop having IPSec related vulnerability every few months (hell we just had one this year in Linux IIRC)

Comment by sgerenser 18 hours ago

That doesn’t really help much if you want to connect an arbitrary number of nodes to eachother (each of which is potentially on an entirely different LAN), does it? That’s really the whole point of Tailscale.

Comment by ctippett 1 day ago

Taildrop only works with devices that are yours, i.e. you can't send or receive anything from a device that's either tagged or owned by someone else. The only way malware is getting onto your NAS (via Taildrop) is if you're the one sending it there.

Comment by JJJollyjim 23 hours ago

My Tailscale RCE[1] in 2022 involved forcing a device onto an attacker-controlled control plane, at which point your device is no longer "yours", and the fact that Taildrop is implemented in your client binary can be exploited. Chaining the bug to RCE would not have been possible against an ideal Tailscale with only the core feature set.

[1]: https://web.archive.org/web/20221222120703/https://emily.id....

Comment by ctippett 22 hours ago

This was(/is?) a great write-up. Would the vulnerability enabled through Taildrop still work if the compromised machine was tagged or would the attacker's control of the... control plane make this additional safeguard redundant?

Comment by aborsy 23 hours ago

See my comments in this thread.

Comment by mintflow 1 day ago

As a indie hacker(viber :) ) built mutiple tailscale based stuffs dated to the early days of tailscale, I like it but actually I do agree with you

There are too much features added now, perhaps some related to enterprise features because that must drive the revenue

Sometimes I am thiking perhaps they provide a light tailscale version but that seems not viable given the complexity now, headscale is great, but the OSS client still have plenty features, though they are gradually added feature based switch in the client source which I think is great, perhaps we can build a light version over it

This is also I do not think the tailscale-rs(which aim to rewrite tailscale with rust) will not be ready in a short time because too much (feature + cross platform) need to be tested

Comment by skowalak 1 day ago

Re: Your question. Taildrop only works between devices in the same account. So if you don't put low trust devices in the same account as your NAS this is mitigated.

Comment by aborsy 1 day ago

Tailnet owner may not trust all their devices to the same extent. Here is a concrete example.

I install tailscale on my cloud vps which I use for downloading torrents. I want my laptop to talk to this node, but I don't want this vm talk to my laptop, which is the whole purpose of ACLs. My ACL is deny by default, so that is great and easy to set. But now this node can still send arbitrary files to my laptop. If I don't remember to tag it, I'm vulnerable. If I do, there is still a window that this node can send me files until I tag it (I can't install tailscale to a tag).

Sending a file otherwise normally requires an app listening to a port, so taildrop permissions should be same as network permissions set by ACL.

BTW, last year or so tailscale advertised a position for someone to manage complexity of tailscale. Founder was talking about this issue. They seem to be aware of the problem.

Comment by ctippett 1 day ago

Tags are a key part of managing ACLs. If you're concerned about an untrustworthy node on your tailnet, use tags. Even if you forget to add a tag to a newly added node (API keys can be set to automatically assume a tag when used fwiw), if your other devices are tagged then they're ineligible for taildrop.

Comment by bradfitz 1 day ago

(Tailscalar)

FWIW you can rebuild tailscaled to omit all the optional stuff:

https://github.com/tailscale/tailscale/tree/main/feature#mod...

e.g.

  % go run ./cmd/featuretags --min
  ts_omit_ace,ts_omit_acme,ts_omit_advertiseexitnode,ts_omit_advertiseroutes,ts_omit_androidbin,ts_omit_androiddns,ts_omit_appconnectors,ts_omit_aws,ts_omit_bakedroots,ts_omit_bird,ts_omit_c2n,ts_omit_cachenetmap,ts_omit_captiveportal,ts_omit_capture,ts_omit_cliconndiag,ts_omit_clientmetrics,ts_omit_clientupdate,ts_omit_cloud,ts_omit_colorable,ts_omit_completion,ts_omit_completion_scripts,ts_omit_conn25,ts_omit_connreject,ts_omit_dbus,ts_omit_debug,ts_omit_debugeventbus,ts_omit_debugportmapper,ts_omit_desktop_sessions,ts_omit_dns,ts_omit_dnsresolvecache,ts_omit_doctor,ts_omit_drive,ts_omit_exitnodehealth,ts_omit_favorites,ts_omit_flashappliance,ts_omit_gro,ts_omit_health,ts_omit_hujsonconf,ts_omit_identityfederation,ts_omit_ipnbus,ts_omit_iptables,ts_omit_kube,ts_omit_linkspeed,ts_omit_linuxdnsfight,ts_omit_listenrawdisco,ts_omit_logtail,ts_omit_nattraversal,ts_omit_netlog,ts_omit_netstack,ts_omit_networkmanager,ts_omit_oauthkey,ts_omit_osrouter,ts_omit_outboundproxy,ts_omit_peerapiclient,ts_omit_peerapiserver,ts_omit_portlist,ts_omit_portmapper,ts_omit_posture,ts_omit_qrcodes,ts_omit_relayserver,ts_omit_remoteconfig,ts_omit_resolved,ts_omit_routecheck,ts_omit_runtimemetrics,ts_omit_sdnotify,ts_omit_serve,ts_omit_serviceclientprefs,ts_omit_ssh,ts_omit_synology,ts_omit_syslog,ts_omit_syspolicy,ts_omit_systray,ts_omit_taildrop,ts_omit_tailnetlock,ts_omit_tap,ts_omit_tpm,ts_omit_tundevstats,ts_omit_udptransport,ts_omit_unixsocketidentity,ts_omit_useexitnode,ts_omit_useproxy,ts_omit_usermetrics,ts_omit_useroutes,ts_omit_wakeonlan,ts_omit_webbrowser,ts_omit_webclient

Comment by traceroute66 18 hours ago

> FWIW you can rebuild tailscaled to omit all the optional stuff:

It would also cost Tailscale nothing to build and ship a minimised version alongside the bloated version. Thereby making it officially supported option.

Because you just know support will start finger-pointing and asking you to install the bloated version if you tell them you're having an issue with your home-compiled minimised version ....

Comment by infogulch 15 hours ago

What about mobile apps?

Comment by InsidiousWombat 18 hours ago

Given the amount of problems 3rd party SSH servers had (Mikrotik had some recent issues too) I think we should add "do not roll your own SSH server, just use OpenSSH" to common security advice

Comment by wrxd 1 day ago

I like Tailascale as a product but I decided to move my homelab to plain WireGuard.

I wasn't using any of the additional features, I can get away with exposing a port on my router and I am changing devices rarely.

News like this make me happy I removed a lot of complexity and attack surface

Comment by femiagbabiaka 1 day ago

> I can get away with exposing a port on my router

using wireguard is awesome, but I wouldn't say deploying and maintaining it lacks complexity exactly. and you didn't get rid of attack surface, you just made it your responsibility ;)

Comment by ectospheno 1 day ago

Always is your responsibility even with tailscale. Now it is his responsibility without tailscale.

Comment by LoganDark 1 day ago

Attack surface almost entirely goes away if you just require a good port knock first! I do that with my protection SSH servers and it works just fine. Also prevents anyone from knowing there even is an SSH server there. Or any server if it doesn't respond to pings either, and only uses reverse tunnels to accept public connections

Comment by neilalexander 1 day ago

Wireguard doesn’t respond to malformed packets or handshakes from unknown devices by design.

Comment by LoganDark 1 day ago

Oh, awesome!

Comment by mbreese 1 day ago

Is there a way to automate port knocks in your .ssh/config? I think that’s the main reason why I haven’t added it to my home server.

Comment by LoganDark 1 day ago

Technically you could use a ProxyCommand that does the knock and then runs nc to your target.

Comment by lakeviewterrace 1 day ago

Port knocking is idiotic and amounts to Biff Tannen's "only I know how to start this car".

Comment by Gigachad 1 day ago

Dunno, in combination with having an up to date system following best practices, port knocking is probably going to save you when bots start scanning the internet with the next SSH 0day. Any amount of friction will cause the bots to move to the next IP address.

Comment by icedchai 1 day ago

Just moving SSH off a default port gets you 98% there.

Comment by LoganDark 1 day ago

An attacker only has to get lucky once, so while I do put SSH on a non-default port too, it's not enough to avoid attacks on its own.

Comment by icedchai 1 day ago

I mostly do it to quiet down the noise from the non-stop bots and scanners.

Comment by LoganDark 1 day ago

Being off the IPv4 space completely can be even more effective at that, FWIW, but that can have other issues. Only feasible for some!

Comment by esseph 1 day ago

LLMs port knock for redteam scans. Have for a long time.

Comment by LoganDark 1 day ago

My knock for SSH was 20 ports within 2 seconds. knockd resets the sequence if you hit any of its ports out of order. I generated the sequence with a CSPRNG.

Comment by esseph 1 day ago

That's not a knock, it's a password :-)

Comment by LoganDark 1 day ago

I mean, it suffers from some of the same issues with knocks, like being replayable. Wonder if anyone's come up with a TOTP-like thing for port knocks, where the code changes every little bit...

Comment by dns_snek 20 hours ago

Yes something like that has been done, the solution uses asymmetric cryptography instead of symmetric and it protects the entire communication stream instead of just opening the gates - it's called Wireguard.

I jest, but you're describing authentication protocol which uses the UDP destination port field to carry each character of the "one time password" in a roundabout way. You could simply send 1 packet containing the password to the daemon.

I've never seen it used in the wild but apparently, it exists: https://github.com/mrash/fwknop

Comment by LoganDark 15 hours ago

LOL'd at Wireguard, but I really miss Plan 9-style networking, where you can basically just scope VPNs to a subprocess rather than having to use some system wide network filter with split tunneling or whatever. Wireguard has huge usability issues due to that.

Reading the fwknop README is such a breath of fresh air, thank you for that treat. I hate AI-generated documentation so much.

Comment by SV_BubbleTime 1 day ago

That’s cute. Have Claude do it. There is already an echo system of TOPT QR to communicate it, so you would just need to convert your six digit code to two or more hits on the 16bit port range.

I’m not even sure you need more than two. One seems bad because that’s just a lucky port scanner one time.

Comment by 1 day ago

Comment by LoganDark 1 day ago

It's more like "only I know how to find this keyhole". You still need the key then, you're just also adding some defense against picking.

It would be idiotic to use the knock as the only authentication, which is why I don't.

To be quite honest, I just like the aesthetics of it.

Comment by megous 1 day ago

You're sending your knock over the public network in the clear. It's observable by anything in the path. And then you're not the only one that can find the keyhole.

Comment by sersi 23 hours ago

Entirely true, it's not true security, same as changing the ssh port. So, relying on it for any security is not useful but it's a good deterrent from automated scans that are not targeted towards you. Defense in depth means not necessarily discarding flawed defenses if they are in addition to proper defense.

Comment by swingandamiss 1 day ago

With AI, the complexity doesn't really matter anymore. Codex or Claude can easily configure complex site-to-site setups and peer configurations on mobile devices. I did this by hand before and it was hard and difficult to get right. Now I just tell AI and it does so perfectly, checks and verifies things. AI made life so easy.

Comment by drum55 1 day ago

No amount of claude can handle the key thing that Tailscale fixes about wireguard, all of the connection to connection logic is handled be it direct, via all sorts of weird hole punching or via relays. Wireguard itself is horrible to do this with, half the time you're just looking at `wg status` wondering why the packet counter isn't incrementing today. It can never achieve the magic that is a direct connection between plane wifi and a host that's behind NAT on the ground.

Comment by hdgvhicv 1 day ago

I can’t think of a single time I’ve wanted to connect between two hosts when I haven’t specifically allowed one through a firewall

Comment by drum55 1 day ago

NAT are not firewalls.

Comment by hdgvhicv 20 hours ago

Functionally they are. Allow traffic aiming at this address and this port to arrive on this server.

Not that my WireGuard connection needs nat as it terminates on my router. And I use private ip ranges as I want to be sure that if WireGuard drops out for some reason traffic doesn’t route over the unencrypted path. Defence in depth.

Comment by swingandamiss 1 day ago

[dead]

Comment by LoganDark 1 day ago

It makes me happy that I can automate some menial tasks or replace a search engine sometimes, but this is just because I already have knowledge of the field and the skill to apply it. You're not wrong that it'll probably get even better, but I don't see a path right now to unskilled people being able to effectively use an agent. Only time will tell if that can happen.

I do disagree with GP's assertion that Claude can never make something comparable to Tailscale, but that's just because I believe I (and probably many others here) could probably prompt that, not because I think just anyone could. LLMs can one-shot a fair few things based on unskilled prompts, but I don't believe this is one of them yet.

Comment by swingandamiss 1 day ago

Right now it's not unskilled. It's managers, PMs, TPM's, or people with some tech knowledge but who previously couldn't do these types tasks (not that they weren't capable, but it takes a lot of effort and time to get skilled in these areas and they didn't devote the time). They know what they want at a high level and now can do it with AI.

Comment by LoganDark 1 day ago

This is mostly true, I think. I think this is a sweet spot for LLMs, too. Maybe with the right approach, a user could know or figure out how to articulate what they're missing, if the first attempt doesn't work well enough for them. I just worry that e.g. if they forget to review things like protocol details, they could silently end up with flaws and such, or jank. But yeah, there's potential where there wasn't before, and I'm all for that.

Comment by arcanemachiner 1 day ago

This is true, but I still wouldn't discount the possibility of mistakes. It's good to know how your home network works. Even Opus 5.5 can make some pretty dumb mistakes, particularly if it doesn't have the full picture.

Comment by swingandamiss 1 day ago

Humans also make mistakes. At this point, I trust AI more than humans when it comes to casually configuring VPNs. I've read many who exposed things by mistake on reddit. AI has nearly solved this, probably is a solved issue at this point.

Comment by femiagbabiaka 1 day ago

I would not personally expose my devices over a VPN that I self-hosted and set up purely through AI unless I felt like I understood how it worked completely. But that's probably just personal paranoia.

Comment by swingandamiss 1 day ago

Just go read reddit threads on people who do this by hand following half baked guides. There's a huge population doing this, now everyone is switching to AI. It's a big win and safer. Not everyone is a trained network engineer, but now they are with AI

Comment by LeBit 1 day ago

It makes no sense at all to configure a >3 nodes wireguard setup with straight wireguard.

Wireguard is the building block for higher level tools like Tailscale to build on.

Comment by fooqux 1 day ago

From the sounds of what info they felt was safe to provide, this wouldn't have impacted you anyway.

Comment by giwook 1 day ago

Don’t think that refutes their point.

Comment by OptionOfT 1 day ago

On the road I find WireGuard to be quirky when connecting to DNS names. When I'm on an IPv6 only network (like T-Mobile), connecting to a DNS name that offers A and AAAA, WireGuard on iOS prefers the A record, making me connect over one of T-Mobile's 464XLAT proxies which time out after x minutes. This then makes the connection look dead until I do something on my phone that re-enables it.

Tailscale doesn't have this issue.

Comment by Dagger2 18 hours ago

An AAAA-only hostname would sidestep that issue, with the obvious downside.

(This behavior is a deliberate choice from Wireguard. They claim it's not broken, but... it's broken.)

Comment by LeBit 1 day ago

You shouldn’t open a port on your router.

Connect from your home to a public VPS and connect your public devices to that VPS.

Comment by jon-wood 18 hours ago

"Should" and "shouldn't" are dependent on context. In this case, that somewhat depends on what the router you're using is, how much you trust it, and your ISP's configuration. Opening a port isn't an inherently insecure thing, it's a choice that you make while being aware of the tradeoffs between that port being exposed to the internet at large and wanting a given service to be available.

Comment by wrxd 21 hours ago

What's the advantage of adding a VPS to the mix?

Comment by LeBit 20 hours ago

Having a VPS gives you a static, non-shared, stable IP with a whois that doesn’t resolve to a residential ISP.

Lot’s of ISP have moved to CGNAT where port forwarding has become a nightmare. That’s one reason. You can look at it as if your router moved from your home behind a public CGNAT, dynamic, shared IP to a public non-shared, static, stable IP on the internet.

In a workplace setting, connecting to your residential IP will raise suspicions or alarms. That being said, if you go for a cheap VPS, it also might raise alarms. Maybe your 'VPS' should be with whichever cloud provider your employer goes with so you don’t even register as a blip.

It is added complexity, but it is really very simple. Using Wireguard, you connect from a always on server in your home to your VPS. Then, using Wireguard, you configure your laptop to always connect to your VPS. Any local LLM can generate the wg-quick config files in seconds.

Also, not opening a port has become a security checkbox. If your home router isn’t a hardened Linux box, why would you prefer to let a ASUS 'WiFi Router' handle that critical networking function instead of a Linux box you can easily update and audit?

Comment by jon-wood 18 hours ago

This is frankly a bit nonsensical. "In a workplace setting, connecting to your residential IP will raise suspicions or alarms." - so will connecting to literally any unexpected VPN endpoint if the network is halfway competently monitored.

Just because your VPN endpoint is on a VPS somewhere rather than a computer on your home network doesn't make it any more secure, and the scare mongering here that somehow a home router will drastically mess up a function as simple as forwarding traffic coming in on a port to another machine is weird... if you don't trust it to do that then I'd get rid of the thing because that's its entire job.

Comment by onel 1 day ago

Using anything on top of wireguard or plain? I heard headscale adds a nice layer

Comment by wrxd 21 hours ago

I use plain WireGuard. My use case is very simple: I have a server and a few clients (phone, laptop) I want to connect to the server when I'm away. I don't care to have VPN connectivity between my phone and laptop

Comment by F_r_k 1 day ago

BTW, netbird is a cool alternative also

Comment by dizhn 1 day ago

I prefer it too and I like that it's 100% open source. Server, client, everything. Pangolin also looks interesting if you don't need a full vpn. And if you go that way, Authentik with its embedded proxy is also worth considering.

Comment by modeless 1 day ago

They may not recommend disabling Tailscale SSH but I do. It interferes with sshd features and adds to your attack surface. Distribute your SSH keys the old-fashioned way and disable Tailscale SSH.

Comment by kccqzy 1 day ago

I make my sshd only listen on the Tailscale IP. Then I don’t even bother setting up SSH keys. I use password authentication, except on mobile where my password is too difficult to type with an onscreen keyboard.

Comment by jaden 1 day ago

As I understand it, that use case isn't impacted by these vulnerabilities since it's not using Tailscale SSH, just OpenSSH over Tailscale.

Comment by kccqzy 18 hours ago

Correct. I have no reason to even turn on Tailscale SSH.

Comment by siva7 1 day ago

I wouldn't be too surprised if tailscale has become a critical vector of the internet like cloudflare over the years as most ai companies seem to use it

Comment by SchemaLoad 1 day ago

Tailscale is best used as an additional layer of security rather than your only layer. I host everything behind tailscale as if it was the public internet, with proper authentication, https, etc.

So a failure in tailscale is not an issue on it's own because you'd then also have to find an exploit in what I'm actually hosting as well. Both of those things happening at the same time is much less likely.

Comment by mrheosuper 1 day ago

Yet another single point of failure.

I don't see the reason for Tailscale instead of plain old simple Wireguard, it's WG under the hood anyway.

Comment by niobe 22 hours ago

+1

I am a relatively old school engineer. When building a bricks and mortar business recently I kept hearing about tailscale for remote access. Looked and saw it was just a wrapper over wireguard. I immediately went aha, there's a solid underlying project without the risk of something over the top trying to make life easier for people unwilling or unable to manage their own systems. Went with a direct wireguard on opnsense approach.

Bamn, proved right today.

Wireguard is perfectly elegant and maintainable as it is for me, and a wonderful improvement on traditional site to site VPNs.

Comment by drnick1 1 day ago

This exactly. All the extra stuff Tailscale provides buys convenience, but increases complexity, the attack surface of the stack, and adds commercial components to an otherwise open protocol. Call me old fashioned but I will keep managing keys manually.

Comment by Gigachad 1 day ago

Everyone is twitchy over security right now, but at first glance this seems not that bad. If the attacker has to be manually added to your tailnet that makes this not very exploitable. You should be more worried about all the iot slop on your local network.

Good idea to set up automatic updates for the tailscale client though.

Comment by slowin 1 day ago

I self-host a tailnet with Headscale and could not be happier: https://headscale.net/stable/

Comment by minimaltom 1 day ago

If you use Tailscale SSH with headscale you are also affected. The vuln is in tailscaled, not the control plane / netmap.

Comment by dist-epoch 1 day ago

Sure, that doesn't mean it's not vulnerable.

> Organizations using headscale v0.29.5 have no mechanism to disable the forwarding behavior at the center of the finding.

https://techjacksolutions.com/scc-vendor-rollup/tailscale-he...

Comment by 6thbit 1 day ago

  > Most of the Tailscale SSH issues we've identified concern privilege escalation. In some circumstances, someone already authorized to connect over Tailscale SSH may be able to perform actions beyond the permissions of their configured local user, including potentially gaining elevated privileges.
"Most"? "In some circumstances"? Makes me think that they could also be investigating a full on unauthorized connection issue, or am I reading too much into it?

Comment by remus 1 day ago

I think they're just writing cautiously because they're midway through the investigation and don't want to give false confidence. Doesn't seem worth reading too much into it until they provide a more thorough write up.

Comment by awildfivreld 1 day ago

The underlying writeup can be found at https://tailworm.com/

Comment by drum55 1 day ago

10,000 words of claudish, it's not worth reading.

Comment by brof 1 day ago

Thanks for linking that. The debate about these issues will rage on, but I do wonder whether giving Tailscale two hours of notice before public release was the right call? The arguments from the post are that "rapid awareness of these flaws will expediate remediation" and that they were "discovered with low effort". Does the author mean that simply pointing Claude at Tailscale identified (a non-trivial subset) of these vulnerabilities, and did the author not feel that Tailscale had a track record of responding to vulnerabilities, via a coordinated disclosure, in a timely manner? I would personally have liked to see a bit more of a justification of the almost-immediate publication against the potential risks to users, vs. e.g. waiting a day or two.

Comment by jasona123 17 hours ago

> whether giving Tailscale two hours of notice before public release was the right call?

In my world I believe we would call that a dick move.

Comment by deanputney 14 hours ago

It's so unusual that I'm starting to wonder if this was intended to harm Tailscale's public image. These don't appear to be major flaws, and if the purpose of publishing was to get them fixed there's a well known process for responsible disclosure. Sounds like they didn't even attempt to follow that.

Comment by pizzafeelsright 1 day ago

Obviously tailscale has some great devs, security is complex.

There is some weaselly language.

In some circumstances default config? with password? this is a condition leading to the next

someone already authorized to connect over Tailscale SSH they have access, but it may be limited or restricted, although scope could be beyond the intended

may be able to perform actions beyond the permissions If there is a full moon and your brother in law's second cousin is wearing a hat, he may be able to use your upstair's thermostat provided you have a second story.

Comment by fooqux 1 day ago

Given what just happened with PHP a few days so, it's not surprising to me that they're holding their cards close to their chest. People will already be doing their damnedest to reverse engineer the exploit with this meager info.

Comment by tptacek 1 day ago

There's already a public writeup.

Comment by p_l 1 day ago

Have you seen the slopreport for the vulnerability?

I fully understand being carefully worded when you need serious brain power just to figure out every claim in the logorrhea

Comment by traceroute66 1 day ago

> Obviously tailscale has some great devs, security is complex.

Quick reminder that in recent months Tailscale has had issues with age-old classic vulnerabilities such as input validation.

The problem with Tailscale is twofold.

First their attack surface is too big. They continuously shoehorn new features into the product.

Second they don't appear to do regular deep-dive code audits and instead they appear to rely mostly on the goodwill of third-parties to poke at select areas of their code as and when those third-parties feel like it. Hence simple vulnerabilities such as input validation keep on popping up.

Fancy SOC2 certificates are not the same thing. That's just dogfood for the corporate auditors.

At $work we are busy working on removing Tailscale. The bells and whistles of Tailscale sure are attractive candy, but we simply can't overlook the security risk of what has become a bloated product.

Comment by devmor 1 day ago

A security researcher colleague of mine was/is exploring an unauthenticated avenue along this line. I don’t have any update, she told me she didn’t feel comfortable exposing information to anyone other than Tailscale staff until she’s confident in what she found or didn’t.

Comment by raggi 1 day ago

we'd love to hear about it at security@tailscale.com, we're happy to work from & with and credit early ideas as well.

Comment by devmor 1 day ago

I will send her your message!

Comment by rhedak 15 hours ago

I reckon most users are fine as long as they use it only for themselves, as in one user many devices.

Comment by pluto_modadic 10 hours ago

isn't the other part of this that you'd have to enable the SSH part per node? it's not like the tailscale control plane can uniformly turn that on.

Comment by hollow-moe 1 day ago

Recently NetBird also had a huge privesc through their SSH component too... I'm using both at $DAYJOB and my personal infra, damn you guys ought to be more careful with such a sensible and important component...

Comment by 99harry 23 hours ago

I've already installed WireGuard, ZeroTier, and OpenVPN. What's the difference between Tailscale and these other tools?

Comment by teekert 23 hours ago

This is the type of question any LLM can explain to perfection on any technical level, from 5 year old to IT pro, so, I'd try that?

If I literally ask an LLM your question it makes beautiful table of pros cons, underlying tech etc. So, LM G<>LLM TFY ;)

Comment by 99harry 22 hours ago

I think it contains a lot of information. Simple definitions always need a human touch

Comment by bsoles 11 hours ago

This must be the new "Just Google it". I am pretty sure the poster is perfectly aware that they could ask an LLM.

Comment by parasyte 9 hours ago

Tailscale is most similar to ZeroTier, in that it's a mesh VPN with a control plane to handle key distribution. It's not a virtual ethernet or similar network, so it doesn't do broadcast or multicast traffic like ZeroTier. It does use WireGuard under the hood, and besides the control plane for Wireguard key distribution, you can set ACLs to restrict which mesh participants can connect to which endpoints (on which ports or services), and it can leverage its authentication for some handy tricks like SSH key management.

Most of the neat tricks like funnel, serve, SSH, etc are off by default and you need to specifically allow them for hosts to use them.

Comment by 99harry 2 hours ago

That makes sense. That might explain why ZeroTier is sometimes unstable for me. It seems to create virtual MAC addresses, which may cause network connectivity issues. I think I might switch to Tailscale, hopefully it will be more stable

Comment by MisterTea 1 day ago

Not a tailscale user so I am a bit lost here: where does SSH fit into a Wireguard based VPN service?

Comment by jubilanti 1 day ago

As added candy to their daemon to get you into their ecosystem

Comment by holoduke 16 hours ago

By coincidence i heard about tailscale for the firsttime today. I tried to understand what the benefit are. seeing people having all their devices accesible from their phone. I am using plain ssh for this for decades. Still not convinced of using tailscale.

Comment by y-c-o-m-b 11 hours ago

I have tailscale for running self-hosted stuff like Jellyfin (media server) and Immich (image host) without completely exposing these things to everyone on the web. When I'm away from home, I can easily access my media and photos without sacrificing my privacy on cloud-hosted providers for those things. It's also completely free with no limitations other than disk space and bandwidth.

My wife is tech-phobic, so having her sign up for Tailscale with her existing Google account and installing the Android app was easy enough for her to agree to this setup.

Comment by redmalang 1 day ago

vaguely related - I want to make it easy for the llms to deploy and manage derp,tailgate etc with the hard requirement that you can continue to use the tailscale clients (very thankful to tailscale for making this so easy!):

https://tailgate.latchlabs.dev

Comment by seany 1 day ago

I have tried to like tailscale ssh, but having used tailscale for many many years. I always just disable it and use normal opensshd.

Comment by wslh 1 day ago

With the success of Tailscale I would like to be sure that their repositories and scenarios are continuously reviewed. I understand that there is always a next hop in cybersecurity though.

Comment by fanruocheng 1 day ago

[flagged]

Comment by krazy_00 1 day ago

[flagged]

Comment by zkscout 21 hours ago

[flagged]

Comment by TacticalCoder 1 day ago

I'll point out that Tailscale has been repeatedly praised here as a replacement "because SSH was too old" by "security professionals". Those very same security professional happen to be the exact same closing their eyes when there's the state doing wrongdoings.

My question to those who kept praising Tailscale and making fun of good old plain SSH: how do you like your state backdoor --excuse me, I mean bug-- in Tailscale now?

Comment by Liuser 1 day ago

I’ve personally never heard any security professional say that “ssh is too old”. The primary selling point of using Tailscale ssh is that you outsource the challenges of ssh key management to Tailscale. Eg. managing and rotation of thousands of ssh keys in an org.

Comment by mediaman 1 day ago

This comment doesn't make any sense. The products are mostly different categories. One is a VPN replacement and the other is encrypted shell access. Most people use ssh over tailscale. Tailscale does have a "Tailscale SSH" product but it's not that widely used compared to regular ssh over Tailscale.

Comment by LeBit 1 day ago

You use ssh to manage a multi node VPN?

Comment by _def 1 day ago

I think it's so invasive that they ship this per default. But thats the business model i guess.

Comment by seemaze 1 day ago

I've always had to manually enable this feature. I've never experienced it on by default across multiple platforms and deployments.

Comment by RobMurray 1 day ago

I've never used it once. don't even see the point. normal ssh works fine over Tailscale.

What is the advantage? Why would I want to swap the well tested auth in ssh for auth managed by tailscale?

Comment by seemaze 1 day ago

It's handy to open a TTY connection in the browser, or to delegate your tailscale MFA account as remote authentication.

Comment by femiagbabiaka 1 day ago

Do they ship this by default? I thought it had to be enabled: https://tailscale.com/docs/features/tailscale-ssh#configure-...

Comment by LeoPanthera 1 day ago

Tailscale SSH is not enabled by default.

Comment by zer0zzz 1 day ago

It’s also not even available on some platforms afaik

Comment by forbiddenlake 1 day ago

How do you mean? You have to run a command to enable tailscale ssh, it isn't on by default.

Comment by pluto_modadic 10 hours ago

no, it is not enabled by default.