"Tailscale is not online" problem

manupfdude

@yobyot
I've SSHed into pfsense
and for the sake of testing I've simply run the command:

tailscale up --auth-key=tskey-client-kQ_THE_REST_IS_A_SECRET\?preauthorized=true\&ephemeral=false --accept-dns=false --accept-routes --advertise-exit-node --advertise-routes=X.X.X.X/24 --advertise-tags=tag:pfsense

Note the preauthorized=true and ephemeral=false
I gave this key all permissions (temporarly as I just wanted to verify it's working)
of course I had to register the tag used also in the ACL tags pane:
https://login.tailscale.com/admin/acls/visual/tags

so far so good

lbm_

@manupfdude Interesting. Thanks for the detailed answer. Im also facing the same issue unfortunately where I need to auth again after an reboot.

Are you with this approach having issues with the advertised routes not working ? I have to manually do (I havent tried your approach yet to see if this fixes the issue).

 tailscale set --advertise-routes=192.168.1/24,192.168.2.0/24 after I login. Even though they are set in the UI

yobyot

Well,

I spent some time tonight playing around with this and I think I have it.

Some suggestions for others:

Generate the OAuth client in the Tailscale admin before anything else.
Make sure to create the tag you'll need. One per pfSense instance (and clearly, one OAuth client per pfSense instance).
Give the OAuth client the permissions you think appropriate.
Very Important: make sure that you can generate an API key with the OAuth creds. The OAuth creds are, apparently, used by the CLI to generate an API key. The latter is what does the trick in tailscale up.
Do this from the pfSense console:
curl -d "client_id=kY5Mv4h8kQ11CNTRL" -d "client_secret=tskey-client-kY5[invalidchars]CNTRL-ZXo2FfBbb[moreinvalidchars]GVT" "https://api.tailscale.com/api/v2/oauth/token"
If you don't get back something like this, you'll never be able to get it to work:
{"access_token":"tskey-api-kM[lotsofinvalidchars]NTRL-[stillmoreinvalidchars]9YevL","token_type":"Bearer","expires_in":3600,"scope":"all"}
Here's what worked for me if the above returned an API token:
/usr/local/bin/tailscale up --auth-key=tskey-client-[greekedout]GVT\?ephemeral=false\&preauthorized=true --accept-dns=false --accept-routes --advertise-exit-node --advertise-routes=192.168.211.0/24 --advertise-tags=tag:[yourtaghere]
Make sure you have the cron package installed. Then add a @reboot entry using the full path (see above). I also added a cron entry every six hours as if Tailscale is up, this command does not interrupt or reset any sessions.

I've left some bytes of the creds in these examples to make it clearer where your full creds should go. The curl command requires the escape symbol (\) in the parameters that will be passed to the control plane.

FWIW, I lost an hour or more because I had (God only knows why) set Tailscale on one pfSense instance to accept DNS. Do this and the router cannot resolve the control plane API endpoint. Dumb. And I own it.

I don't know if this "fixes" everything. But it's a lot of work and it shouldn't be necessary. Somehow, this package to be useful needs to survive reboot without the need to go to these lengths.

left4apple

@manupfdude Thanks for the info. If I understand correctly, pfSense Tailscale package will ensure tailscaled is running all the time even after reboot, yet Tailscale is actually using your command to authenticate itself, right? The KEY entered in the pfSense Tailscale UI has already expired(or you can enter fake KEY there).

manupfdude

@left4apple Basicaslly - Yes.
The key on the UI is no longer relevant (once you go with the oauth route)
I'm on version 1.88.3 of tailscale, after several reboots (due to power outages...) - TS still connected and authenticated :-)

KStarRunner

This seems like a bug to me. Every other platform where TailScale is setup, the tskey-auth is only used once, and then immediately discarded. Clients don't maintain the original tskey-auth, but rather a separate credential which was bootstrapped by the tskey-auth.

So why is pfSense storing this key? IMHO, the GUI should accept the key, and once bootstrapped, delete it. The persistent presence of the tskey-auth makes me think pfSense keeps attempting to re-use it each time the service restarts.

The correct behavior would seemingly be to accept the tskey-auth in the GUI, and then once the bootstrap is completed, delete it from the stored configuration. Should a new value be entered into the GUI, presume that means a new bootstrap is needed and start fresh. But as long as that box remains blank, keep on using the existing credentials.

yobyot

@KStarRunner Clearly, it’s a bug.

KStarRunner

@yobyot Sorry, meant to say the bug appears to be in the way pfsense manages the TailScale client, vs a bug in the TailScale client itself.

marcg

@Jim-Coogan said in "Tailscale is not online" problem:

fwiw I have discovered that running in shell
tailscale down
tailscale up --force-reauth
will give you a URL you can then paste in browser and it re authenticates and gets pfsense back online as the same machine and status shows this in pf tailscale UI. This is reauthing the node key.

@Jim-Coogan Looks like your fix is working for me with the packaged Tailscale 1.82.4 on 25.07. Thanks.

Previously, if I rebooted pfSense or started/stopped tailscale, tailscale wouldn't reconnect even though key expiry was disabled. After following your steps with an expiry-disabled key, tailscale reconnects after restarts and reboots. Hoping that continues ... I'm sure I'll find out.

Vad-B

Faced the same "Tailscale is not online" issue today - 7 days after the initial setup (October 23rd).
Resolved it by regenerating the key and authenticating again. I noticed the ongoing discussions about OAuth, but chose to dig deeper into this problem first.

Here’s what I found:

This appears to be a known issue affecting Tailscale across all FreeBSD-based systems, including vanilla FreeBSD, pfSense, and OPNsense.
There is an open bug in the Tailscale repository with a solution for pfSense. I plan to test it today: https://github.com/tailscale/tailscale/issues/17047

This is how it works (source):
"Auth keys and node keys are two separate things. The auth key is used one to register the device and a node key is obtained. This node key is part of the tailscale state and authenticates this device. You can remove the expiration for this node key
...
After the initial auth, you don't need the auth key anymore. The node key in the tailscale state (with expiration removed) will auth the node forever."

But in reality, on boot or restart, the Tailscale package attempts to start the Tailscale with "tailscale up --auth-key=..." (as defined in /usr/local/etc/rc.d/pfsense_tailscaled) which lead to re-register the node with the Tailscale network using the auth-key (that is set to expire), overriding the valid existing node key that is configured not to expire.

So to fix it, we should force tailscale up without the auth-key using one of the options from the git comment above.

I was still puzzled by this 7-day timeline - where it comes from? Following a link to the related OPNsense GitHub issue, I found this explanation:

"Folks who create and use reusable auth keys won't see the issue until that key expires (default 90 days). Folks like me who created a one off key will see the issue after the first reboot after one week (default expiration for a one off key is a week.)"

Vad-B

I just tested the fix from the message above. Here's what happened:

On the pfSense device connected to Tailscale, I set the Pre-authentication Key to file:/dev/null and saved the settings. Tailscale immediately disconnected, and the Status tab gave me a login link. I followed the link, logged in, and this machine was added as a new device in Tailscale. After deleting the old record for this machine and setting a proper Tailscale IP all worked fine.

After that, I restarted the service a couple of times to confirm it reconnects automatically - so far, so good.

However, keep in mind that I had already reconnected it earlier today with a new one-off key, so I'll see how it really behaves when it reboots in seven days.

Vad-B

said in "Tailscale is not online" problem:

But in reality, on boot or restart, the Tailscale package attempts to start the Tailscale with "tailscale up --auth-key=..." (as defined in /usr/local/etc/rc.d/pfsense_tailscaled) which lead to re-register the node with the Tailscale network using the auth-key (that is set to expire), overriding the valid existing node key that is configured not to expire.

So to fix it, we should force tailscale up without the auth-key using one of the options from the git comment above.

@cmcdonald, could you take a look at this, please?

lbm_

@Vad-B
Interesting indeed! I just tried to fill the Pre-authentication Key with file:/dev/null. I get an crash in pfsense after some time, but when I login again is saved.

For me this for after service restarts at least this solves it, including the issue with the routes not being advertised even set in the WebUI.

Havent done an full restart of pfsense (yet)