Tailscale not online
-
Hi, I'm running PFSense 2.7.2 on a Protectli 4 port device. Tailscale has run flawlessly for about 18 months, but while looking at a different issue I saw that it was offline in the tailscale web admin page, since yesterday.
I removed the PFSense from the tailscale web page and with some difficulty removed the tailscale packages and rebooted the pfsense.
I reinstalled the package for tailscale and generated a new auth key on the site. Tailscale is still not coming online, I recall the first time i did this it was nearly instantaneous.
I'm getting these errors in the status page on pfsense;
Tailscale Status (/usr/local/bin/tailscale status)
Error executing command (/usr/local/bin/tailscale status)
Logged out.Tailscale IP (/usr/local/bin/tailscale ip)
Error executing command (/usr/local/bin/tailscale ip)This is exactly the same error that I had before I removed everything to start again.
Any suggestions anyone?
My previous key was set to not expire, my current key expires in 3 months, but I thought that could be changed in the admin pages on the tailscale website, but only if the machine is recognised there..
I was running the tailscale to allow all local devices access without having it installed on each device. My current usage is free - about 15 devices on the tailnet.
Be appreciative if anyone can suggest where i go from here- as far as i can tell i unchecked every box which might persist data after package removal to do a clean install.
-
@IanMcLeish Facing the same problem.
It happens after a few months, even if they key is set to "Do not expire".Reported this not long ago: 16004
-
Can you get your's back online though?
I have uninstalled everything and reinstalled. Deleted the instance in the Tailscale admin page and made a new key. But I am still offline and no further forward.
-
Did you have any joy with the --unattended option?
And where would I add that if you did?
Thanks
-
@IanMcLeish said in Tailscale not online:
Can you get your's back online though?
I have uninstalled everything and reinstalled. Deleted the instance in the Tailscale admin page and made a new key. But I am still offline and no further forward.
Yeap, you need to delete the device in the tailnet, generate a new key, paste it to pfsense and save.
This alone will make it go online again.Then, check the tailnet IP your device got and changed the virtual IP for the NAT to match it.
Accept the advertised routes if any, and that is it.Did you have any joy with the --unattended option?
I didn't do anything yet besides of reporting the problem.
-
@mcury said in Tailscale not online:
@IanMcLeish said in Tailscale not online:
Can you get your's back online though?
I have uninstalled everything and reinstalled. Deleted the instance in the Tailscale admin page and made a new key. But I am still offline and no further forward.
Yeap, you need to delete the device in the tailnet, generate a new key, paste it to pfsense and save.
This alone will make it go online again.Then, check the tailnet IP your device got and changed the virtual IP for the NAT to match it.
Accept the advertised routes if any, and that is it.Did you have any joy with the --unattended option?
I didn't do anything yet besides of reporting the problem.
Aargh.
This is not working for me, it isn't going online again!
Ill try removing the packages again and starting over - i would rather not remove the entire pfsense and start again!! I was following Tom Lawrence's Youtube video. it worked for me before.
-
@IanMcLeish said in Tailscale not online:
Ill try removing the packages again and starting over - i would rather not remove the entire pfsense and start again!!
Did you generate a new key ?
Go to the tailnet, click add device, choose linux server, set the expiry to 90 days and click generate install script.
Then, you will see a code being generated at the bottom of the page, copy from the tskey onwards and paste in pfsense tailscale authentication tab.
After that, just follow the instructions in my last post. -
I was generating a key in the tailscail admin domain. I did try your method as well, but still no luck.
Thanks for your help.I wonder what went wrong on Feb 3 to cause this?
I tried restoring an older backup as well from before Feb 3 and that too made not a jot of a difference.
I think I will give up.
Personally, I know a router can be complicated, as networks and firewall are complicated, but tailscale is so simple on most every product, except this one.
-
@IanMcLeish said in Tailscale not online:
I think I will give up.
Weird, those exactly steps worked for me yesterday..
-
@mcury
I got it!I had to go to the command prompt and run the command tailscale up which then gave me a link to the tailscale web admin page to authenticate. I kinda thought that the whole point of the key was to avoid that, but it is back up and running anyway.
Thanks for your help - was nice at least to know i wasn't doing anything stupid, but maybe missing that last part was me doing something stupid!?
-
-
This still seems to be an issue, and makes the Tailscale client unreliable. I have 4 nodes down now with expiry disabled, after some unknown time, and then a router reboot they can no longer authenticate.
Error executing command (/usr/local/bin/tailscale status) # Health check: # - not logged in, last login error=invalid key: API key does not exist unexpected state: NoStateFrom the CLI I can run tailscale login, and it re-authenticates the same node, I can tailscale down + up and it connects fine, status on the webpage looks good, but if I reboot or restart the Tailscale service in the webpage it can no longer connect again with the same error needing to login again. The only way to make it work reliably is to clear the config, delete the node and reconnect as a new node.
Pfsense 2.7.2, Tailscale package 0.1.4
-
Tailscale 1.54.0 is 2+ years out of date. Tailscale has made quite a number of changes since Tailscale 1.54.0, likely rendering it incompatible with their servers.
I would consider manually updating the Tailscale FreeBSD package.
FreshPorts does not maintain an archive of all the releases, only the latest compiled by the volunteer maintainers.
The key to manually upgrading is knowing which FreeBSD version your pfSense release is running, i.e. 14 or 15.
You can following along here.
-
Upgraded 25.07 and Tailscale is broken in the way users here describe. I can manually log in using
sudo /usr/local/bin/tailscale login, but the tailscale service in pfSense does not pick this up and restarting the service clobbers the login state. Given 16004 was logged 7 months ago with zero activity, this is an indication that Netgate devices no longer support Tailscale. -
Upgraded to 25.07 and facing the same issue. Tried the "tailscale up" command as suggested above but restarting the tailscale service kills the login again.
-
@satrajitdas I had same problem, just go to tailscale control panel and generate a new key, then paste on pfsense gui, you should be ok.
-
@Wolf666 Just reboot your router - bet the key wont auth again.
-
Hi guys,
Just $0.01, because I faced same issue on pfSense+ 24.07.1 upgrade, but I think root cause may NOT be pfSense+ per se.
To my understanding (which may be wrong):
- Tailscale uses 2 keys:
- node auth key - by default expires after 180 days. We do not see it, but you can disable its expiration and we normally do that, right? So node registration should not expire after 180 days and re-authentication should not be necessary.
- preauth key - it is valid for maximum 90 days and you input it to VPN / Tailscale / Authentication as Pre-authentication Key (eg. tskey-auth-123456789011CNTRL-2pz1kCcaSjJucckK7U5Xbz123456a890). Remember that this key is valid for no longer than 90 days. So usually when you upgrade pfSense+ (=> Tailscale upgrade) this key is well expired. Device can not (re)authenticate using it - it is expired:
- not logged in, last login error=invalid key: API key does not exist(because key that pfSense+ tried to use is expired!)
- Tailscale says that device may sometimes require re-authentication. Here is info from Tailscale KB:
Auth keys
Auth keys are available for all plans.
Pre-authentication keys (called auth keys) let you register new nodes without needing to sign in using a web browser. This is most useful when spinning up containers, IoT devices, or using infrastructure-as-code systems like Terraform.
An auth key authenticates a device as the user who generated the key. That is, if Alice generates an auth key, and uses it to add a server to her tailnet, then that device is authenticated with Alice's identity. Think of it as logging into a device.
However, if you use tags with an auth key, after a device logs in as the user who generated the auth key, the device assumes the identity of the auth key's tags.
As an alternative to directly creating auth keys, consider using an OAuth client. You can use an OAuth client and the Tailscale API to programmatically create auth keys.
Types of auth keys
Auth keys can either be:
One-off, for one-time use. They can only be used to connect a device or server one time.
This is meant for situations where you can't authenticate on the device yourself, so using a key is more practical.
For example, a cloud server might use a one-off key to connect.
Reusable, for multiple uses. They can be used to connect multiple devices. For example, multiple instances of an on-premises database might use a reusable key to connect.Be very careful with reusable keys! These can be very dangerous if stolen. They're best kept in a key vault product specially designed for the purpose.
Key expiry
An auth key automatically expires after the number of days you specified when you generated the key. You can choose the number of days, between 1 and 90 inclusive, for the key expiry. If you don't specify an expiry time, the auth key will expire after the maximum of 90 days. If you want to continue using an auth key after it expires, you need to generate a new key.
You can enable or disable key expiry on a device by using the Machines page of the admin console and by using the Update device key method in the Tailscale API.
If an auth key expires, any device authorized by it remains authorized until its node key expires. Each device generates a node key when you log in to Tailscale and uses it to identify itself to the tailnet. By default, node keys automatically expire every 180 days. You can change the default node key expiry from the Key Expiry section of the Device management page of the admin console.
Learn more about key management.
- You can use tags as "service accounts" and have some devices NOT bound to any specific user (removal of user removes devices he own). You can define 1 or multiple tag owners (users managing tag).
Tag vs. user authentication
Tags are parallel to user authentication. They serve the same role as a user account, except they're intended for service-based devices, such as a web server or an app connector. As a result, it's impossible for a user account identity and a tag identity to exist on the same device. Applying a tag to a device previously authenticated with a user account removes the user account. Similarly, authenticating a device with a user account removes all tags from the device.
Because tags are intended for non-user devices, they have qualities and limitations that make them unsuitable for authenticating end-user devices, such as a MacBook or a mobile device. For example, devices with a tag-based identity cannot use SSH to connect to a device with a user-based identity.
Key expiry
When you apply a tag to a device for the first time and authenticate it, the tagged device's key expiry is disabled by default.
If you re-authenticate a device tagged before March 10, 2022, its expiry will be disabled by default.
If you change the tags on the device from the admin console, Tailscale CLI, or the Tailscale API, the device's key expiry will not change unless you re-authenticate. After you re-authenticate, the device's key expiry will be disabled.
You can enable or disable key expiry on a device from the Machines page of the admin console or by using the Tailscale API.
Key expiry for tagged devices
Key expiry for tagged devices is disabled by default. If you change the tags on the device through the admin console, Tailscale CLI, or Tailscale API, the device's key expiry will not change unless you re-authenticate. That is, if it is enabled, it stays enabled; and if it is disabled, it stays disabled. After you re-authenticate, the device's key expiry will be disabled.
You can find recently revoked or expired keys on the Keys page of the admin console.Best practices
Depending on what devices you're authenticating, consider using an auth key that is:
- Ephemeral, for authenticating ephemeral nodes as part of short-lived workloads. Because node keys do not persist when a workload restarts, they reconnect as a different node. Tailscale automatically removes inactive nodes. For example, containers or Lambda functions should use an ephemeral key to connect.
- Pre-approved, for servers. If your tailnet has device approval enabled, this lets you add a device to your tailnet without further authorization. For example, shared devices, such as servers, should use a pre-approved auth key to connect in a network with device approval.
- Pre-signed, for nodes whose auth keys are signed locally on a signing node, which applies to tailnets with Tailnet Lock enabled. You can make an auth key (created by any means) pre-signed only by using the tailscale lock sign CLI command.
- Tagged, for servers. You can automatically apply a tag to a device by including the tag in the auth key. Access control policies restricting the device's permissions based on the tag apply after provisioning the device. For example, shared devices, such as servers, should use a tagged auth key to connect.
-
I am personally going to try Tagged (preauth) key and all my pfSense+ exit nodes (3) are already tagged as "router". So I will remove nodes from tailnet and re-add with Tagged key providing "router" tag.
-
I read that Tailscale on software upgrade MAY (rarely) REQUIRE device reauthentication. Having in mind that normal preauth keys expire after 90 days you should provide valid (non-expired) preauth key before pfSense+ upgrade? Eventually maybe if routers are tagged and preauth key is tagged then there will be no problem?
-
Currently pfSense+ does not ask us about expiration date of Preauth Key, so it can not remind us that key is expired and it may lead to problems. When valid key is needed (reauthentication) then device will fail with the message we see.
I would not blame pfSense+ yet, because I think Tailscale may require device reauthorisation sometimes and message you got tells you are trying to use expired key for authentication thus process is failing. I agree reauthentications should be rare or non-needed, but we may not know everything here.
-
@totalimpact in my case I dsid not reboot the router, after I copied the new key tailscale went online.
