Hi guys,
Just $0.01, because I faced same issue on pfSense+ 24.07.1 upgrade, but I think root cause may NOT be pfSense+ per se.
To my understanding (which may be wrong):
Tailscale uses 2 keys:
node auth key - by default expires after 180 days. We do not see it, but you can disable its expiration and we normally do that, right? So node registration should not expire after 180 days and re-authentication should not be necessary.
preauth key - it is valid for maximum 90 days and you input it to VPN / Tailscale / Authentication as Pre-authentication Key (eg. tskey-auth-123456789011CNTRL-2pz1kCcaSjJucckK7U5Xbz123456a890). Remember that this key is valid for no longer than 90 days. So usually when you upgrade pfSense+ (=> Tailscale upgrade) this key is well expired. Device can not (re)authenticate using it - it is expired:
- not logged in, last login error=invalid key: API key does not exist
(because key that pfSense+ tried to use is expired!)
Tailscale says that device may sometimes require re-authentication. Here is info from Tailscale KB:
Auth keys
Auth keys are available for all plans.
Pre-authentication keys (called auth keys) let you register new nodes without needing to sign in using a web browser. This is most useful when spinning up containers, IoT devices, or using infrastructure-as-code systems like Terraform.
An auth key authenticates a device as the user who generated the key. That is, if Alice generates an auth key, and uses it to add a server to her tailnet, then that device is authenticated with Alice's identity. Think of it as logging into a device.
However, if you use tags with an auth key, after a device logs in as the user who generated the auth key, the device assumes the identity of the auth key's tags.
As an alternative to directly creating auth keys, consider using an OAuth client. You can use an OAuth client and the Tailscale API to programmatically create auth keys.
Types of auth keys
Auth keys can either be:
One-off, for one-time use. They can only be used to connect a device or server one time.
This is meant for situations where you can't authenticate on the device yourself, so using a key is more practical.
For example, a cloud server might use a one-off key to connect.
Reusable, for multiple uses. They can be used to connect multiple devices. For example, multiple instances of an on-premises database might use a reusable key to connect.
Be very careful with reusable keys! These can be very dangerous if stolen. They're best kept in a key vault product specially designed for the purpose.
Key expiry
An auth key automatically expires after the number of days you specified when you generated the key. You can choose the number of days, between 1 and 90 inclusive, for the key expiry. If you don't specify an expiry time, the auth key will expire after the maximum of 90 days. If you want to continue using an auth key after it expires, you need to generate a new key.
You can enable or disable key expiry on a device by using the Machines page of the admin console and by using the Update device key method in the Tailscale API.
If an auth key expires, any device authorized by it remains authorized until its node key expires. Each device generates a node key when you log in to Tailscale and uses it to identify itself to the tailnet. By default, node keys automatically expire every 180 days. You can change the default node key expiry from the Key Expiry section of the Device management page of the admin console.
Learn more about key management.
You can use tags as "service accounts" and have some devices NOT bound to any specific user (removal of user removes devices he own). You can define 1 or multiple tag owners (users managing tag).
Tag vs. user authentication
Tags are parallel to user authentication. They serve the same role as a user account, except they're intended for service-based devices, such as a web server or an app connector. As a result, it's impossible for a user account identity and a tag identity to exist on the same device. Applying a tag to a device previously authenticated with a user account removes the user account. Similarly, authenticating a device with a user account removes all tags from the device.
Because tags are intended for non-user devices, they have qualities and limitations that make them unsuitable for authenticating end-user devices, such as a MacBook or a mobile device. For example, devices with a tag-based identity cannot use SSH to connect to a device with a user-based identity.
Key expiry
When you apply a tag to a device for the first time and authenticate it, the tagged device's key expiry is disabled by default.
If you re-authenticate a device tagged before March 10, 2022, its expiry will be disabled by default.
If you change the tags on the device from the admin console, Tailscale CLI, or the Tailscale API, the device's key expiry will not change unless you re-authenticate. After you re-authenticate, the device's key expiry will be disabled.
You can enable or disable key expiry on a device from the Machines page of the admin console or by using the Tailscale API.
Key expiry for tagged devices
Key expiry for tagged devices is disabled by default. If you change the tags on the device through the admin console, Tailscale CLI, or Tailscale API, the device's key expiry will not change unless you re-authenticate. That is, if it is enabled, it stays enabled; and if it is disabled, it stays disabled. After you re-authenticate, the device's key expiry will be disabled.
You can find recently revoked or expired keys on the Keys page of the admin console.
Best practices
Depending on what devices you're authenticating, consider using an auth key that is:
Ephemeral, for authenticating ephemeral nodes as part of short-lived workloads. Because node keys do not persist when a workload restarts, they reconnect as a different node. Tailscale automatically removes inactive nodes. For example, containers or Lambda functions should use an ephemeral key to connect.
Pre-approved, for servers. If your tailnet has device approval enabled, this lets you add a device to your tailnet without further authorization. For example, shared devices, such as servers, should use a pre-approved auth key to connect in a network with device approval.
Pre-signed, for nodes whose auth keys are signed locally on a signing node, which applies to tailnets with Tailnet Lock enabled. You can make an auth key (created by any means) pre-signed only by using the tailscale lock sign CLI command.
Tagged, for servers. You can automatically apply a tag to a device by including the tag in the auth key. Access control policies restricting the device's permissions based on the tag apply after provisioning the device. For example, shared devices, such as servers, should use a tagged auth key to connect.
I am personally going to try Tagged (preauth) key and all my pfSense+ exit nodes (3) are already tagged as "router". So I will remove nodes from tailnet and re-add with Tagged key providing "router" tag.
I read that Tailscale on software upgrade MAY (rarely) REQUIRE device reauthentication. Having in mind that normal preauth keys expire after 90 days you should provide valid (non-expired) preauth key before pfSense+ upgrade? Eventually maybe if routers are tagged and preauth key is tagged then there will be no problem?
Currently pfSense+ does not ask us about expiration date of Preauth Key, so it can not remind us that key is expired and it may lead to problems. When valid key is needed (reauthentication) then device will fail with the message we see.
I would not blame pfSense+ yet, because I think Tailscale may require device reauthorisation sometimes and message you got tells you are trying to use expired key for authentication thus process is failing. I agree reauthentications should be rare or non-needed, but we may not know everything here.
