[Bug + Patch] tailscale0 loses "Tailscale" interface group after reboot / service restart

timo254

Summary

On pfSense Plus / CE with pfSense-pkg-Tailscale installed, tailscale0 sometimes
comes up after a reboot without the "Tailscale" interface group, leaving it
only in the "tun" group. Any firewall rules, NAT rules, aliases, or floating
rules that reference the Tailscale interface group silently stop matching
traffic.

Running ifconfig tailscale0 group Tailscale by hand restores the group until
the next reboot. service tailscaled restart && service pfsense_tailscaled restart reproduces the same broken state.

Tested on: pfSense Plus 26.03-RELEASE with pfSense-pkg-Tailscale.

Root cause

/usr/local/etc/rc.d/pfsense_tailscaled has this poststart sequence:

pfsense_tailscaled_poststart() {
    # wait for the tun device to come up
    wait_for_tun_dev ${tailscale_tun_dev_wait} || return 1

    # add the tun device to the interface group
    tun_dev_add_group          # <-- group added HERE

    # bring up Tailscale
    tailscale_up               # <-- 'tailscale up' called AFTER

    # reload the packet filter
    pfsense_reload_filter
}

The order is wrong. tun_dev_add_group applies the Tailscale group, then
tailscale_up is called. When tailscale up reconnects to the control
plane, the daemon can reset the interface state as part of bringing the
tunnel up, wiping the group membership that was just set. On boxes where
the tailscale up reconnect is fast/uneventful, the group sticks; on
boxes where it actually re-configures the interface, the group is lost.

Reproduction

Any of these will reproduce the broken state on an affected box:

reboot
service tailscaled restart && service pfsense_tailscaled restart
service pfsense_tailscaled restart

After the command:

# ifconfig tailscale0 | grep groups
groups: tun                          <-- missing "Tailscale"

Proposed patch

Add a second tun_dev_add_group call AFTER tailscale_up so the group is
re-applied once tailscale_up has finished reconnecting. ifconfig ... group
is idempotent, so running it twice is harmless. The tun_dev_add_group
function is already defined earlier in the same file, no new code needed.

--- /usr/local/etc/rc.d/pfsense_tailscaled.orig
+++ /usr/local/etc/rc.d/pfsense_tailscaled
@@ pfsense_tailscaled_poststart() {
     # wait for the tun device to come up
     wait_for_tun_dev ${tailscale_tun_dev_wait} || return 1

     # add the tun device to the interface group
     tun_dev_add_group

     # bring up Tailscale
     tailscale_up

+    # re-apply the interface group after tailscale_up, which can reset
+    # interface state and drop group membership
+    tun_dev_add_group
+
     # reload the packet filter
     pfsense_reload_filter
 }

Verification

After applying the patch, a reboot should log:

Apr 20 10:34:28 pfSense tailscale[1923]:  Added tailscale0 to interface group Tailscale
Apr 20 10:34:28 pfSense tailscale[14426]: Added tailscale0 to interface group Tailscale

The two "Added tailscale0 to interface group" lines confirm both calls ran.
ifconfig tailscale0 | grep groups should reliably show:

groups: tun Tailscale

Related note

In the same file on my pfSense, the --auth-key line is currently shipped commented out:

# handle the --auth-key parameter
# pfsense_tailscaled_up_flags="--auth-key=${pfsense_tailscaled_authkey}"

This is correct and should stay commented for installs that have already
enrolled the node (re-applying the auth-key on every tailscale up causes
issues with non-expiring keys). Flagging this only because a conditional
form would be more robust:

if [ -n "${pfsense_tailscaled_authkey}" ]; then
    pfsense_tailscaled_up_flags="--auth-key=${pfsense_tailscaled_authkey}"
fi

That way the flag is applied only when an auth-key is actually configured,
and the line can be left uncommented in the shipped file.

Happy to provide any additional logs / testing if helpful.

timo254

For now ive resorted to adding a file under: /usr/local/etc/rc.d/tailscale_fix_group
It is called tailscale_fix_group, it ensures tailscale0 is always a member of the 'Tailscale' interface group after pfsense_tailscaled starts.

Code in tailscale_fix_group below, you can add it from GUI. Just open edit file and save path: /usr/local/etc/rc.d/tailscale_fix_group

Paste the code below and press save.

#!/bin/sh

. /etc/rc.subr

name="tailscale_fix_group"
start_cmd="tailscale_fix_group_start"
stop_cmd=":"

tailscale_fix_group_start() {
    local _iface="tailscale0"
    local _group="Tailscale"
    local _waited=0
    local _max=30

    while [ $_waited -lt $_max ]; do
        if /sbin/ifconfig ${_iface} >/dev/null 2>&1; then
            if /sbin/ifconfig ${_iface} | grep -qw "${_group}"; then
                logger -s -t tailscale "tailscale_fix_group: ${_iface} already in group ${_group}"
                return 0
            fi
            /sbin/ifconfig ${_iface} group ${_group} >/dev/null 2>&1 && {
                logger -s -t tailscale "tailscale_fix_group: added ${_iface} to group ${_group}"
                return 0
            }
        fi
        sleep 1
        _waited=$((_waited + 1))
    done

    logger -s -t tailscale "tailscale_fix_group: ERROR could not add ${_iface} to group ${_group} after ${_max}s"
    return 1
}

run_rc_command "$1"

marcosm

I've added a patch for testing here:
https://redmine.pfsense.org/issues/17034#note-5

keyser

@marcosm said in [Bug + Patch] tailscale0 loses "Tailscale" interface group after reboot / service restart:

I've added a patch for testing here:
https://redmine.pfsense.org/issues/17034#note-5

I have also suffered from this problem in a larger tailscale testsetup that I have, and this patch is really welcome.
I hope this can quickly lead to a new build of the Tailscale package that includes the patch?
If my setup goes into production I will have 100+ pfSenses that will have Tailscale installed, and it’s not a solution to have a required manual patch to apply on every one of them.

enhance_rethink2180

@marcosm said in [Bug + Patch] tailscale0 loses "Tailscale" interface group after reboot / service restart:

I've added a patch for testing here:
https://redmine.pfsense.org/issues/17034#note-5

I can confirm this on pfSense Plus 26.07-RELEASE (amd64), and I have data that extends the scope of the report.

The group is also lost without a reboot, a service restart, or a re-auth. On my box tailscaled recreates the tun device on its own during normal operation, and in that path nothing ever re-applies the interface group, because pfsense_tailscaled_poststart is not involved at all.

System log, showing the difference between the two paths:

Self-initiated tunnel recreation (no reboot, no service restart) — group is NOT re-applied:

Sep 5 19:17:39 kernel: tailscale0: link state changed to DOWN
Sep 5 19:17:39 kernel: tun0: changing name to 'tailscale0'
Sep 6 04:06:11 kernel: tailscale0: link state changed to DOWN
Sep 6 04:06:11 kernel: tun0: changing name to 'tailscale0'
Sep 6 08:21:52 kernel: tailscale0: link state changed to DOWN
Sep 6 08:21:52 kernel: tun0: changing name to 'tailscale0'

Manual service restart — group IS re-applied:

Sep 3 07:13:52 php-fpm: NOTICE The command '/usr/local/etc/rc.d/pfsense_tailscaled stop' returned exit code '1'
Sep 3 07:13:52 kernel: tailscale0: link state changed to DOWN
Sep 3 07:13:52 kernel: tun0: changing name to 'tailscale0'
Sep 3 07:13:53 tailscale: Found device tailscale0
Sep 3 07:13:53 tailscale: Added tailscale0 to interface group Tailscale
Sep 3 07:13:53 tailscale: Bringing up tailscale0 with --auth-key=... --advertise-exit-node --accept-routes --accept-dns

The "Added tailscale0 to interface group Tailscale" line only ever appears when the rc script runs. After a self-initiated recreation it is absent, and the interface stays in the tun group only.

State in the broken condition:

$ ifconfig tailscale0 | grep groups
groups: tun

Everything else looks healthy, which is what makes this hard to spot:

$ tailscale status
100.x.x.x net-router-1 user@ freebsd -
100.x.x.x nas-host user@ linux -

tailscale netcheck is clean (UDP works, MappingVariesByDestIP: false, nearest DERP 5.7ms), the connection is direct rather than relayed, and the daemon is running. From the firewall itself the tailnet is reachable; from LAN it is not. The GUI reports Tailscale as online throughout.

This happens roughly every 1-2 days. The only visible correlation is the link-state-down / rename pair above. gateways.log shows no gateway events anywhere near these timestamps, so gateway monitoring is not involved.

Versions affected: pfSense-pkg-Tailscale 0.1.9_2 and 0.1.10 (the 0.1.10 update did not change this behaviour), tailscale 1.98.5_1, pfSense Plus 26.07-RELEASE.

The workaround is the same as reported, plus a filter reload, otherwise the rules do not pick up the change:

ifconfig tailscale0 group Tailscale
/etc/rc.filter_configure

Regarding the proposed fix: calling tun_dev_add_group again after tailscale_up returns would address the reboot / restart / re-auth cases, but it would not cover the case above, since the rc script never executes when the daemon recreates the device by itself. A mechanism that re-applies the group whenever tailscale0 appears — a devd hook or a periodic check — would be needed to cover all paths.

Happy to provide further logs or test a patch.

keyser

@enhance_rethink2180 so just to be clear - your’e seeing this issue with the patch/update applied as well? So we need another patch/update that makes sure the interface stays in the group during those changes you observe?

Hope @marcosm can fix this as they did with the initial issue.

enhance_rethink2180

@keyser said in [Bug + Patch] tailscale0 loses "Tailscale" interface group after reboot / service restart:

REBEL ALLIANCE
about an hour ago

@enhance_rethink2180 so just to be clear - your’e seeing this issue with the patch/update applied as well? So we need another patch/update that makes sure the interface stays in the group during those changes you observe?

No, I have not applied the patch. I am running the stock package, first 0.1.9_2 and now 0.1.10, on 26.07-RELEASE.

I did not apply it because in my case it should not make a difference. As I read it, the patch adds a second tun_dev_add_group call after tailscale_up returns inside pfsense_tailscaled_poststart. That covers the paths where the rc script actually runs: boot, service restart, re-auth. My failures happen at runtime, with no reboot, no service restart and no re-auth, when tailscaled tears down and recreates the tun device by itself. The rc script is never invoked in that path, so there is nothing for the patch to fix there.

What I see in the log is the device going down and tun0 being renamed to tailscale0, with no "Added tailscale0 to interface group Tailscale" line following it. After that the interface is in the tun group only, and anything keyed on the Tailscale group stops matching. A manual service restart does log that line, which is why restarting the service appears to fix it.

So yes, to your second question: what is needed is something that re-applies the group whenever tailscale0 appears, independently of the rc script. A devd hook on interface arrival, or a periodic check, would cover all paths including mine.

To be fair, this is reasoning from the code path and my logs, not something I have tested.

keyser

@enhance_rethink2180 Okay - I have just deployed a test box where I will attempt to verify that this will happen eventually when the tunnel is recreated - I don't know how long/when that happens, so I'll have to wait.

I think the updated 1.10 package includes the patch Marcosm created earlier - so there is no need to apply that if you use the updated package.

marcosm

@enhance_rethink2180 said in [Bug + Patch] tailscale0 loses "Tailscale" interface group after reboot / service restart:

Happy to provide further logs or test a patch.

You can test the latest version of the package

The fix I submitted is not the proposed patch from this thread.