"Tailscale is not online" problem

chudak

Hello

My Tailscale somehow got messed up, I see "Tailscale is not online" error

The status show:

Error executing command (/usr/local/bin/tailscale status)
# Health check:
#     - not logged in, last login error=invalid key: API key does not exist

unexpected state: NoState

BTW the Services Status shows it green (seems like a UI bug)!

I will try to look into "API key does not exist", but wanted to share for the group.

Thx

chudak

@chudak said in "Tailscale is not online" problem:

API key does not exis

I regenerated Auth keys and got it online, but not local local hosts resolve when connected, interesting...

LHoust

I have had several occasions where the Tailscale installed on my pfSense CE (Zimaboard 432), failed to connect to my Tailnet following a pfSense Restart and it last night's instance the Tailscale Status was reporting my "API key does not exist"...

Although I tried Restarting the Tailscale Service a few times, my pfSense's Tailscale was failing to connect to my Tailnet...

Then I tried a Tailscale Service "Stop", followed by a Tailscale Service "Start" and curiously this time Tailscale was able to connect to my Tailnet!!!

Although my last resort would have been to generate a "New" Key, in this case I did NOT have to, since API key following the Stop/Start the API key did exist???

NOTE: Although I did have "Service Watchdog" running and Tailscale was selected with "Notification", it appears the Service was still running...

chudak

@LHoust

That happened again and was very disappointing.

The UI bug was never fixed :(

mcury

For those interested, I got a script here in the forum and changed it a little bit to work with tailscale and to check the connectivity every two minutes. It restarts the service if the ping fails.

#!/bin/sh

ALLDEST="headquarters"

COUNT=1
while [ $COUNT -le 2 ]
do

        for DEST in $ALLDEST
        do
                tailscale ping --c 1 $DEST >/dev/null 2>/dev/null
                if [ $? -eq 0 ]
                then
                        exit 0
                fi
        done

        if [ $COUNT -le 1 ]
        then
                /usr/local/sbin/pfSsh.php playback svc stop tailscale
#                /usr/local/sbin/pfSsh.php playback svc restart tailscale
                sleep 5
               /usr/local/sbin/pfSsh.php playback svc start tailscale
                sleep 10
        exit 1
        fi

        COUNT=`expr $COUNT + 1`
done

Then, chmod +x this script, install cron package and create a routine to run every 5 minutes, or 2 minutes, you choose.

If you improve this script somehow, please post the changes.

chudak

@mcury said in "Tailscale is not online" problem:

For those interested, I got a script here in the forum and changed it a little bit to work with tailscale and to check the connectivity every two minutes. It restarts the service if the ping fails.
#!/bin/sh

ALLDEST="192.168.10.1"

COUNT=1
while [ $COUNT -le 2 ]
do

        for DEST in $ALLDEST
        do
                tailscale ping --c1 --timeout 2 $DEST >/dev/null 2>/dev/null
                if [ $? -eq 0 ]
                then
                        exit 0
                fi
        done

        if [ $COUNT -le 1 ]
        then
                #/usr/local/sbin/pfSsh.php playback svc restart tailscale
                /usr/local/sbin/pfSsh.php playback svc stop tailscale
                sleep 5
                /usr/local/sbin/pfSsh.php playback svc start tailscale
                sleep 10
                exit 1
        fi

        COUNT=`expr $COUNT + 1`
done
Then, chmod +x this script, install cron package and create a routine to run every 5 minutes, or 2 minutes, you choose.

If you improve this script somehow, please post the changes.

Is ALLDEST="192.168.10.1" your TS IP?

mcury

@chudak said in "Tailscale is not online" problem:

Is ALLDEST="192.168.10.1" your TS IP?

This is the remote site LAN IP address (other pfsense).
This network is advertised through tailscale.

You can use any ip address that tailscale is expected to ping when it is online.

chudak

@mcury said in "Tailscale is not online" problem:

@chudak said in "Tailscale is not online" problem:

Is ALLDEST="192.168.10.1" your TS IP?

This is the remote site LAN IP address (other pfsense).
This network is advertised through tailscale.

You can use any ip address that tailscale is expected to ping when it is online.

It restarted TS for me all the time regardless of the condition

./restart_tailscale
Attempting to issue stop to tailscale service...

tailscale has been stopped.
Attempting to issue start to tailscale service...

tailscale has been started.

mcury

@chudak What is the output of the command: tailscale ping IP that you are trying to ping ?

chudak

@mcury said in "Tailscale is not online" problem:

@chudak What is the output of the command: tailscale ping IP that you are trying to ping ?

I think you are missing a value for "--c "

But the line tailscale ping --c 3 --timeout 2 XYZ still does not work for me :(

mcury

@chudak said in "Tailscale is not online" problem:

I think you are missing a value for "--c "

But the line tailscale ping --c 3 --timeout 2 XYZ still does not work for me :(

hmm, I think I may have provided the previous script, the one I was testing before deployment, let me search for the fixed one.

Edit: https://tailscale.com/kb/1080/cli#ping

You can ping a 100.x.x.x address or the node's name directly.
Also, if I remember correctly, try --c1(without space),

Edit2: script is updated in the previous post.

chudak

@chudak

UI but tracker in case someone wants to add comments
https://redmine.pfsense.org/issues/15319

mcury

@chudak said in "Tailscale is not online" problem:

UI but tracker in case someone wants to add comments

nice, good job for reporting that.

Did you manage to use the script ?

chudak

@mcury said in "Tailscale is not online" problem:

@chudak said in "Tailscale is not online" problem:

UI but tracker in case someone wants to add comments

nice, good job for reporting that.

Did you manage to use the script ?

I thought you were looking for a better version of the script.
Were you?

Did you see the Christian McDonald said it’s not a bug?
Apparently the current widget reports only about if the TS process running, no wonder it’s not useful

mcury

@chudak said in "Tailscale is not online" problem:

I thought you were looking for a better version of the script.
Were you?

I updated it, check above. Replace headquarters for the node's name.

@chudak said in "Tailscale is not online" problem:

Did you see the Christian McDonald said it’s not a bug?
Apparently the current widget reports only about if the TS process running, no wonder it’s not useful

I hope they include this in the next pfSense upgrade.

chudak

@mcury

The scripts worked, thx!

--timeout option seems to be broken. Did you get it to work?

cmcdonald

@mcury said in "Tailscale is not online" problem:

@chudak said in "Tailscale is not online" problem:

I thought you were looking for a better version of the script.
Were you?

I updated it, check above. Replace headquarters for the node's name.

@chudak said in "Tailscale is not online" problem:

Did you see the Christian McDonald said it’s not a bug?
Apparently the current widget reports only about if the TS process running, no wonder it’s not useful

I hope they include this in the next pfSense upgrade.

Maybe for the release after 24.03 I'll do a proper Tailscale widget. As I said in the Redmine, service status in pfSense can only report service running or not running. There are two Tailscale binaries, tailscale and tailscaled. tailscaled is the service and tailscale is the front end. By design tailscaled has to be running to receive commands from tailscale, even before it's authenticated and connected.

chudak

@cmcdonald

I hope they include this in the next pfSense upgrade.

Maybe for the release after 24.03 I'll do a proper Tailscale widget. As I said in the Redmine, service status in pfSense can only report service running or not running. There are two Tailscale binaries, tailscale and tailscaled. tailscaled is the service and tailscale is the front end. By design tailscaled has to be running to receive commands from tailscale, even before it's authenticated and connected.

Pls do!
This seems like a simple UI bug but in fact it’s very misleading.
I suspect that start/stop from the current widget also non functional

TIA

mcury

@chudak said in "Tailscale is not online" problem:

--timeout option seems to be broken. Did you get it to work?

The default is fine, 5s.
The "Tailscale is not online" issue happens pretty fast, so there is no need for the timeout option as I see it..

mathwilp1011

Hi Guys,

For anyone interested: here is the script that I used that is working 100%.

The --timeout 2 is not a flag within the tailscale CLI commands.

SUBCOMMANDS for Tailscale
up Connect to Tailscale, logging in if needed
down Disconnect from Tailscale
set Change specified preferences
login Log in to a Tailscale account
logout Disconnect from Tailscale and expire current node key
switch Switches to a different Tailscale account
configure [ALPHA] Configure the host to enable more Tailscale features
netcheck Print an analysis of local network conditions
ip Show Tailscale IP addresses
status Show state of tailscaled and its connections
ping Ping a host at the Tailscale layer, see how it routed
nc Connect to a port on a host, connected to stdin/stdout
ssh SSH to a Tailscale machine
funnel Serve content and local servers on the internet
serve Serve content and local servers on your tailnet
version Print Tailscale version
web Run a web server for controlling Tailscale
file Send or receive files
bugreport Print a shareable identifier to help diagnose issues
cert Get TLS certs
lock Manage tailnet lock
licenses Get open source license information
exit-node
update [BETA] Update Tailscale to the latest/different version
whois Show the machine and user associated with a Tailscale IP (v4 or v6)

Anyone has comments, please let leave them.

Note: you must make it executable with chmod +x and I just modified the above script to make it work for my use case. The tailscale node keeps on falling off (exit node unavailable) after either a reboot or it fails after a few days ofd being online. Added error checking display message.

@cmcdonald, this is still occurring in the 24.03 BETA (latest revision) as you are aware.

============
Script:

#!/bin/sh

ALLDEST="tailscaleexternalNODE"

COUNT=1
while [ $COUNT -le 2 ]
do
for DEST in $ALLDEST
do
tailscale ping --c 1 $DEST >/dev/null 2>/dev/null
if [ $? -eq 0 ]
then
echo "Tailscale is up"
exit 0
fi
done
if [ $COUNT -le 1 ]
then
echo "Tailscale down"
/usr/local/sbin/pfSsh.php playback svc stop tailscale
sleep 2
/usr/local/sbin/pfSsh.php playback svc start tailscale
sleep 10
echo "Tailscale is up"
exit 1
fi
COUNT=expr $COUNT + 1
done