"Tailscale is not online" problem

chudak

@mcury said in "Tailscale is not online" problem:

For those interested, I got a script here in the forum and changed it a little bit to work with tailscale and to check the connectivity every two minutes. It restarts the service if the ping fails.
#!/bin/sh

ALLDEST="192.168.10.1"

COUNT=1
while [ $COUNT -le 2 ]
do

        for DEST in $ALLDEST
        do
                tailscale ping --c1 --timeout 2 $DEST >/dev/null 2>/dev/null
                if [ $? -eq 0 ]
                then
                        exit 0
                fi
        done

        if [ $COUNT -le 1 ]
        then
                #/usr/local/sbin/pfSsh.php playback svc restart tailscale
                /usr/local/sbin/pfSsh.php playback svc stop tailscale
                sleep 5
                /usr/local/sbin/pfSsh.php playback svc start tailscale
                sleep 10
                exit 1
        fi

        COUNT=`expr $COUNT + 1`
done
Then, chmod +x this script, install cron package and create a routine to run every 5 minutes, or 2 minutes, you choose.

If you improve this script somehow, please post the changes.

Is ALLDEST="192.168.10.1" your TS IP?

mcury

@chudak said in "Tailscale is not online" problem:

Is ALLDEST="192.168.10.1" your TS IP?

This is the remote site LAN IP address (other pfsense).
This network is advertised through tailscale.

You can use any ip address that tailscale is expected to ping when it is online.

chudak

@mcury said in "Tailscale is not online" problem:

@chudak said in "Tailscale is not online" problem:

Is ALLDEST="192.168.10.1" your TS IP?

This is the remote site LAN IP address (other pfsense).
This network is advertised through tailscale.

You can use any ip address that tailscale is expected to ping when it is online.

It restarted TS for me all the time regardless of the condition

./restart_tailscale
Attempting to issue stop to tailscale service...

tailscale has been stopped.
Attempting to issue start to tailscale service...

tailscale has been started.

mcury

@chudak What is the output of the command: tailscale ping IP that you are trying to ping ?

chudak

@mcury said in "Tailscale is not online" problem:

@chudak What is the output of the command: tailscale ping IP that you are trying to ping ?

I think you are missing a value for "--c "

But the line tailscale ping --c 3 --timeout 2 XYZ still does not work for me :(

mcury

@chudak said in "Tailscale is not online" problem:

I think you are missing a value for "--c "

But the line tailscale ping --c 3 --timeout 2 XYZ still does not work for me :(

hmm, I think I may have provided the previous script, the one I was testing before deployment, let me search for the fixed one.

Edit: https://tailscale.com/kb/1080/cli#ping

You can ping a 100.x.x.x address or the node's name directly.
Also, if I remember correctly, try --c1(without space),

Edit2: script is updated in the previous post.

chudak

@chudak

UI but tracker in case someone wants to add comments
https://redmine.pfsense.org/issues/15319

mcury

@chudak said in "Tailscale is not online" problem:

UI but tracker in case someone wants to add comments

nice, good job for reporting that.

Did you manage to use the script ?

chudak

@mcury said in "Tailscale is not online" problem:

@chudak said in "Tailscale is not online" problem:

UI but tracker in case someone wants to add comments

nice, good job for reporting that.

Did you manage to use the script ?

I thought you were looking for a better version of the script.
Were you?

Did you see the Christian McDonald said it’s not a bug?
Apparently the current widget reports only about if the TS process running, no wonder it’s not useful

mcury

@chudak said in "Tailscale is not online" problem:

I thought you were looking for a better version of the script.
Were you?

I updated it, check above. Replace headquarters for the node's name.

@chudak said in "Tailscale is not online" problem:

Did you see the Christian McDonald said it’s not a bug?
Apparently the current widget reports only about if the TS process running, no wonder it’s not useful

I hope they include this in the next pfSense upgrade.

chudak

@mcury

The scripts worked, thx!

--timeout option seems to be broken. Did you get it to work?

cmcdonald

@mcury said in "Tailscale is not online" problem:

@chudak said in "Tailscale is not online" problem:

I thought you were looking for a better version of the script.
Were you?

I updated it, check above. Replace headquarters for the node's name.

@chudak said in "Tailscale is not online" problem:

Did you see the Christian McDonald said it’s not a bug?
Apparently the current widget reports only about if the TS process running, no wonder it’s not useful

I hope they include this in the next pfSense upgrade.

Maybe for the release after 24.03 I'll do a proper Tailscale widget. As I said in the Redmine, service status in pfSense can only report service running or not running. There are two Tailscale binaries, tailscale and tailscaled. tailscaled is the service and tailscale is the front end. By design tailscaled has to be running to receive commands from tailscale, even before it's authenticated and connected.

chudak

@cmcdonald

I hope they include this in the next pfSense upgrade.

Maybe for the release after 24.03 I'll do a proper Tailscale widget. As I said in the Redmine, service status in pfSense can only report service running or not running. There are two Tailscale binaries, tailscale and tailscaled. tailscaled is the service and tailscale is the front end. By design tailscaled has to be running to receive commands from tailscale, even before it's authenticated and connected.

Pls do!
This seems like a simple UI bug but in fact it’s very misleading.
I suspect that start/stop from the current widget also non functional

TIA

mcury

@chudak said in "Tailscale is not online" problem:

--timeout option seems to be broken. Did you get it to work?

The default is fine, 5s.
The "Tailscale is not online" issue happens pretty fast, so there is no need for the timeout option as I see it..

mathwilp1011

Hi Guys,

For anyone interested: here is the script that I used that is working 100%.

The --timeout 2 is not a flag within the tailscale CLI commands.

SUBCOMMANDS for Tailscale
up Connect to Tailscale, logging in if needed
down Disconnect from Tailscale
set Change specified preferences
login Log in to a Tailscale account
logout Disconnect from Tailscale and expire current node key
switch Switches to a different Tailscale account
configure [ALPHA] Configure the host to enable more Tailscale features
netcheck Print an analysis of local network conditions
ip Show Tailscale IP addresses
status Show state of tailscaled and its connections
ping Ping a host at the Tailscale layer, see how it routed
nc Connect to a port on a host, connected to stdin/stdout
ssh SSH to a Tailscale machine
funnel Serve content and local servers on the internet
serve Serve content and local servers on your tailnet
version Print Tailscale version
web Run a web server for controlling Tailscale
file Send or receive files
bugreport Print a shareable identifier to help diagnose issues
cert Get TLS certs
lock Manage tailnet lock
licenses Get open source license information
exit-node
update [BETA] Update Tailscale to the latest/different version
whois Show the machine and user associated with a Tailscale IP (v4 or v6)

Anyone has comments, please let leave them.

Note: you must make it executable with chmod +x and I just modified the above script to make it work for my use case. The tailscale node keeps on falling off (exit node unavailable) after either a reboot or it fails after a few days ofd being online. Added error checking display message.

@cmcdonald, this is still occurring in the 24.03 BETA (latest revision) as you are aware.

============
Script:

#!/bin/sh

ALLDEST="tailscaleexternalNODE"

COUNT=1
while [ $COUNT -le 2 ]
do
for DEST in $ALLDEST
do
tailscale ping --c 1 $DEST >/dev/null 2>/dev/null
if [ $? -eq 0 ]
then
echo "Tailscale is up"
exit 0
fi
done
if [ $COUNT -le 1 ]
then
echo "Tailscale down"
/usr/local/sbin/pfSsh.php playback svc stop tailscale
sleep 2
/usr/local/sbin/pfSsh.php playback svc start tailscale
sleep 10
echo "Tailscale is up"
exit 1
fi
COUNT=expr $COUNT + 1
done

chudak

@mathwilp1011

Today TS is again shows Tailscale is not online. Refresh or check the Tailscale status page.

The scrip says tailscale has been started.

But in fact TS is down :(

WTH

mcury

@chudak 4 months later, I think this time frame tell us something.
Problem should be with tailscale itself, or the other node..

chudak

@mcury said in "Tailscale is not online" problem:

@chudak 4 months later, I think this time frame tell us something.
Problem should be with tailscale itself, or the other node..

Frankly, there is nothing to update and I still did not get to the bottom of it.

TS sometimes is up and running and very stable for long periods. And then it gets flaky and can't connect.

I do run "restart_tailscale" script in crontab, so I assume it makes it start.

But in general, I am puzzled ...

Any clues?

mcury

@chudak said in "Tailscale is not online" problem:

But in general, I am puzzled ...

Any clues?

Any logs when the problem starts ?
What happens when you try to ping the other node ?
What status it shows in the GUI ?

I have been using that script with the following scenario, which works fine:

I have a customer that runs multi WAN in their headquarters.
One of this links is a CGNAT and other is not.

The branch office connects directly to the primary non CGNAT link (I have opened a port in the firewall for that connection).
If a link failover happens in the headquarters, sometimes it loses connections to the TS network and that's when the script "fixes" the problem by forcing the headquarter firewall to restart the service but now using the CGNAT link, thus connecting through the TS node and not a directly connection anymore.

The reverse is also true, I mean, when the primary link which is not CGNAT comes back online.

chudak

@mcury

I don't know exactly where to look :(

On a high level, only what I see is TS service is green (which is confusing but that's in a different thread and unrelated) and TS connection status is down.

My use case is:

pfS runs TS
iPad runs TS
iPhone runs TS
Windows 11 VM runs TS

So when psF is down, others actually work fine.
I even noticed that routes get resolved.