How to run sh or php script for filer or cron

wakson005

I code a good portion of the script but can't tell why it doesn't run is there a way to run portion of it?
Here are the few things i think maybe the issue:

Dont know how to run this script or where to run it
Not sure if the code will work or not but the idea is there.

If there are tutorial on how to run these scripts let me know please on a good read. Also would like to access the code by gui to do things automated too.

#!/usr/local/bin/php-cgi -f

<?php
require_once("config.inc");     // Include the pfSense configuration file

// Interface name of the CARP interface OpenVPN is connected to
$carpInterface = 'xn6';

// VPN Interfaces that will be disabled when going to backup mode
$ovpns00Interface = 'ovpns18';
$ovpns01Interface = 'ovpns19';

// Set the delay between each check (in seconds)
$delay = 5;
// Delay before bringing up the VPN Interfaces (need to wait until the VPN tunnel switch over properly)
// If interfaces are enabled too early tunnel will fail.
$tapVPN_UpDelay = 15;

while (true) {
    // Check the status of the CARP interface
    $carpStatus = exec('/sbin/ifconfig ' . $carpInterface);

    // Detect when the CARP interface changes from master to BACKUP
    if (strpos($carpStatus, 'BACKUP') === 0) {
        // Execute the necessary action when the CARP interface changes from master to backup
        exec('/sbin/ifconfig ' . $ovpns00Interface . ' down'); // Disable VPN interface on master
        exec('/sbin/ifconfig ' . $ovpns01Interface . ' down'); // Disable VPN interface on master

        // Additional actions to be performed when the CARP interface changes from master to backup
        // ...
    }
    // Detect when the CARP interface changes from backup to MASTER
    elseif (strpos($carpStatus, 'MASTER') === 0) {
        // Delay before bringing up the VPN Interfaces
        sleep($tapVPN_UpDelay);

        // Execute the necessary action when the CARP interface changes from backup to master
        exec('/sbin/ifconfig ' . $ovpns00Interface . ' up'); // Enable VPN interface on master
        exec('/sbin/ifconfig ' . $ovpns01Interface . ' up'); // Enable VPN interface on master

        // Additional actions to be performed when the CARP interface changes from backup to master
        // ...
    }

    // Delay before the next check
    sleep($delay);
}
?>

stephenw10

You mean it runs but doesn't do anything or it fails to run at all?

You shouldn't need to do that though. If the OpenVPN servers are running on the CARP VIP they should already be stopped and started automatically.

wakson005

@stephenw10 Thanks a lot for the reply. Yep both TAP and TUN reference the CARP IP. This is for the OpenVPN TAP side. The OpenVPN TUN works fine so i can access both end without problem. Sadly the TAP works too but the failover doesn't reconnect to the backup server unless i manually do the following:

Turn OFF VPN TAP interfaces
Restart the VPN TAP server/client that is not working
Then turn ON the VPN interface used in the bridge

Manually I can do it but would like to have a script to do that automatically. Sadly in PFSense there is no solution for that yet in the GUI as far as i can tell. If there is please let me know.

As far as I can tell there are issue with HA with Bridging (Even worst this is HA Bridging with OpenVPN):

FYI I don't know why it doesn't work in the "Filer" location so i decided to rewrite above slightly differently and with approvement in shell script .sh and it work much better still need to test why the connections is up and switch is occurring but tunnel cannot be reach on other side... It appear as if during the switch between MASTER to BACKUP and vise versa the script is triggered multiple time...

# Variables for timing and interface names
carp_interface_vpn_tied_to="xn6"   # Carp interface that is bridged with the VPN Tap Interface
wait_time=30                       # Time until all tap/tunnels are expected to be up on BACKUP
check_interval=1                   # Script carp check interval

# Function to check the CARP interface status
check_carp_interface_status() {

	# Wait for the xn6 interface to become available
	while ! ifconfig "$carp_interface_vpn_tied_to" >/dev/null 2>&1; do
		sleep 1
	done

    # Get the CARP status for the xn6 interface
    carp_status=$(ifconfig "$carp_interface_vpn_tied_to" | grep 'carp:')

    if [ -n "$carp_status" ]; then
        if echo "$carp_status" | grep -q 'carp: MASTER'; then
            # Actions to perform when the interface is the master
            echo "$carp_interface_vpn_tied_to interface is operating as the master. Taking action..."
            # Add your desired actions here

            # Sleep for specified time until all tap/tunnels are expected to be up on BACKUP
            sleep "$wait_time"

            # Bring up ovpns18 and ovpns19 interfaces on the backup node
            if [ -f /usr/local/bin/enable_ovpns_tap.txt ]; then
                /usr/local/sbin/pfSsh.php < /usr/local/bin/enable_ovpns_tap.txt
            else
                echo "enable_ovpns_tap.txt not found. Please check the file path."
            fi
        else
            # Actions to perform when the interface is the backup
            echo "$carp_interface_vpn_tied_to interface is operating as the backup. Taking action..."
            # Add your desired actions here

            # Bring down ovpns18 and ovpns19 interfaces on the master node
            if [ -f /usr/local/bin/disable_ovpns_tap.txt ]; then
                /usr/local/sbin/pfSsh.php < /usr/local/bin/disable_ovpns_tap.txt
            else
                echo "disable_ovpns_tap.txt not found. Please check the file path."
            fi
        fi
    else
        # Check if xn6 interface exists
        if ifconfig "$carp_interface_vpn_tied_to" &> /dev/null; then
            echo "$carp_interface_vpn_tied_to interface exists but CARP is not configured."
            # Add your desired actions here
        else
            echo "$carp_interface_vpn_tied_to interface does not exist or is not configured as a CARP interface."
            # Add your desired actions here
        fi
    fi
}

# Infinite loop to continuously check the CARP interface status
while true; do
	# Call the function to check the CARP interface status
	check_carp_interface_status
    sleep "$check_interval"
done

stephenw10

Ah, yes bridging and HA is usually better avoided. However in this case you wouldn't hit issues with loops since the two OpenVPN TAP instances are not connected.

Seems like you might be better running the TAP server on localhost with forwarding so both sides are always up?

wakson005

@stephenw10 Just to clarify as I wasn't being too clear. I do have two tap OpenVPN TAP up and they actually are bridged but each TAP is to a different endpoint. So looping most likely is occurring (not sure how i would check that but will look into it somehow). So i enable Spanning Tree to help but not sure it is helping much.

So to clarify my setup I have the following:
Site0
Server:
Site0 to Site1 TAP
Site1 to Site2 TAP
Client: N/A
both interface bridged.

Site1
Server:
Site1 to Site2 TAP
Client:
Site0 to Site1 TAP
both interface bridged.

Site2
Server: N/A
Client:
Site0 to Site1 TAP
Site1 to Site2 TAP
both interface bridged.

That's how TAP connection work between all 3 sites. It works fine except during HA failover. So each site therefore has a Master and Backup PFSense. It is during failover I have issue. I follow similar concept for my TUN and TUN at all 3 to N location has flawless failover. But TAP requires manual intervention to deal with it. I do have script to turn off interfaces and back on but I can't find a script option to restart the VPN client/server as if we are pressing the play button:

Script i thought to turn on the VPN TAP as if pressing play didn't seem to work:

/usr/local/sbin/pfSsh.php playback svc start 21
// or
/usr/local/sbin/pfSsh.php playback svc start ovpns21

only show something like this for both case:

But when i manually press the button it switch
FROM:

TO:

without any issue.

Does anyone know the actually script to press the play button to turn it on?

stephenw10

Hmm, so at all three sites the two TAP tunnels are bridged together and bridged to a local interface?

wakson005

@stephenw10 Yes the 2 TAP tunnels at each of the 3 site are bridge to a single LAN interface. therefore the same LAN subnet are all bridge together. All DHCP range are different just subnet are the same as that was the hard requirement :( if not I would have stick with TUN as it was easier and works flawlessly between the 3 sites each with HA. I am able to ping do iperf from one pfsense to another without issue like this.

Its difficult as the TAP connection and iperf work between then though experience high retr when doing iperf but they are all up and running just during HA it never switch over flawlessly...

stephenw10

Hmm, I mean just to be clear those TAP tunnels are effectively in a mesh between the sites? I sounds like a L2 loop is inevitable without something in place to prevent it. STP on the bridges perhaps.

You would certainly need to have them run on the VIPs to avoid further loops between the HA nodes in that case.

wakson005

@stephenw10 Yea they are in a mesh between site (meaning all site share the same subnet here. Just the DHCP server handoff is a different range of that same subnet.). I do have RSTP on the bridge and put rules in place to for source to destination on each bridge hopefully to help with that but still some weird issue. Could be my setup is not correct. This is a site to site setup and I haven't find a concrete guide on the setup. Do you know if anyone being able to successfully set this up. I understand there is no guide for this and think PFSense doc said it is not recommended probably for a reason i think lol. If not i will just stick with the manual process for now until future improvement is added.

with Tunnel setting all blank beside the Tunnel Network. All else below this is left as default:

Client everything under Tunnel Network all blank and left as default.

stephenw10

Hmm, well I would expect that to work but the addition of HA makes things.... interesting.

I would expect to see some errors logged on the secondary node when it fails over. Basically I don't expect to need a script there.

wakson005

@stephenw10 Ok i have identified the issue in more detail. When the bridge interface is enabled the Failover fails. I need to disable TAP interface on both Site 0 HA1 and HA2 and Site 1 HA1 and HA 2, same for Site 2 end for it to work properly. When all TAP interface is disable failover work flawlessly. Trickly part is the timing of this which is when failover occurs all the TAP interface needs to be disable and re-enable after the TAP connection is automatically re-established. When doing failover if all the TAP interface is disable you can do as many failover as you want "using maintenance mode" without breaking the connection. Only turn on TAP interface after the failover is completed and all VPN TAP are up.

So TAP interface causing issue as that connection probably disappear and it doesn't work... wonder what makes this different from the TUN case for the interface viewpoint.

wakson005

@stephenw10 Really appreciate your guidance :) . Oh so you confirm that it definitely work by you or other people before? If script is not needed is there a step by step guidance somewhere for this?

stephenw10

Hmm, how does the failover fail when TAP is enabled? Like it actually doesn't switch nodes?

wakson005

@stephenw10 Just did multiple test on it and notice that the interfaces either disappear or becomes down and never turn back on. I did a ifconfig and it gives me much more detail. The interface disappear or got rename for some reason which i thought is weird.

@stephenw10 How does the failover fail when TAP is enabled? (There is the TAP VPN Server, Client, and Interface all 3 different things)
Answers: Just to be as clear as possible for others beside me and Stephenw10 reading the "TAP VPN Server" when enable work fine and failover work flawlessly IF the TAP interfaces is disabled. So me claiming failover not working in general is probably not 100% completely true as the VIP IP failover for the LAN network work flawlessly. This is the LAN network used in the Bridge connection with the VPN. As the bridge, LAN, VIP creation doesn't directly impact TAP VPN connections it works fine. But IF TAP interfaces are enable there is a high chance of failing (maybe because the original master is holding onto the connection. Interfaces start disappearing or up status becomes down and bridge loses the interfaces that was part of the bridge. Bridge sometime don't have the TAP interfaces anymore as it is down and doesn't register it again when it is up later on.)

Things that work:

Failover for VIPs master to backup is working great for TAP and TUN. (But this is just the VIP IP that work for the failover doesn't mean the connections for the TAP still work.)

Issues notices during the switch IF interfaces is not down for the failover:

Interface disappear from ifconfig (worst case there is a new interface called tap## which used to be ovpns## which is definitely the more weirder case...)
Interface is not part of the bridge anymore
system log or openvpn log was not too helpful only show link up, link down, fatal error... (maybe i can try a higher lv log > default to get more status...)

Ways to resolve the issue manually after all those issue appears and it work almost every time:

Turn off all TAP interfaces.
Reset only TAP interfaces that has issues (Notice that the status is not reporting correctly on the gui as ifconfig status don't match with gui. Example gui show up and ifconfig show interface down [without the up].)
Turn back on the TAP interface and everything is back to normal.

I conclude that sh script with "config interface down/up" wont be enough to resolve the issue. Same with php script to enable/disable interface is not enough too. The TUN seem to be doing much more than turning on and off the interfaces. If i do that for TAP i am definitely missing some key component in the script. Manually changing does more than just turn on/off the interfaces it actually reset the bridge, interfaces, and routes in some way i believe that's probably why it work but not with script.

So as far as i can tell there is no perfect solution yet for TAP use TUN if possible as its faster and more reliable unless absolutely necessary like poor me where I have to use it no matter what for a share subnet across both site.

Thanks!

In the mean time if others have ideas i would like to try :)

wakson005

@wakson005 is this the same as pressing the vpn restart? "/usr/local/sbin/pfSsh.php playback svc restart openvpn server Server1" if so what do i need to put into Server1 is it "S00000C00001TAP00" or "ovpns18" or "18" or "Server 18" same how do i do this for client. Though client might be fine.

Like which one is the correct one to run as it just said run

Like i tried "/usr/local/sbin/pfSsh.php playback svc stop 18" and got back

but gui shows:

and status stayed the same in gui which makes me think gui is not updating as script doesn't update gui like all the other cases i seen for interfaces.

Think the above will get me many step closer to solution as restart of vpn need to be done per TAP interface based as TUN is working i don't want to touch those.

wakson005

ok for vpn restart, start, stop refer to:
https://forum.netgate.com/topic/176435/disable-openvpn-clients-on-reboot/3

will try this with my current code hopefully should fix lots of my issues i think as this was probably the key ingredient i was missing...

stephenw10

Yup you would use: pfSsh.php playback svc restart openvpn server 18

As shown:

Netgate pfSense Plus shell: playback svc

Playback of file svc started.

Usage: playback svc <action> <service name> [service-specific options]

Examples:
playback svc stop dhcpd
playback svc restart openvpn client 2
playback svc stop captiveportal zone1

wakson005

@stephenw10 Thanks that resolved my issue :) as it let me restart the openvpn server and client perfectly. Final testing prior to calling everything fool proof.

wakson005

@stephenw10 Script is suppose to running continuously and checking carp for when the master to backup transition occurs.

Script work fine when i do the following:
DiagnosticsCommand>Prompt>Execute Shell Command and enter:
/usr/local/bin/openvpn_server_client_tap_auto_failover.sh

Issue is this forever loop stop at some point as I think it is not meant running forever until shutdown.
Tried moving .sh script to:
/usr/local/etc/rc.d/openvpn_server_client_tap_auto_failover.sh
and it causes it to trigger multiple times for some reason as if it reset itself and run.

Is there somewhere to run sh script at boot up and let the loop run forever until shutdown? Restarting the script doesn't work as it stores a temporary state of what the carp state previously so it know to reset or not reset. If script start up running every time it will reset as it assume carp status changes.

stephenw10

Can you see what's killing the script?