new if_pppoe Backend - getting HA/CARP to work like in MPD
-
Hello everyone,
I have a question regarding HA / CARP when using the new pppoe backend.
We are using a pair of PfSense+ 25.07.1 firewalls as QEMU / Proxmox guests in a HA environment. Our connection is a 1G/1G FTTH.Currently our working setup with MPD is:
Both Firewalls (MASTER/BACKUP) are having a single vtnet0 interface only as a trunk only. all traffic needs to be tagged within VLANS
For the HA WAN configuration we use:
- Interface WAN_PHY is attached to a VLAN (501)
- on that interface we are using a CARP VIP with private IP Adresses unused in our environment (10.34.56.254)
- on top of that CARP IP we created the pppoe0 interface
- that interface is bound to WAN with WAN having a MTU and MSS of 1492
With this configuration we have zero issues with MLD and failover just works within seconds. The rationale for that setup is that only the master firewall has the pppoe connection and the connection is switched quickly when a master/slave failover occurs.
When switching to if_pppoe the WAN interface won't connect at all - most probably due to the "quirky but working perfectly" configuration. Binding the PPP towards the VLAN interfaces makes the interface come up and work as expected.
So, my question would be: is there any way to support HA/CARP using if_pppoe and making sure that in a failover case the connection is switched quickly?
I could not find any migration steps to be done to convert our setup to be usind if_pppoe in the docs.
Thanks for your help.
-
@perrin
https://redmine.pfsense.org/issues/16174
I don’t know if this will ever be implemented. For now, I solved the issue with scripts, but they’re so ugly and clumsy…
-
Thanks for the hint towards redmine. Yeah, having priority very low for that issue makes me think that they won't implement a feature like this.
However implementing it the same way as it worked before might not be a good solution. Better would be some sort of GUI option to have the wan interface enable or disable itself depending on the status of a carp interface.I believe that is probably what you've done with your scripts. Attaching your script to the devd carp event and depending on the event do an ifup or ifdown on the wan interface.
Would you mind sharing those scripts? If I find the time maybe I'd create an addon package containing that logic so it is available to everyone else having this issue.
-
@perrin said in new if_pppoe Backend - getting HA/CARP to work like in MPD:
Would you mind sharing those scripts?
Sure. Actually I use only one but on both firewalls.
/usr/local/etc/rc.d/run.sh 0755
#!/bin/sh ############################################################################### ############################################################################### # ──────────────────────────────── # Tunables # ──────────────────────────────── LOCKFILE="/var/run/run.sh.lock" # stores PID + role PPPOE_IF="pppoe0" # logical PPPoE interface LAN_IF="ix0" # CARP LAN interface MASTER_VIP_VHID_IPv4_LAN="MASTER vhid 5" # grep token for MASTER PHP_BIN="/usr/local/bin/php" # path to php on pfSense PPPOE_ALREADY_STARTED="false" # runtime flag ############################################################################### # Exit if another copy is already running ############################################################################### check_already_running() { if [ -f "$LOCKFILE" ]; then # shellcheck disable=SC2034 read -r LOCKPID PREV_ROLE < "$LOCKFILE" if [ -n "$LOCKPID" ] && ps -p "$LOCKPID" -o pid= >/dev/null 2>&1; then echo "Another instance is active (PID $LOCKPID) – aborting." exit 1 fi fi } ############################################################################### update_lockfile() { printf "%s\n%s\n" "$$" "$1" > "$LOCKFILE"; } ############################################################################### ############################################################################### # Optional helper ############################################################################### find_pppoe_info() { INTERFACE=$(ifconfig | awk '/pppoe[0-9]+:/ {sub(":", "", $1); print $1; exit}') IP_ADDRESS=$(ifconfig "$INTERFACE" | awk '$1 == "inet" {print $2}') } ############################################################################### # Main monitoring loop ############################################################################### start_monitoring() { echo "Monitoring CARP role (logs only when role flips)" logger "PPPoE_script_message001: CARP watchdog launched" CUR_ROLE="" [ -f "$LOCKFILE" ] && read -r _ CUR_ROLE < "$LOCKFILE" while true; do sleep 30 if ifconfig "$LAN_IF" | grep -q "$MASTER_VIP_VHID_IPv4_LAN"; then NEW_ROLE="MASTER" else NEW_ROLE="BACKUP" fi # No change → continue [ "$NEW_ROLE" = "$CUR_ROLE" ] && continue # Role changed → act if [ "$NEW_ROLE" = "MASTER" ]; then handle_master_carp logger "PPPoE_script_message002: CARP became MASTER – PPPoE handling executed" else handle_non_master_carp logger "PPPoE_script_message003: CARP became BACKUP – PPPoE handling executed" fi CUR_ROLE="$NEW_ROLE" update_lockfile "$CUR_ROLE" done } ############################################################################### # MASTER logic ############################################################################### handle_master_carp() { if [ "$PPPOE_ALREADY_STARTED" != "true" ]; then handle_pppoe_start else echo "PPPoE already started – verifying link" logger "PPPoE_script_message004: PPPoE already started – verifying link" check_pppoe fi } ############################################################################### # BACKUP logic # – Shuts down PPPoE if present # – Restores VHID 5 if it vanished ############################################################################### handle_non_master_carp() { # 1) Bring PPPoE down local existing_pppoe existing_pppoe=$(ifconfig | awk '/pppoe[0-9]+:/ {sub(":", "", $1); print $1; exit}') if [ -n "$existing_pppoe" ]; then sleep 10 ifconfig "$PPPOE_IF" down PPPOE_ALREADY_STARTED="false" logger "PPPoE_script_message005: PPPoE interface DOWN on BACKUP" fi # 2) Ensure VHID 5 still exists – if not, re-install VIPs if ! ifconfig "$LAN_IF" | grep -q "vhid 5"; then logger "PPPoE_script_message006: VHID 5 missing on BACKUP – reconfiguring VIPs" $PHP_BIN -r 'require_once "/etc/inc/interfaces.inc"; interfaces_vips_configure();' fi } ############################################################################### # Bring PPPoE UP (run when we just became MASTER) ############################################################################### handle_pppoe_start() { sleep 130 # allow CARP convergence local existing_pppoe existing_pppoe=$(ifconfig | awk '/pppoe[0-9]+:/ {sub(":", "", $1); print $1; exit}') if [ -z "$existing_pppoe" ]; then logger "PPPoE_script_message007: No PPPoE interface – reloading WAN via pfSctl" /usr/local/sbin/pfSctl -c 'interface reload wan' PPPOE_ALREADY_STARTED="true" else if ifconfig "$PPPOE_IF" | head -n1 | grep -q '<.*UP'; then logger "PPPoE_script_message008: PPPoE '${PPPOE_IF}' already UP – nothing to do" else ifconfig "$PPPOE_IF" up logger "PPPoE_script_message009: PPPoE '${PPPOE_IF}' brought UP" fi fi } ############################################################################### # Verify PPPoE connectivity – reload WAN if iface vanished ############################################################################### check_pppoe() { sleep 180 local iface iface=$(ifconfig | awk '/pppoe[0-9]+:/ {sub(":", "", $1); print $1; exit}') if [ -z "$iface" ]; then logger "PPPoE_script_message010: PPPoE interface lost – reloading WAN" if /usr/local/sbin/pfSctl -c 'interface reload wan'; then PPPOE_ALREADY_STARTED="true" else logger "PPPoE_script_message011: WAN reload failed" return 1 fi fi } ############################################################################### # Entry point ############################################################################### case "$1" in start) check_already_running find_pppoe_info # keeps old behaviour (not strictly required) start_monitoring ;; stop) echo "Stop command received (not implemented)." ;; *) echo "Usage: $0 {start|stop}" exit 1 ;; esac exit 0Ensure that VHID 5 still exists – if not, re-install the VIPs. This was a workaround for cases when the VIP disappeared for some reason. I’m not sure if it’s really needed, but it seemed that after rebooting the firewall, the VIPs were sometimes missing on the LAN. So it was added as a workaround, though you may not need it.
if ! ifconfig "$LAN_IF" | grep -q "vhid 5"; then logger "PPPoE_script_message006: VHID 5 missing on BACKUP – reconfiguring VIPs" $PHP_BIN -r 'require_once "/etc/inc/interfaces.inc"; interfaces_vips_configure();' fi -
@w0w thanks for sharing those scripts.
I did invest some time to create a pfSense package for handling PPPoe High Availability using CARP - more or less the same way it worked before.
My package has a GUI for selecting the PPPoE interface(s) and the corresponding CARP interface. Whenever a CARP event occurs for the selected interfaces, the corresponding PPPoE interface is taken down on BACKUP and brought up on MASTER, just as your scripts do.
As my logic is responding to the CARP event directly, there is no need for a timely approach using a main loop and sleep.
i've uploaded the files and the most recent pkg to GitHub:
https://github.com/perrin-1/pfSense-pppoe-haI'd be happy if someone would test that on their machines.
-
@perrin i have downloaded and try on my env and i encounter that the pppoe interface still up on backup node. I have leave 2 issue into github project.
-
@zjamali Did you try a manual reconcile?
Can you post the log output and your config? -
This post is deleted! -
-
@zjamali i suspect that i might have an old pkg in the github repo. can you please check the file pppoe_ha_even.php (in /usr/local/sbin) and check if the function reconcile_all (line 176ff) looks like this:
function reconcile_all() { $rows = ppha_get_rows(); if (!$rows){ ha_log("Reconcile: no mappings configured"); return; } ha_log("Running reconcile for all configured mappings"); $vips = config_get_path('virtualip/vip', []); foreach ($rows as $row) { if (empty($row['enabled']) || empty($row['vipref']) || empty($row['iface'])) continue; $vip = $vips[$row['vipref']] ?? null; if (!$vip || ($vip['mode'] ?? '')!=='carp') continue; $vhid = (int)($vip['vhid'] ?? -1); if ($vhid<0) continue; $state = get_carp_state_for_vhid($vhid) ?? 'INIT'; $friendly = (string)$row['iface']; $real = real_ifname_for_friendly($friendly); if (!$real){ ha_log("Reconcile: {$friendly} real if not found; skip"); continue; } if ($state==='MASTER'){ ha_log("Reconcile: VHID {$vhid} MASTER - UP {$friendly} ({$real})"); iface_up($real); } elseif ($state==='BACKUP'){ ha_log("Reconcile: VHID {$vhid} BACKUP - DOWN {$friendly} ({$real})"); iface_down($real); } elseif ($state==='INIT'){ ha_log("Reconcile: VHID {$vhid} INIT - DOWN {$friendly} ({$real})"); iface_down($real); } else { ha_log("Reconcile: VHID {$vhid} {$state} - no action"); } } }the script should log the set interface states (UP, DOWN) in the log - thats why I am wondering...
-
@perrin i have compared with yours above, its match
-
seems like $rows not getting any value and not go into the foreach loop
too soon, its already have the return for no mapping
-
@zjamali said in new if_pppoe Backend - getting HA/CARP to work like in MPD:
seems like $rows not getting any value and not go into the foreach loop
yes, this is what i am suspecting as well. can you check your config.xml and look for the <pppoeha> section. it should look something like this:
<pppoeha> <config> <row> <enabled>ON</enabled> <iface>wan</iface> <vipref>17</vipref> </row> </config> </pppoeha> -
@perrin said in new if_pppoe Backend - getting HA/CARP to work like in MPD:
yes, this is what i am suspecting as well. can you check your config.xml and look for the <pppoeha> section. it should look something like this:
<pppoeha> <config> <row> <enabled>ON</enabled> <iface>wan</iface> <vipref>17</vipref> </row> </config> </pppoeha>Where i can find this config.xml?
-
@zjamali ah sorry. you can either download a backup (under backup/restore) or directly on the box with ssh do something like:
grep -A10 pppoeha /conf/config.xml -
-
said in new if_pppoe Backend - getting HA/CARP to work like in MPD:
is it because vipref == 0? my first vhid comes with id = 0
Yup, that cause it. i changed to VHID LAN, id != 0, its appear in log
-
@zjamali great, thanks for figuring that out. I need to check why it's not working with vipref=0.
currently i have no idea why this happens. vipref 0 is just the first item in the vips. config_get_path is a standard pfsense function.
Maybe it is something with the way you configured that vip? is your first VIP configured differently ffrom the one you were using later?
-
@perrin Its the first vip configured in both pfsense node. So it gets id = 0.
-
@zjamali this is the way the dropdown in the PPPoE-HA GUI looks in my production firewall:
in my case VHID 18 is the one that is configured for failover.Can you check if yours maybe starts with VHID 0?
