new if_pppoe Backend - getting HA/CARP to work like in MPD

perrin

@zjamali i suspect that i might have an old pkg in the github repo. can you please check the file pppoe_ha_even.php (in /usr/local/sbin) and check if the function reconcile_all (line 176ff) looks like this:

function reconcile_all() {
    $rows = ppha_get_rows();
    if (!$rows){ ha_log("Reconcile: no mappings configured"); return; }
    ha_log("Running reconcile for all configured mappings");
    $vips = config_get_path('virtualip/vip', []);
    foreach ($rows as $row) {
        if (empty($row['enabled']) || empty($row['vipref']) || empty($row['iface'])) continue;
        $vip = $vips[$row['vipref']] ?? null;
        if (!$vip || ($vip['mode'] ?? '')!=='carp') continue;
        $vhid = (int)($vip['vhid'] ?? -1); if ($vhid<0) continue;

        $state = get_carp_state_for_vhid($vhid) ?? 'INIT';
        $friendly = (string)$row['iface'];
        $real = real_ifname_for_friendly($friendly);
        if (!$real){ ha_log("Reconcile: {$friendly} real if not found; skip"); continue; }

        if ($state==='MASTER'){ ha_log("Reconcile: VHID {$vhid} MASTER - UP {$friendly} ({$real})"); iface_up($real); }
        elseif ($state==='BACKUP'){ ha_log("Reconcile: VHID {$vhid} BACKUP - DOWN {$friendly} ({$real})"); iface_down($real); }
        elseif ($state==='INIT'){ ha_log("Reconcile: VHID {$vhid} INIT - DOWN {$friendly} ({$real})"); iface_down($real); }
        else { ha_log("Reconcile: VHID {$vhid} {$state} - no action"); }
    }
}

the script should log the set interface states (UP, DOWN) in the log - thats why I am wondering...

zjamali

@perrin i have compared with yours above, its match

zjamali

seems like $rows not getting any value and not go into the foreach loop

too soon, its already have the return for no mapping

perrin

@zjamali said in new if_pppoe Backend - getting HA/CARP to work like in MPD:

seems like $rows not getting any value and not go into the foreach loop

yes, this is what i am suspecting as well. can you check your config.xml and look for the <pppoeha> section. it should look something like this:

 <pppoeha>
                        <config>
                                <row>
                                        <enabled>ON</enabled>
                                        <iface>wan</iface>
                                        <vipref>17</vipref>
                                </row>
                        </config>
                </pppoeha>

zjamali

@perrin said in new if_pppoe Backend - getting HA/CARP to work like in MPD:

yes, this is what i am suspecting as well. can you check your config.xml and look for the <pppoeha> section. it should look something like this:

 <pppoeha>
                        <config>
                                <row>
                                        <enabled>ON</enabled>
                                        <iface>wan</iface>
                                        <vipref>17</vipref>
                                </row>
                        </config>
                </pppoeha>

Where i can find this config.xml?

perrin

@zjamali ah sorry. you can either download a backup (under backup/restore) or directly on the box with ssh do something like:

grep -A10 pppoeha /conf/config.xml

zjamali

@perrin

zjamali

said in new if_pppoe Backend - getting HA/CARP to work like in MPD:

@perrin

is it because vipref == 0? my first vhid comes with id = 0

Yup, that cause it. i changed to VHID LAN, id != 0, its appear in log

perrin

@zjamali great, thanks for figuring that out. I need to check why it's not working with vipref=0.

currently i have no idea why this happens. vipref 0 is just the first item in the vips. config_get_path is a standard pfsense function.

Maybe it is something with the way you configured that vip? is your first VIP configured differently ffrom the one you were using later?

zjamali

@perrin Its the first vip configured in both pfsense node. So it gets id = 0.

perrin

@zjamali this is the way the dropdown in the PPPoE-HA GUI looks in my production firewall:

in my case VHID 18 is the one that is configured for failover.

Can you check if yours maybe starts with VHID 0?

zjamali

@perrin Mine also start with VHID 1 but if you go to Firewall -> Virtual IP, there is listing of all VIP, you hover on the edit button for first VHID, the url link will say its id=0

id = 0 is for internal system id for record creation whilst VHID 1 is for user POV, if i am not mistaken.

perrin

@zjamali yep, same here. I'll debug why it is not working on the first VIP later.
Can you temporarely work with a different VIP?

zjamali

@perrin said in new if_pppoe Backend - getting HA/CARP to work like in MPD:

@zjamali yep, same here. I'll debug why it is not working on the first VIP later.
Can you temporarely work with a different VIP?

Should be OK. no issue

perrin

@zjamali said in new if_pppoe Backend - getting HA/CARP to work like in MPD:

@perrin Mine also start with VHID 1 but if you go to Firewall -> Virtual IP, there is listing of all VIP, you hover on the edit button for first VHID, the url link will say its id=0

id = 0 is for internal system id for record creation whilst VHID 1 is for user POV, if i am not mistaken.

I tried reproducing the behaviour by removing all carp VIPs from my test firewall and adding just a single new one so it gets "id=0". In my test VM the script behaved as expected:

Sep 17 07:46:31 	pppoe-ha 	1464 	VHID 1 BACKUP -> DOWN wan (pppoe0)
Sep 17 07:46:31 	pppoe-ha 	1464 	Handle CARP command for 1@vtnet0.510 - BACKUP

and

Sep 17 07:46:35 	pppoe-ha 	52762 	VHID 1 MASTER -> UP wan (pppoe0)
Sep 17 07:46:35 	pppoe-ha 	52762 	Handle CARP command for 1@vtnet0.510 - MASTER

so, there must be some differences in the config between your and my firewall.

this is the Virtual IP definition in the my config XML:

<vip>
			<mode>carp</mode>
			<interface>opt12</interface>
			<vhid>1</vhid>
			[remaining data ommited]
</vip>

so, in my case it starts with vhid = 1 instead of zero. can you confirm this in your installation?

zjamali

@perrin

The record that i having the issue is this

Check if your opt12 with vhid 1, when you hover the edit button, id = 0. that causing the trouble when selected on the mapping. If i using other carp vip, it can reconcile

perrin

@zjamali Yep, in my case my VHID18 is ID=0

but in my case this VIP also works:

Sep 17 08:08:42 	pppoe-ha 	41559 	VHID 18 MASTER -> UP wan (pppoe0)
Sep 17 08:08:42 	pppoe-ha 	41559 	Handle CARP command for 18@vtnet0.510 - MASTER
Sep 17 08:08:39 	pppoe-ha 	28341 	VHID 18 BACKUP -> DOWN wan (pppoe0)
Sep 17 08:08:39 	pppoe-ha 	28341 	Handle CARP command for 18@vtnet0.510 - BACKUP

Can you please confirm that your VIP correctly switches from MASTER to BACKUP?
Also your vhid 2 network overlaps with vhid 101 and vhid 3 with 102

w0w

Hmm...

empty($row['enabled']) || empty($row['vipref']) || empty($row['iface'])) continue;
$vip =

empty($row['vipref']) will then exlude viperf=0 is not it?

I think this must be changed to

if (empty($row['enabled']) || !isset($row['vipref']) || $row['vipref'] === '' || empty($row['iface'])) continue;
$vip = $vips[$row['vipref']] ?? null;

If we want to use 0 from viperf array (or any other not empty value)

perrin

@w0w said in new if_pppoe Backend - getting HA/CARP to work like in MPD:

empty($row['enabled']) || empty($row['vipref'])

you are right. there is a bug in the reconcile_all function.

i will fix that.

perrin

ok, i did a complete refactor of the logic within handle_carp_change and reconcile_all so that in the end both functions use the same logic. tested it on my machine and it seems to work for me.
I uploaded a new pkg version 0.1.1 to github.

Happy to get feedback from your tests.