CARP interfaces work separately

[Jakub_]

CARP interfaces work separately

My setup 2x PFsense 23.01 on 7100U.
ix1 WAN; ix0 LAN and SYNC on an additional interface.
When ix1 is turned off on Master, on Backup only xi1 takes up the work. I am left with the situation that I have LAN UP on the master and WAN UP on the backup.
Traffic, of course, is impossible.

How to make that when one interface goes down, both interfaces switch over.

Master and Backup after xi1 on Master down :
login-to-view

[Derelict]

@jakub_ something is misconfigured. When the interface goes down on the primary, all CARP VIPs should be demoted.

Post an ifconfig -a from both when the nodes are in that state.

[Jakub_]

@derelict
Thanks for your response,
I am uploading the ifconfig dumps for Master and Backups.
master_ifconf.txt carp_ifconf.txt

[Derelict]

@jakub_ Take the secondary node out of maintenance mode and test again.

There is pretty much no valid reason to ever put a secondary node in maintenance mode.

Swing traffic from the primary to the secondary by putting the primary in maintenance mode.

Swing it back by taking the primary out of maintenance mode.

[viragomann]

@jakub_
Seems that you have assigned the same IP to both nodes:
inet xx.xx.xx.170 netmask 0xffffffff broadcast xx.xx.xx.170

Possibly you used here accidentally IP alias type instead of CARP VIP.

[Jakub_]

@derelict
I just did it.
Everything switched over correctly.
Tomorrow when I'm in the server room I'll do a wire pull test and let you know what the results are.

[Jakub_]

@viragomann
Yes you are right I corrected it, it should be CARP

[Derelict]

@jakub_ said in CARP interfaces work separately:

@viragomann
Yes you are right I corrected it, it should be CARP

Or an IP alias with the interface set to the existing CARP VIP on the interface, not the interface itself.

[Jakub_]

@derelict
Ok, I did the tests everything switches correctly.
I don't know how it happened that the maintenance mode was on.

[Jakub_]

@derelict
This morning the problem returned.
First the first 7200U (master) traffic stalled , I put it in "CARP maitenence mode " the backup switched to the master but, the stalled master only half switched to the backup role (see pictures).
When I turned off "Maitenence mode" on this first 7200 on the second one only half returned to the backup role. The issue was fixed by turning CARPA off and on.
But it looks poor because HA should be reliable on this critical link.
I got screen shots and ifconfig's from both.
login-to-view

[viragomann]

@jakub_
Something CARP relating in the system log?

The reason for interfaces in master state on both nodes is often that the secondary (with higher skew) doesn't get the advertisements from the master.
So ensure that the interfaces of both can communicate properly using the CARP protocol.

[Jakub_]

@viragomann
Not much :

Apr 19 08:27:00 Node1 sshguard[7411]: Now monitoring attacks.
Apr 19 08:39:30 Node1 php-fpm[94281]: /status_logs_filter.php: Successful login for user 'xxxxx' from: xx.xxx.xxx.10 (Local Database Fallback)
Apr 19 08:42:30 Node1 check_reload_status[392]: Syncing firewall
Apr 19 08:42:30 Node1 check_reload_status[392]: Carp backup event
Apr 19 08:42:30 Node1 kernel: carp: 1@ix0: MASTER -> BACKUP (more frequent advertisement received)
Apr 19 08:42:30 Node1 kernel: carp: 2@ix1: MASTER -> BACKUP (more frequent advertisement received)
Apr 19 08:42:30 Node1 kernel: in_scrubprefix: err=65, prefix delete failed
Apr 19 08:42:30 Node1 check_reload_status[392]: Carp backup event
Apr 19 08:42:36 Node1 check_reload_status[392]: Carp master event
Apr 19 08:42:36 Node1 kernel: carp: 2@ix1: BACKUP -> MASTER (master timed out)
Apr 19 08:43:00 Node1 sshguard[7411]: Exiting on signal.
Apr 19 08:43:00 Node1 sshguard[55540]: Now monitoring attacks.
Apr 19 08:44:00 Node1 sshguard[55540]: Exiting on signal.

login-to-view

But the carp only worked after my intervention.

[Derelict]

@jakub_ You have to figure out why the CARP heartbeats from the MASTER node are not making it to the secondary node.

pcap for CARP on that interface on the primary node. You should see advskew=0 heartbeats.

pcap for CARP on the secondary node. You should see those heartbeats. If you do not and see the heartbeats from the secondary (advskew=100) instead your Layer 2 is broken.

If the protocol is set for CARP on the pcap page it will properly decode the advbase/advskew so you can tell them apart. They will be from the same virtual MAC address so you can't tell by that.

[Jakub_]

This post is deleted!

[Jakub_]

@derelict

Hi again, I checked the vrrp packages :
IP xx.xx.xx.3 > 224.0.0.18: CARPv2-advertise 36: vhid=2 advbase=1 advskew=100 authlen=7 counter=14661700377225625920
IP xx.xx.xx.3 > 224.0.0.18: CARPv2-advertise 36: vhid=99 advbase=1 advskew=0 authlen=7 counter=316479634456754718
IP xx.xx.xx.3 > 224.0.0.18: CARPv2-advertise 36: vhid=99 advbase=1 advskew=0 authlen=7 counter=316479634456754719
IP xx.xx.xx.3 > 224.0.0.18: CARPv2-advertise 36: vhid=2 advbase=1 advskew=100 authlen=7 counter=14661700377225625921
IP xx.xx.xx.3 > 224.0.0.18: CARPv2-advertise 36: vhid=99 advbase=1 advskew=0 authlen=7 counter=316479634456754720
IP xx.xx.xx.3 > 224.0.0.18: CARPv2-advertise 36: vhid=2 advbase=1 advskew=100 authlen=7 counter=14661700377225625922
IP xx.xx.xx.3 > 224.0.0.18: CARPv2-advertise 36: vhid=99 advbase=1 advskew=0 authlen=7 counter=316479634456754721

I sorted out the vrrp and they look ok, my only doubt is the address xx.xx.xx.xx.3 is the physical interface of the master and not the CARP VIP. unless that is ok ?

[Derelict]

@jakub_ Yes. The advertisements are sourced from the interface IP address and CARP MAC.

Not sure why you are seen advertisements from both the primary (advskew 0) and secondary (advskew 100) there.