Carp work fine on all interfaces but one



  • hello,

    we have installed 2 pfSense 2, that are connected via private interface.

    all is working fine, we have 13 interfaces (one wan, two lans, a lot of DMZ, one for each subnet).
    configuring the interface work correctly and put the second firewall as backup.

    the problem is that if we reboot the backup firewall, all the interfaces but one goes up in backup state inside carp status.
    of course we checked for differences between the configuration of the VIP, of the carp, inside the switch (vlans) and so on…
    the only difference is the ip of published VRRP packets, see next.

    deleting the carp and re-creating it again will set again correctly the backup firewall as backup in carp status.
    seems like the secondary does not get the primary VRRPv2 packet that mark it "online".

    the network is XXX.XXX.30.240/28,
    the first router is 30.241,
    the second router is 30.242,
    the gateway for the subnet (the vip) is 30.243.

    show filter log on secondary:

    00:00:00.000000 rule 11/0(match): block in on igb1_vlan102: XXX.XXX.30.243 > 224.0.0.18: VRRPv2, Advertisement, vrid 14, prio 0, authtype none, intvl 1s, length 36
    00:00:01.000978 rule 11/0(match): block in on igb1_vlan102: XXX.XXX.30.243 > 224.0.0.18: VRRPv2, Advertisement, vrid 14, prio 0, authtype none, intvl 1s, length 36
    00:00:01.001000 rule 11/0(match): block in on igb1_vlan102: XXX.XXX.30.243 > 224.0.0.18: VRRPv2, Advertisement, vrid 14, prio 0, authtype none, intvl 1s, length 36
    00:00:01.000968 rule 11/0(match): block in on igb1_vlan102: XXX.XXX.30.243 > 224.0.0.18: VRRPv2, Advertisement, vrid 14, prio 0, authtype none, intvl 1s, length 36
    00:00:01.000973 rule 11/0(match): block in on igb1_vlan102: XXX.XXX.30.243 > 224.0.0.18: VRRPv2, Advertisement, vrid 14, prio 0, authtype none, intvl 1s, length 36
    00:00:01.000980 rule 11/0(match): block in on igb1_vlan102: XXX.XXX.30.243 > 224.0.0.18: VRRPv2, Advertisement, vrid 14, prio 0, authtype none, intvl 1s, length 36
    00:00:01.000981 rule 11/0(match): block in on igb1_vlan102: XXX.XXX.30.243 > 224.0.0.18: VRRPv2, Advertisement, vrid 14, prio 0, authtype none, intvl 1s, length 36
    00:00:01.000980 rule 11/0(match): block in on igb1_vlan102: XXX.XXX.30.243 > 224.0.0.18: VRRPv2, Advertisement, vrid 14, prio 0, authtype none, intvl 1s, length 36

    tcpdump on the interface on secondary

    12:01:11.027030 IP XXX.XXX.30.243 > 224.0.0.18: VRRPv2, Advertisement, vrid 5, prio 0, authtype none, intvl 1s, length 36                                                                                         
    12:01:11.510254 IP XXX.XXX.30.242 > 224.0.0.18: VRRPv2, Advertisement, vrid 5, prio 100, authtype none, intvl 1s, length 36                                                                                       
    12:01:12.027503 IP XXX.XXX.30.243 > 224.0.0.18: VRRPv2, Advertisement, vrid 5, prio 0, authtype none, intvl 1s, length 36                                                                                         
    12:01:12.901548 IP XXX.XXX.30.242 > 224.0.0.18: VRRPv2, Advertisement, vrid 5, prio 100, authtype none, intvl 1s, length 36                                                                                       
    12:01:13.028009 IP XXX.XXX.30.243 > 224.0.0.18: VRRPv2, Advertisement, vrid 5, prio 0, authtype none, intvl 1s, length 36                                                                                         
    12:01:14.028468 IP XXX.XXX.30.243 > 224.0.0.18: VRRPv2, Advertisement, vrid 5, prio 0, authtype none, intvl 1s, length 36                                                                                         
    12:01:14.292843 IP XXX.XXX.30.242 > 224.0.0.18: VRRPv2, Advertisement, vrid 5, prio 100, authtype none, intvl 1s, length 36                                                                                       
    12:01:15.028943 IP XXX.XXX.30.243 > 224.0.0.18: VRRPv2, Advertisement, vrid 5, prio 0, authtype none, intvl 1s, length 36                                                                                         
    12:01:15.684141 IP XXX.XXX.30.242 > 224.0.0.18: VRRPv2, Advertisement, vrid 5, prio 100, authtype none, intvl 1s, length 36                                                                                       
    12:01:16.029423 IP XXX.XXX.30.243 > 224.0.0.18: VRRPv2, Advertisement, vrid 5, prio 0, authtype none, intvl 1s, length 36                                                                                         
    12:01:17.029901 IP XXX.XXX.30.243 > 224.0.0.18: VRRPv2, Advertisement, vrid 5, prio 0, authtype none, intvl 1s, length 36                                                                                         
    12:01:17.075435 IP XXX.XXX.30.242 > 224.0.0.18: VRRPv2, Advertisement, vrid 5, prio 100, authtype none, intvl 1s, length 36

    tcpdump on primary:

    12:01:11.153204 IP XXX.XXX.30.242 > 224.0.0.18: VRRPv2, Advertisement, vrid 5, prio 100, authtype none, intvl 1s, length 36                                                                                       
    12:01:11.670669 IP XXX.XXX.30.243 > 224.0.0.18: VRRPv2, Advertisement, vrid 5, prio 0, authtype none, intvl 1s, length 36                                                                                         
    12:01:11.712192 IP XXX.XXX.30.244 > 173.194.35.19: ICMP echo request, id 60206, seq 7086, length 64                                                                                                               
    12:01:11.712562 IP 173.194.35.19 > XXX.XXX.30.244: ICMP echo reply, id 60206, seq 7086, length 64                                                                                                                 
    12:01:11.713491 IP XXX.XXX.30.244.2273 > 93.95.210.20.64436: Flags [P.], ack 0, win 250, options [nop,nop,TS val 256114177 ecr 1762791], length 128                                                               
    12:01:11.781887 IP 93.95.210.20.64436 > XXX.XXX.30.244.2273: Flags [.], ack 18961, win 1002, options [nop,nop,TS val 1763046 ecr 256114177], length 0                                                             
    12:01:11.781958 IP 93.95.210.20.64436 > XXX.XXX.30.244.2273: Flags [P.], ack 18961, win 1002, options [nop,nop,TS val 1763046 ecr 256114177], length 48                                                           
    12:01:11.782110 IP XXX.XXX.30.244.2273 > 93.95.210.20.64436: Flags [.], ack 48, win 250, options [nop,nop,TS val 256114194 ecr 1763046], length 0                                   
    –-

    As you can see in the logs, the prio0 is announced through 30.243 and not 30.241 as expected (and as it work in all the other interfaces, using the real ip of the device and not the virtual ip).

    we didn't rebooted the first firewall, as we are in the middle of a migration and we would like to do it when we are in server farm (300 Km away)...

    Any hints apart from reboot the first firewall?

    thanks,
    d.


Locked