CARP abnormal (?) behaviour



  • Hi all,
    First, I want to thank all of you for PFSense. I really enjoy working on it. Actually I'm doing a lot of test trying to push this solution to my company. Although the previous version was great, we couldn't use it widely due to some limitation in the standard conf (mostly IPSEC and GRE).  With PFsense 2, I think I'll be able to replace most of our firewalls.
    The problem :
    using pfsense 2 RC3
    My lab : 2 Redundant pfsense (A and B) using CARP, each box has 3 interfaces : WAN, SYN, LAN.
    1 virtual IP for Wan.
    1 virtual IP for Lan.
    Pfsense A is Master
    Pfsense B is Backup
    What works :
    If I shut down and restart pfsense A, CARP works. B becomes Master and after the reboot of A, B goes back to Backup.
    If I unplug cable on A (LAN or WAN) CARP works also.
    What doesn't work :
    If I disable CARP on A then B becomes Master but when I re-enable CARP on A, it goes Master but B stays Master too.
    The virtual ip is in fact still owned by B. The only way to fix the problem is to restart B.
    I don't have this behaviour when I do the same test with pfsense 1.2.3. In that case CARP works.

    Stephane



  • Usually there should be a little delay until pfsync syncs with A.
    Is this your observation or even after some time it still does not switch?

    A packet trace should help on finding why this happens.



  • Even after few minutes it's still the same. Both A and B are Master.
    ifconfig WAN and LAN : vip is MASTER on both.

    A Wan: 70.70.70.2 Lan 10.150.1.2 Syn 172.16.0.1
    B Wan: 70.70.70.3 Lan 10.150.1.3 Syn 172.16.0.2
    Vip Wan 70.70.70.1
    Vip Lan 10.150.1.1

    Packet trace on WAN
    00:19:01.518690 08:00:27:59:28:30 > cc:01:0b:fc:00:00, ethertype IPv4 (0x0800), length 94: (tos 0x0, ttl 64, id 971, offset 0, flags [none], proto ICMP (1), length 80)
        70.70.70.2 > 70.70.70.5: ICMP echo request, id 25716, seq 2562, length 60
    00:19:01.519043 08:00:27:59:28:30 > cc:01:0b:fc:00:00, ethertype IPv4 (0x0800), length 118: (tos 0x0, ttl 30, id 17679, offset 0, flags [none], proto GRE (47), length 104)
        70.70.70.1 > 80.80.80.2: GREv0, Flags [none], proto IPv4 (0x0800), length 84
    (tos 0x0, ttl 64, id 17679, offset 0, flags [none], proto ICMP (1), length 80)
        192.168.1.1 > 192.168.1.2: ICMP echo request, id 25716, seq 2562, length 60
    00:19:01.524038 cc:01:0b:fc:00:00 > 08:00:27:59:28:30, ethertype IPv4 (0x0800), length 94: (tos 0x0, ttl 255, id 971, offset 0, flags [none], proto ICMP (1), length 80)
        70.70.70.5 > 70.70.70.2: ICMP echo reply, id 25716, seq 2562, length 60
    00:19:01.763983 00:00:5e:00:01:02 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: (tos 0x10, ttl 255, id 41117, offset 0, flags [DF], proto VRRP (112), length 56)
        70.70.70.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 100, authtype none, intvl 1s, length 36, addrs(7): 72.52.77.42,56.224.73.2,133.98.157.29,85.81.104.29,201.162.139.25,185.58.202.255,7.230.69.236
    00:19:02.528818 08:00:27:59:28:30 > cc:01:0b:fc:00:00, ethertype IPv4 (0x0800), length 94: (tos 0x0, ttl 64, id 37026, offset 0, flags [none], proto ICMP (1), length 80)
        70.70.70.2 > 70.70.70.5: ICMP echo request, id 25716, seq 2818, length 60
    00:19:02.529150 08:00:27:59:28:30 > cc:01:0b:fc:00:00, ethertype IPv4 (0x0800), length 118: (tos 0x0, ttl 30, id 38249, offset 0, flags [none], proto GRE (47), length 104)
        70.70.70.1 > 80.80.80.2: GREv0, Flags [none], proto IPv4 (0x0800), length 84
    (tos 0x0, ttl 64, id 38249, offset 0, flags [none], proto ICMP (1), length 80)
        192.168.1.1 > 192.168.1.2: ICMP echo request, id 25716, seq 2818, length 60
    00:19:02.534190 cc:01:0b:fc:00:00 > 08:00:27:59:28:30, ethertype IPv4 (0x0800), length 94: (tos 0x0, ttl 255, id 37026, offset 0, flags [none], proto ICMP (1), length 80)
        70.70.70.5 > 70.70.70.2: ICMP echo reply, id 25716, seq 2818, length 60
    00:19:03.174640 00:00:5e:00:01:02 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: (tos 0x10, ttl 255, id 57738, offset 0, flags [DF], proto VRRP (112), length 56)
        70.70.70.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 100, authtype none, intvl 1s, length 36, addrs(7): 72.52.77.42,56.224.73.2,133.98.157.29,85.81.104.29,201.162.139.25,185.58.202.255,7.230.69.236
    00:19:03.538190 08:00:27:59:28:30 > cc:01:0b:fc:00:00, ethertype IPv4 (0x0800), length 94: (tos 0x0, ttl 64, id 10197, offset 0, flags [none], proto ICMP (1), length 80)
        70.70.70.2 > 70.70.70.5: ICMP echo request, id 25716, seq 3074, length 60
    00:19:03.538332 08:00:27:59:28:30 > cc:01:0b:fc:00:00, ethertype IPv4 (0x0800), length 118: (tos 0x0, ttl 30, id 4969, offset 0, flags [none], proto GRE (47), length 104)
        70.70.70.1 > 80.80.80.2: GREv0, Flags [none], proto IPv4 (0x0800), length 84
    (tos 0x0, ttl 64, id 4969, offset 0, flags [none], proto ICMP (1), length 80)
        192.168.1.1 > 192.168.1.2: ICMP echo request, id 25716, seq 3074, length 60
    00:19:03.544333 cc:01:0b:fc:00:00 > 08:00:27:59:28:30, ethertype IPv4 (0x0800), length 94: (tos 0x0, ttl 255, id 10197, offset 0, flags [none], proto ICMP (1), length 80)
        70.70.70.5 > 70.70.70.2: ICMP echo reply, id 25716, seq 3074, length 60
    Packet Trace on LAN
    00:20:08.737239 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: (tos 0x10, ttl 255, id 56844, offset 0, flags [DF], proto VRRP (112), length 56)
        10.150.1.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 165.78.28.91,230.198.22.54,124.150.11.222,212.181.40.14,255.131.64.104,144.243.162.250,148.88.194.208
    00:20:10.147187 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: (tos 0x10, ttl 255, id 40079, offset 0, flags [DF], proto VRRP (112), length 56)
        10.150.1.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 165.78.28.91,230.198.22.54,124.150.11.222,212.181.40.14,255.131.64.104,144.243.162.250,148.88.194.208
    00:20:11.557420 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: (tos 0x10, ttl 255, id 25843, offset 0, flags [DF], proto VRRP (112), length 56)
        10.150.1.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 165.78.28.91,230.198.22.54,124.150.11.222,212.181.40.14,255.131.64.104,144.243.162.250,148.88.194.208
    00:20:12.968668 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: (tos 0x10, ttl 255, id 23698, offset 0, flags [DF], proto VRRP (112), length 56)
        10.150.1.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 165.78.28.91,230.198.22.54,124.150.11.222,212.181.40.14,255.131.64.104,144.243.162.250,148.88.194.208
    Packet Trace on SYN
    00:21:27.738473 08:00:27:f6:35:fb > 01:00:5e:00:00:f0, ethertype IPv4 (0x0800), length 582: (tos 0x10, ttl 255, id 32661, offset 0, flags [DF], proto unknown (240), length 568)
        172.16.0.2 > 224.0.0.240:  pfsync 548
    00:21:28.013008 08:00:27:f6:35:fb > 01:00:5e:00:00:f0, ethertype IPv4 (0x0800), length 582: (tos 0x10, ttl 255, id 17391, offset 0, flags [DF], proto unknown (240), length 568)
        172.16.0.2 > 224.0.0.240:  pfsync 548
    00:21:28.478968 08:00:27💿11:98 > 01:00:5e:00:00:f0, ethertype IPv4 (0x0800), length 582: (tos 0x10, ttl 255, id 25466, offset 0, flags [DF], proto unknown (240), length 568)
        172.16.0.1 > 224.0.0.240:  pfsync 548
    00:21:28.944427 08:00:27:f6:35:fb > 01:00:5e:00:00:f0, ethertype IPv4 (0x0800), length 582: (tos 0x10, ttl 255, id 2825, offset 0, flags [DF], proto unknown (240), length 568)
        172.16.0.2 > 224.0.0.240:  pfsync 548
    00:21:29.881374 08:00:27💿11:98 > 01:00:5e:00:00:f0, ethertype IPv4 (0x0800), length 494: (tos 0x10, ttl 255, id 35091, offset 0, flags [DF], proto unknown (240), length 480)
        172.16.0.1 > 224.0.0.240:  pfsync 460
    00:21:29.889879 08:00:27:f6:35:fb > 01:00:5e:00:00:f0, ethertype IPv4 (0x0800), length 582: (tos 0x10, ttl 255, id 42174, offset 0, flags [DF], proto unknown (240), length 568)
        172.16.0.2 > 224.0.0.240:  pfsync 548
    00:21:30.550182 08:00:27:f6:35:fb > 01:00:5e:00:00:f0, ethertype IPv4 (0x0800), length 582: (tos 0x10, ttl 255, id 47180, offset 0, flags [DF], proto unknown (240), length 568)
        172.16.0.2 > 224.0.0.240:  pfsync 548
    00:21:30.881483 08:00:27💿11:98 > 01:00:5e:00:00:f0, ethertype IPv4 (0x0800), length 494: (tos 0x10, ttl 255, id 51093, offset 0, flags [DF], proto unknown (240), length 480)
        172.16.0.1 > 224.0.0.240:  pfsync 460
    00:21:30.964528 08:00:27:f6:35:fb > 01:00:5e:00:00:f0, ethertype IPv4 (0x0800), length 582: (tos 0x10, ttl 255, id 22842, offset 0, flags [DF], proto unknown (240), length 568)
        172.16.0.2 > 224.0.0.240:  pfsync 548
    00:21:31.901618 08:00:27💿11:98 > 01:00:5e:00:00:f0, ethertype IPv4 (0x0800), length 406: (tos 0x10, ttl 255, id 17408, offset 0, flags [DF], proto unknown (240), length 392)
        172.16.0.1 > 224.0.0.240:  pfsync 372
    00:21:31.954620 08:00:27:f6:35:fb > 01:00:5e:00:00:f0, ethertype IPv4 (0x0800), length 582: (tos 0x10, ttl 255, id 18703, offset 0, flags [DF], proto unknown (240), length 568)
        172.16.0.2 > 224.0.0.240:  pfsync 548

    Ifconfig A
    em0: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
    options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 08:00:27:5b:4c:44
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    em1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
    options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 08:00:27:59:28:30
    inet 70.70.70.2 netmask 0xfffffff8 broadcast 70.70.70.7
    inet6 fe80::a00:27ff:fe59:2830%em1 prefixlen 64 scopeid 0x2
    nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    em2: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
    options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 08:00:27:a2:3c:bc
    inet 10.150.1.2 netmask 0xffffff00 broadcast 10.150.1.255
    inet6 fe80::a00:27ff:fea2:3cbc%em2 prefixlen 64 scopeid 0x3
    nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    em3: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
    options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 08:00:27💿11:98
    inet 172.16.0.1 netmask 0xfffffffc broadcast 172.16.0.3
    inet6 fe80::a00:27ff:fecd:1198%em3 prefixlen 64 scopeid 0x4
    nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
    options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
    nd6 options=3 <performnud,accept_rtadv>pfsync0: flags=41 <up,running>metric 0 mtu 1460
    pfsync: syncdev: em3 syncpeer: 224.0.0.240 maxupd: 128
    pflog0: flags=100 <promisc>metric 0 mtu 33664
    enc0: flags=0<> metric 0 mtu 1536
    gre0: flags=9051 <up,pointopoint,running,link0,multicast>metric 0 mtu 1476
    tunnel inet 70.70.70.1 –> 80.80.80.2
    inet 192.168.1.1 --> 192.168.1.2 netmask 0xfffffffc
    inet6 fe80::a00:27ff:fe5b:4c44%gre0 prefixlen 64 scopeid 0xb
    nd6 options=3 <performnud,accept_rtadv>vip1: flags=49 <up,loopback,running>metric 0 mtu 1500
    inet 10.150.1.1 netmask 0xffffff00
    carp: MASTER vhid 1 advbase 1 advskew 0
    vip2: flags=49 <up,loopback,running>metric 0 mtu 1500
    inet 70.70.70.1 netmask 0xffffffff
    carp: MASTER vhid 2 advbase 1 advskew 0

    ifconfig B
    em0: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
    options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 08:00:27:f4:27:d0
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    em1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
    options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 08:00:27:10:90:b8
    inet 70.70.70.3 netmask 0xfffffff8 broadcast 70.70.70.7
    inet6 fe80::a00:27ff:fe10:90b8%em1 prefixlen 64 scopeid 0x2
    nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    em2: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
    options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 08:00:27:ce:12:d8
    inet 10.150.1.3 netmask 0xffffff00 broadcast 10.150.1.255
    inet6 fe80::a00:27ff:fece:12d8%em2 prefixlen 64 scopeid 0x3
    nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    em3: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
    options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 08:00:27:f6:35:fb
    inet 172.16.0.2 netmask 0xfffffffc broadcast 172.16.0.3
    inet6 fe80::a00:27ff:fef6:35fb%em3 prefixlen 64 scopeid 0x4
    nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
    options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
    nd6 options=3 <performnud,accept_rtadv>pfsync0: flags=41 <up,running>metric 0 mtu 1460
    pfsync: syncdev: em3 syncpeer: 224.0.0.240 maxupd: 128
    pflog0: flags=100 <promisc>metric 0 mtu 33664
    enc0: flags=0<> metric 0 mtu 1536
    vip1: flags=49 <up,loopback,running>metric 0 mtu 1500
    inet 10.150.1.1 netmask 0xffffff00
    carp: MASTER vhid 1 advbase 1 advskew 100
    vip2: flags=49 <up,loopback,running>metric 0 mtu 1500
    inet 70.70.70.1 netmask 0xffffffff
    carp: MASTER vhid 2 advbase 1 advskew 100
    gre0: flags=9051 <up,pointopoint,running,link0,multicast>metric 0 mtu 1476
    tunnel inet 70.70.70.1 --> 80.80.80.2
    inet 192.168.1.1 --> 192.168.1.2 netmask 0xfffffffc
    inet6 fe80::a00:27ff:fef4:27d0%gre0 prefixlen 64 scopeid 0xb
    nd6 options=3 <performnud,accept_rtadv></performnud,accept_rtadv></up,pointopoint,running,link0,multicast></up,loopback,running></up,loopback,running></promisc></up,running></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,promisc,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></broadcast,simplex,multicast></up,loopback,running></up,loopback,running></performnud,accept_rtadv></up,pointopoint,running,link0,multicast></promisc></up,running></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,promisc,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></broadcast,simplex,multicast>


Locked
 

© Copyright 2002 - 2018 Rubicon Communications, LLC | Privacy Policy