Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    CARP abnormal (?) behaviour

    Scheduled Pinned Locked Moved
    2.0-RC Snapshot Feedback and Problems - RETIRED
    2
    3
    5.5k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      Stephane
      last edited by

      Hi all,
      First, I want to thank all of you for PFSense. I really enjoy working on it. Actually I'm doing a lot of test trying to push this solution to my company. Although the previous version was great, we couldn't use it widely due to some limitation in the standard conf (mostly IPSEC and GRE).  With PFsense 2, I think I'll be able to replace most of our firewalls.
      The problem :
      using pfsense 2 RC3
      My lab : 2 Redundant pfsense (A and B) using CARP, each box has 3 interfaces : WAN, SYN, LAN.
      1 virtual IP for Wan.
      1 virtual IP for Lan.
      Pfsense A is Master
      Pfsense B is Backup
      What works :
      If I shut down and restart pfsense A, CARP works. B becomes Master and after the reboot of A, B goes back to Backup.
      If I unplug cable on A (LAN or WAN) CARP works also.
      What doesn't work :
      If I disable CARP on A then B becomes Master but when I re-enable CARP on A, it goes Master but B stays Master too.
      The virtual ip is in fact still owned by B. The only way to fix the problem is to restart B.
      I don't have this behaviour when I do the same test with pfsense 1.2.3. In that case CARP works.

      Stephane

      1 Reply Last reply Reply Quote 0
      • E
        eri--
        last edited by

        Usually there should be a little delay until pfsync syncs with A.
        Is this your observation or even after some time it still does not switch?

        A packet trace should help on finding why this happens.

        1 Reply Last reply Reply Quote 0
        • S
          Stephane
          last edited by

          Even after few minutes it's still the same. Both A and B are Master.
          ifconfig WAN and LAN : vip is MASTER on both.

          A Wan: 70.70.70.2 Lan 10.150.1.2 Syn 172.16.0.1
          B Wan: 70.70.70.3 Lan 10.150.1.3 Syn 172.16.0.2
          Vip Wan 70.70.70.1
          Vip Lan 10.150.1.1

          Packet trace on WAN
          00:19:01.518690 08:00:27:59:28:30 > cc:01:0b:fc:00:00, ethertype IPv4 (0x0800), length 94: (tos 0x0, ttl 64, id 971, offset 0, flags [none], proto ICMP (1), length 80)
              70.70.70.2 > 70.70.70.5: ICMP echo request, id 25716, seq 2562, length 60
          00:19:01.519043 08:00:27:59:28:30 > cc:01:0b:fc:00:00, ethertype IPv4 (0x0800), length 118: (tos 0x0, ttl 30, id 17679, offset 0, flags [none], proto GRE (47), length 104)
              70.70.70.1 > 80.80.80.2: GREv0, Flags [none], proto IPv4 (0x0800), length 84
          (tos 0x0, ttl 64, id 17679, offset 0, flags [none], proto ICMP (1), length 80)
              192.168.1.1 > 192.168.1.2: ICMP echo request, id 25716, seq 2562, length 60
          00:19:01.524038 cc:01:0b:fc:00:00 > 08:00:27:59:28:30, ethertype IPv4 (0x0800), length 94: (tos 0x0, ttl 255, id 971, offset 0, flags [none], proto ICMP (1), length 80)
              70.70.70.5 > 70.70.70.2: ICMP echo reply, id 25716, seq 2562, length 60
          00:19:01.763983 00:00:5e:00:01:02 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: (tos 0x10, ttl 255, id 41117, offset 0, flags [DF], proto VRRP (112), length 56)
              70.70.70.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 100, authtype none, intvl 1s, length 36, addrs(7): 72.52.77.42,56.224.73.2,133.98.157.29,85.81.104.29,201.162.139.25,185.58.202.255,7.230.69.236
          00:19:02.528818 08:00:27:59:28:30 > cc:01:0b:fc:00:00, ethertype IPv4 (0x0800), length 94: (tos 0x0, ttl 64, id 37026, offset 0, flags [none], proto ICMP (1), length 80)
              70.70.70.2 > 70.70.70.5: ICMP echo request, id 25716, seq 2818, length 60
          00:19:02.529150 08:00:27:59:28:30 > cc:01:0b:fc:00:00, ethertype IPv4 (0x0800), length 118: (tos 0x0, ttl 30, id 38249, offset 0, flags [none], proto GRE (47), length 104)
              70.70.70.1 > 80.80.80.2: GREv0, Flags [none], proto IPv4 (0x0800), length 84
          (tos 0x0, ttl 64, id 38249, offset 0, flags [none], proto ICMP (1), length 80)
              192.168.1.1 > 192.168.1.2: ICMP echo request, id 25716, seq 2818, length 60
          00:19:02.534190 cc:01:0b:fc:00:00 > 08:00:27:59:28:30, ethertype IPv4 (0x0800), length 94: (tos 0x0, ttl 255, id 37026, offset 0, flags [none], proto ICMP (1), length 80)
              70.70.70.5 > 70.70.70.2: ICMP echo reply, id 25716, seq 2818, length 60
          00:19:03.174640 00:00:5e:00:01:02 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: (tos 0x10, ttl 255, id 57738, offset 0, flags [DF], proto VRRP (112), length 56)
              70.70.70.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 100, authtype none, intvl 1s, length 36, addrs(7): 72.52.77.42,56.224.73.2,133.98.157.29,85.81.104.29,201.162.139.25,185.58.202.255,7.230.69.236
          00:19:03.538190 08:00:27:59:28:30 > cc:01:0b:fc:00:00, ethertype IPv4 (0x0800), length 94: (tos 0x0, ttl 64, id 10197, offset 0, flags [none], proto ICMP (1), length 80)
              70.70.70.2 > 70.70.70.5: ICMP echo request, id 25716, seq 3074, length 60
          00:19:03.538332 08:00:27:59:28:30 > cc:01:0b:fc:00:00, ethertype IPv4 (0x0800), length 118: (tos 0x0, ttl 30, id 4969, offset 0, flags [none], proto GRE (47), length 104)
              70.70.70.1 > 80.80.80.2: GREv0, Flags [none], proto IPv4 (0x0800), length 84
          (tos 0x0, ttl 64, id 4969, offset 0, flags [none], proto ICMP (1), length 80)
              192.168.1.1 > 192.168.1.2: ICMP echo request, id 25716, seq 3074, length 60
          00:19:03.544333 cc:01:0b:fc:00:00 > 08:00:27:59:28:30, ethertype IPv4 (0x0800), length 94: (tos 0x0, ttl 255, id 10197, offset 0, flags [none], proto ICMP (1), length 80)
              70.70.70.5 > 70.70.70.2: ICMP echo reply, id 25716, seq 3074, length 60
          Packet Trace on LAN
          00:20:08.737239 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: (tos 0x10, ttl 255, id 56844, offset 0, flags [DF], proto VRRP (112), length 56)
              10.150.1.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 165.78.28.91,230.198.22.54,124.150.11.222,212.181.40.14,255.131.64.104,144.243.162.250,148.88.194.208
          00:20:10.147187 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: (tos 0x10, ttl 255, id 40079, offset 0, flags [DF], proto VRRP (112), length 56)
              10.150.1.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 165.78.28.91,230.198.22.54,124.150.11.222,212.181.40.14,255.131.64.104,144.243.162.250,148.88.194.208
          00:20:11.557420 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: (tos 0x10, ttl 255, id 25843, offset 0, flags [DF], proto VRRP (112), length 56)
              10.150.1.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 165.78.28.91,230.198.22.54,124.150.11.222,212.181.40.14,255.131.64.104,144.243.162.250,148.88.194.208
          00:20:12.968668 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: (tos 0x10, ttl 255, id 23698, offset 0, flags [DF], proto VRRP (112), length 56)
              10.150.1.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 165.78.28.91,230.198.22.54,124.150.11.222,212.181.40.14,255.131.64.104,144.243.162.250,148.88.194.208
          Packet Trace on SYN
          00:21:27.738473 08:00:27:f6:35:fb > 01:00:5e:00:00:f0, ethertype IPv4 (0x0800), length 582: (tos 0x10, ttl 255, id 32661, offset 0, flags [DF], proto unknown (240), length 568)
              172.16.0.2 > 224.0.0.240:  pfsync 548
          00:21:28.013008 08:00:27:f6:35:fb > 01:00:5e:00:00:f0, ethertype IPv4 (0x0800), length 582: (tos 0x10, ttl 255, id 17391, offset 0, flags [DF], proto unknown (240), length 568)
              172.16.0.2 > 224.0.0.240:  pfsync 548
          00:21:28.478968 08:00:27💿11:98 > 01:00:5e:00:00:f0, ethertype IPv4 (0x0800), length 582: (tos 0x10, ttl 255, id 25466, offset 0, flags [DF], proto unknown (240), length 568)
              172.16.0.1 > 224.0.0.240:  pfsync 548
          00:21:28.944427 08:00:27:f6:35:fb > 01:00:5e:00:00:f0, ethertype IPv4 (0x0800), length 582: (tos 0x10, ttl 255, id 2825, offset 0, flags [DF], proto unknown (240), length 568)
              172.16.0.2 > 224.0.0.240:  pfsync 548
          00:21:29.881374 08:00:27💿11:98 > 01:00:5e:00:00:f0, ethertype IPv4 (0x0800), length 494: (tos 0x10, ttl 255, id 35091, offset 0, flags [DF], proto unknown (240), length 480)
              172.16.0.1 > 224.0.0.240:  pfsync 460
          00:21:29.889879 08:00:27:f6:35:fb > 01:00:5e:00:00:f0, ethertype IPv4 (0x0800), length 582: (tos 0x10, ttl 255, id 42174, offset 0, flags [DF], proto unknown (240), length 568)
              172.16.0.2 > 224.0.0.240:  pfsync 548
          00:21:30.550182 08:00:27:f6:35:fb > 01:00:5e:00:00:f0, ethertype IPv4 (0x0800), length 582: (tos 0x10, ttl 255, id 47180, offset 0, flags [DF], proto unknown (240), length 568)
              172.16.0.2 > 224.0.0.240:  pfsync 548
          00:21:30.881483 08:00:27💿11:98 > 01:00:5e:00:00:f0, ethertype IPv4 (0x0800), length 494: (tos 0x10, ttl 255, id 51093, offset 0, flags [DF], proto unknown (240), length 480)
              172.16.0.1 > 224.0.0.240:  pfsync 460
          00:21:30.964528 08:00:27:f6:35:fb > 01:00:5e:00:00:f0, ethertype IPv4 (0x0800), length 582: (tos 0x10, ttl 255, id 22842, offset 0, flags [DF], proto unknown (240), length 568)
              172.16.0.2 > 224.0.0.240:  pfsync 548
          00:21:31.901618 08:00:27💿11:98 > 01:00:5e:00:00:f0, ethertype IPv4 (0x0800), length 406: (tos 0x10, ttl 255, id 17408, offset 0, flags [DF], proto unknown (240), length 392)
              172.16.0.1 > 224.0.0.240:  pfsync 372
          00:21:31.954620 08:00:27:f6:35:fb > 01:00:5e:00:00:f0, ethertype IPv4 (0x0800), length 582: (tos 0x10, ttl 255, id 18703, offset 0, flags [DF], proto unknown (240), length 568)
              172.16.0.2 > 224.0.0.240:  pfsync 548

          Ifconfig A
          em0: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
          options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 08:00:27:5b:4c:44
          media: Ethernet autoselect (1000baseT <full-duplex>)
          status: active
          em1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
          options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 08:00:27:59:28:30
          inet 70.70.70.2 netmask 0xfffffff8 broadcast 70.70.70.7
          inet6 fe80::a00:27ff:fe59:2830%em1 prefixlen 64 scopeid 0x2
          nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
          status: active
          em2: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
          options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 08:00:27:a2:3c:bc
          inet 10.150.1.2 netmask 0xffffff00 broadcast 10.150.1.255
          inet6 fe80::a00:27ff:fea2:3cbc%em2 prefixlen 64 scopeid 0x3
          nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
          status: active
          em3: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
          options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 08:00:27💿11:98
          inet 172.16.0.1 netmask 0xfffffffc broadcast 172.16.0.3
          inet6 fe80::a00:27ff:fecd:1198%em3 prefixlen 64 scopeid 0x4
          nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
          status: active
          lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
          options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000
          inet6 ::1 prefixlen 128
          inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
          nd6 options=3 <performnud,accept_rtadv>pfsync0: flags=41 <up,running>metric 0 mtu 1460
          pfsync: syncdev: em3 syncpeer: 224.0.0.240 maxupd: 128
          pflog0: flags=100 <promisc>metric 0 mtu 33664
          enc0: flags=0<> metric 0 mtu 1536
          gre0: flags=9051 <up,pointopoint,running,link0,multicast>metric 0 mtu 1476
          tunnel inet 70.70.70.1 –> 80.80.80.2
          inet 192.168.1.1 --> 192.168.1.2 netmask 0xfffffffc
          inet6 fe80::a00:27ff:fe5b:4c44%gre0 prefixlen 64 scopeid 0xb
          nd6 options=3 <performnud,accept_rtadv>vip1: flags=49 <up,loopback,running>metric 0 mtu 1500
          inet 10.150.1.1 netmask 0xffffff00
          carp: MASTER vhid 1 advbase 1 advskew 0
          vip2: flags=49 <up,loopback,running>metric 0 mtu 1500
          inet 70.70.70.1 netmask 0xffffffff
          carp: MASTER vhid 2 advbase 1 advskew 0

          ifconfig B
          em0: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
          options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 08:00:27:f4:27:d0
          media: Ethernet autoselect (1000baseT <full-duplex>)
          status: active
          em1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
          options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 08:00:27:10:90:b8
          inet 70.70.70.3 netmask 0xfffffff8 broadcast 70.70.70.7
          inet6 fe80::a00:27ff:fe10:90b8%em1 prefixlen 64 scopeid 0x2
          nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
          status: active
          em2: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
          options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 08:00:27:ce:12:d8
          inet 10.150.1.3 netmask 0xffffff00 broadcast 10.150.1.255
          inet6 fe80::a00:27ff:fece:12d8%em2 prefixlen 64 scopeid 0x3
          nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
          status: active
          em3: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
          options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 08:00:27:f6:35:fb
          inet 172.16.0.2 netmask 0xfffffffc broadcast 172.16.0.3
          inet6 fe80::a00:27ff:fef6:35fb%em3 prefixlen 64 scopeid 0x4
          nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
          status: active
          lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
          options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000
          inet6 ::1 prefixlen 128
          inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
          nd6 options=3 <performnud,accept_rtadv>pfsync0: flags=41 <up,running>metric 0 mtu 1460
          pfsync: syncdev: em3 syncpeer: 224.0.0.240 maxupd: 128
          pflog0: flags=100 <promisc>metric 0 mtu 33664
          enc0: flags=0<> metric 0 mtu 1536
          vip1: flags=49 <up,loopback,running>metric 0 mtu 1500
          inet 10.150.1.1 netmask 0xffffff00
          carp: MASTER vhid 1 advbase 1 advskew 100
          vip2: flags=49 <up,loopback,running>metric 0 mtu 1500
          inet 70.70.70.1 netmask 0xffffffff
          carp: MASTER vhid 2 advbase 1 advskew 100
          gre0: flags=9051 <up,pointopoint,running,link0,multicast>metric 0 mtu 1476
          tunnel inet 70.70.70.1 --> 80.80.80.2
          inet 192.168.1.1 --> 192.168.1.2 netmask 0xfffffffc
          inet6 fe80::a00:27ff:fef4:27d0%gre0 prefixlen 64 scopeid 0xb
          nd6 options=3 <performnud,accept_rtadv></performnud,accept_rtadv></up,pointopoint,running,link0,multicast></up,loopback,running></up,loopback,running></promisc></up,running></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,promisc,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></broadcast,simplex,multicast></up,loopback,running></up,loopback,running></performnud,accept_rtadv></up,pointopoint,running,link0,multicast></promisc></up,running></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,promisc,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></broadcast,simplex,multicast>

          1 Reply Last reply Reply Quote 0
          • First post
            Last post