Carp issues with one interface



  • Here is my issue:  the wan, lan & carp interfaces on the primary become master.  The wan & carp interfaces on the 2nd become backup, the lan stays in master mode for some reason.

    Here is my setup:

    2 pfsense servers

    server A    primary  RC2h

    wan ip aaa.bbb.ccc.27 real  .25 virtual 
    lan  ip  192.168.20.14  .4 virtual
    carp ip 192.168.4.1 real  .3 virtual
    mgmt  xxx.yyy.ppp.207  no carp

    server B secondary  RC2

    wan ip aaa.bbb.ccc.26 real  .25 virtual 
    lan  ip  192.168.20.5  .4 virtual
    carp ip 192.168.4.2 real  .3 virtual
    mgmt  xxx.yyy.ppp.215  no carp

    ifconfig

    em0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
            options=b <rxcsum,txcsum,vlan_mtu>inet 192.168.20.5 netmask 0xffffff00 broadcast 192.168.20.255
            inet6 fe80::204:23ff:fec9:3418%em0 prefixlen 64 scopeid 0x1
            ether 00:04:23:c9:34:18
            media: Ethernet autoselect (1000baseTX <full-duplex>)
            status: active
    em1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
            options=b <rxcsum,txcsum,vlan_mtu>inet6 fe80::204:23ff:fec9:3419%em1 prefixlen 64 scopeid 0x2
            inet aaa.bbb.ccc.26 netmask 0xffffff00 broadcast aaa.bbb.ccc.255
            ether 00:04:23:c9:34:19
            media: Ethernet autoselect (1000baseTX <full-duplex>)
            status: active
    bge0: flags=8843 <up,broadcast,running,simplex,multicast>mtu 1500
            options=1b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging>inet xxx.yyy.ppp.207 netmask 0xffffff00 broadcast xxx.yyy.ppp.255
            inet6 fe80::216:36ff:fe13:a608%bge0 prefixlen 64 scopeid 0x3
            ether 00:16:36:13:a6:08
            media: Ethernet autoselect (1000baseTX <full-duplex>)
            status: active
    bge1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
            options=1b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging>inet 192.168.4.1 netmask 0xffffff00 broadcast 192.168.4.255
            inet6 fe80::216:36ff:fe13:a609%bge1 prefixlen 64 scopeid 0x4
            ether 00:16:36:13:a6:09
            media: Ethernet autoselect (1000baseTX <full-duplex>)
            status: active
    pfsync0: flags=41 <up,running>mtu 1348
            pfsync: syncdev: lo0 maxupd: 128
    lo0: flags=8049 <up,loopback,running,multicast>mtu 16384
            inet 127.0.0.1 netmask 0xff000000
            inet6 ::1 prefixlen 128
            inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
    pflog0: flags=100 <promisc>mtu 33208
    carp0: flags=49 <up,loopback,running>mtu 1500
            inet 192.168.4.3 netmask 0xffffff00
            carp: BACKUP vhid 1 advbase 1 advskew 29
    carp1: flags=49 <up,loopback,running>mtu 1500
            inet aaa.bbb.ccc.25 netmask 0xffffff00
            carp: BACKUP vhid 2 advbase 1 advskew 29
    carp2: flags=49 <up,loopback,running>mtu 1500
            inet 192.168.20.4 netmask 0xffffff00
            carp: MASTER vhid 3 advbase 1 advskew 29

    ping 192.168.20.14
    PING 192.168.20.14 (192.168.20.14): 56 data bytes
    64 bytes from 192.168.20.14: icmp_seq=0 ttl=64 time=0.293 ms
    64 bytes from 192.168.20.14: icmp_seq=1 ttl=64 time=0.407 ms
    64 bytes from 192.168.20.14: icmp_seq=2 ttl=64 time=0.283 ms
    64 bytes from 192.168.20.14: icmp_seq=3 ttl=64 time=0.292 ms

    ffff
    ifconfig
    bge0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
            options=1b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging>inet6 fe80::210:18ff:fe06:76b3%bge0 prefixlen 64 scopeid 0x1
            inet 192.168.20.14 netmask 0xffffff00 broadcast 192.168.20.255
            ether 00:10:18:06:76:b3
            media: Ethernet autoselect (1000baseTX <full-duplex>)
            status: active
    bge1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
            options=1b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging>inet aaa.bbb.ccc.27 netmask 0xffffff00 broadcast aaa.bbb.ccc.255
            inet6 fe80::211:43ff:fe5b:72d6%bge1 prefixlen 64 scopeid 0x2
            ether 00:11:43:5b:72:d6
            media: Ethernet autoselect (100baseTX <half-duplex>)
            status: active
    bge2: flags=8843 <up,broadcast,running,simplex,multicast>mtu 1500
            options=1b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging>inet xxx.yyy.ppp.215 netmask 0xffffff00 broadcast xxx.yyy.ppp.255
            inet6 fe80::211:43ff:fe5b:72d7%bge2 prefixlen 64 scopeid 0x3
            ether 00:11:43:5b:72:d7
            media: Ethernet autoselect (100baseTX <half-duplex>)
            status: active
    fxp0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
            options=b <rxcsum,txcsum,vlan_mtu>inet 192.168.4.2 netmask 0xffffff00 broadcast 192.168.4.255
            inet6 fe80::202:b3ff:febb:9776%fxp0 prefixlen 64 scopeid 0x4
            ether 00:02:b3:bb:97:76
            media: Ethernet autoselect (100baseTX <full-duplex>)
            status: active
    pflog0: flags=100 <promisc>mtu 33208
    enc0: flags=0<> mtu 1536
    lo0: flags=8049 <up,loopback,running,multicast>mtu 16384
            inet 127.0.0.1 netmask 0xff000000
            inet6 ::1 prefixlen 128
            inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
    pfsync0: flags=41 <up,running>mtu 1348
            pfsync: syncdev: fxp0 maxupd: 128
    carp0: flags=49 <up,loopback,running>mtu 1500
            inet 192.168.4.3 netmask 0xffffff00
            carp: MASTER vhid 1 advbase 1 advskew 0
    carp1: flags=49 <up,loopback,running>mtu 1500
            inet aaa.bbb.ccc.25 netmask 0xffffff00
            carp: MASTER vhid 2 advbase 1 advskew 0
    carp2: flags=49 <up,loopback,running>mtu 1500
            inet 192.168.20.4 netmask 0xffffff00
            carp: MASTER vhid 3 advbase 1 advskew 0

    ping 192.168.20.5
    PING 192.168.20.5 (192.168.20.5): 56 data bytes
    64 bytes from 192.168.20.5: icmp_seq=0 ttl=64 time=0.644 ms
    64 bytes from 192.168.20.5: icmp_seq=1 ttl=64 time=0.436 ms
    64 bytes from 192.168.20.5: icmp_seq=2 ttl=64 time=0.459 ms</up,loopback,running></up,loopback,running></up,loopback,running></up,running></up,loopback,running,multicast></promisc></full-duplex></rxcsum,txcsum,vlan_mtu></up,broadcast,running,promisc,simplex,multicast></half-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging></up,broadcast,running,simplex,multicast></half-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging></up,broadcast,running,promisc,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging></up,broadcast,running,promisc,simplex,multicast></up,loopback,running></up,loopback,running></up,loopback,running></promisc></up,loopback,running,multicast></up,running></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging></up,broadcast,running,promisc,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging></up,broadcast,running,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu></up,broadcast,running,promisc,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu></up,broadcast,running,promisc,simplex,multicast>

    both servers can ping each other, the gateway & the virtual ip on the lan,wan, carp & mgmt networks. per /cf/conf/config.xml both servers have matching passwords for carp

    Also, If I select "disable carp" , why does it not stay disabled?



  • Check that the vhids are the same on the backup and master for the CARP'd ip's.  Also verify that pfsync is working, you can ping the dedicated interface that hosts pfsync, etc.

    Oh, I notice you don't have a dedicated pfsync interface?    Check out the tutorial, you are not doing this like we suggest.



  • vhids match.    I ran thru the tutorial.  I noticed on slide 18 under CARP settings that it mentions enabling preemption.  I do not have a check box for preemption or load balancing.  I am running RC2h on the primary & RC2 on the secondary.

    The interface labeled carp is the sync interface on both servers



  • pfsync0: flags=41 <up,running>mtu 1348
            pfsync: syncdev: lo0 maxupd: 128

    You have not set your pfsync interface in the carp settings area.</up,running>



  • hmmmm.  The secondary lan interface goes into backup for a few seconds then reverts back to master. thanks for all your help in this matter

    Primary
    bge1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
            options=1b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging>inet 192.168.4.1 netmask 0xffffff00 broadcast 192.168.4.255
            inet6 fe80::216:36ff:fe13:a609%bge1 prefixlen 64 scopeid 0x4
            ether 00:16:36:13:a6:09
            media: Ethernet autoselect (1000baseTX <full-duplex>)
            status: active

    pfsync0: flags=41 <up,running>mtu 1348
            pfsync: syncdev: fxp0 maxupd: 128</up,running>

    carp0: flags=49 <up,loopback,running>mtu 1500
            inet 192.168.4.3 netmask 0xffffff00
            carp: MASTER vhid 1 advbase 1 advskew 0
    carp1: flags=49 <up,loopback,running>mtu 1500
            inet bbb.yyy.xxx.25 netmask 0xffffff00
            carp: MASTER vhid 2 advbase 1 advskew 0
    carp2: flags=49 <up,loopback,running>mtu 1500
            inet 192.168.20.4 netmask 0xffffff00
            carp: MASTER vhid 3 advbase 1 advskew 0

    Secondary
    bge1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
            options=1b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging>inet 192.168.4.1 netmask 0xffffff00 broadcast 192.168.4.255
            inet6 fe80::216:36ff:fe13:a609%bge1 prefixlen 64 scopeid 0x4
            ether 00:16:36:13:a6:09
            media: Ethernet autoselect (1000baseTX <full-duplex>)
            status: active
    pfsync0: flags=41 <up,running>mtu 1348
            pfsync: syncdev: bge1 maxupd: 128</up,running>

    lo0: flags=8049 <up,loopback,running,multicast>mtu 16384
            inet 127.0.0.1 netmask 0xff000000
            inet6 ::1 prefixlen 128
            inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
    pflog0: flags=100 <promisc>mtu 33208
    carp0: flags=49 <up,loopback,running>mtu 1500
            inet 192.168.4.3 netmask 0xffffff00
            carp: BACKUP vhid 1 advbase 1 advskew 100
    carp1: flags=49 <up,loopback,running>mtu 1500
            inet bbb.xxx.yyy.25 netmask 0xffffff00
            carp: BACKUP vhid 2 advbase 1 advskew 100
    carp2: flags=49 <up,loopback,running>mtu 1500
            inet 192.168.20.4 netmask 0xffffff00
            carp: MASTER vhid 3 advbase 1 advskew 254</up,loopback,running></up,loopback,running></up,loopback,running></promisc></up,loopback,running,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging></up,broadcast,running,promisc,simplex,multicast></up,loopback,running></up,loopback,running></up,loopback,running></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging></up,broadcast,running,promisc,simplex,multicast>



  • on bge1, both machines need a unique ip in the  same subnet.



  • my bad… I copied & pasted the same info twice.
    Here is the correct info for the primary carp interface:

    fxp0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
            options=b <rxcsum,txcsum,vlan_mtu>inet 192.168.4.2 netmask 0xffffff00 broadcast 192.168.4.255
            inet6 fe80::202:b3ff:febb:9776%fxp0 prefixlen 64 scopeid 0x4
            ether 00:02:b3:bb:97:76
            media: Ethernet autoselect (100baseTX <full-duplex>)
            status: active</full-duplex></rxcsum,txcsum,vlan_mtu></up,broadcast,running,promisc,simplex,multicast>



  • Then I am not sure.  It looks okay to me.  Double check your switch ports, vlans, etc.

    Also refer to the archive, a lot of people have asked a LOT of questions about CARP.



  • thanks for your help.

    Just for grins, I am going to try to swap out the nic on the primary server. It is an intel 10/100 pro pci  plugged into a PCI-X slot. It should work fine and appears to be, but I have a PCI-X nic card available that came from another dell poweredge server that would be consistent with the other nics and also provide 1000 Mbs.



  • Couple of thoughts from one of the people that has asked a LOT of questions. :)

    1.  Have you looked at the wiki?
    2.  VHID's need to be unique not just for that set of carp interfaces, but for anything else that might be on that same network segment, as CARP is broadcast, not a peer-to-peer technology.
    3.  Triple check that you are using the correct subnet mask for your CARP IP's, and not /32.
    4.  Make sure that your CARP interface has an allow any any statement on it, just to be safe, and that the systems are connected by a crossover cable for additional security.
    5.  If you must do carp without a dedicated interface, make sure you have an allow statement for CARP and pfSync from the opposing system, and block traffic from any other hosts.

    That's about all I can think of.



  • I have done all that you mention.  I am using a dedicated interface for carp.  Both carp interfaces are connected via the same vlan and xmlrpc updates are successful.  I have not had the chance to swap out the nic for a pci-x nic yet, but I will start with a fresh install when I do. I will have to wait until the next maintenance window


Locked