Two VIPs showing as MASTER on both servers, all others are working fine.



  • I have two pfSense (2.0.1) boxes running carp with 5 virtual IPs. Of the five VIPs, two are showing as master on both boxes.

    Both boxes are DL380 G3s, they're connected through two Cisco 3750G's that are trunked together. I've putting the sync interface on it's own dumb switch, and that didn't help.

    The two VIPs that are failing are for VLANs, though other VLANs are working fine. Below is the ifconfig from both boxes. I appreciate any help or pointers that anyone can offer.

    Master
    bge0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
            options=8009b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate>ether 00:11:85:bc:a1:3d
            inet6 fe80::211:85ff:febc:a13d%bge0 prefixlen 64 scopeid 0x1
            inet 10.0.0.156 netmask 0xffffff00 broadcast 10.0.0.255
            nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
            status: active
    bge1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
            options=8009b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate>ether 00:11:85:bc:a1:3c
            inet6 fe80::211:85ff:febc:a13c%bge1 prefixlen 64 scopeid 0x2
            inet 10.3.0.2 netmask 0xffffff00 broadcast 10.3.0.255
            nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
            status: active
    em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
            options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>ether 00:02:a5:4e:a0:e8
            inet 10.3.253.1 netmask 0xfffffffc broadcast 10.3.253.3
            inet6 fe80::202:a5ff:fe4e:a0e8%em0 prefixlen 64 scopeid 0x3
            nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
            status: active
    em1: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
            options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>ether 00:02:a5:4e:a0:e9
            media: Ethernet autoselect
            status: no carrier
    lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
            options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000
            inet6 ::1 prefixlen 128
            inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
            nd6 options=3 <performnud,accept_rtadv>pfsync0: flags=41 <up,running>metric 0 mtu 1460
            pfsync: syncdev: em0 syncpeer: 10.3.253.2 maxupd: 128 syncok: 1
    pflog0: flags=100 <promisc>metric 0 mtu 33200
    enc0: flags=0<> metric 0 mtu 1536
    bge1_vlan2: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
            options=3 <rxcsum,txcsum>ether 00:11:85:bc:a1:3c
            inet6 fe80::211:85ff:febc:a13d%bge1_vlan2 prefixlen 64 scopeid 0x9
            inet 10.3.2.2 netmask 0xffffff00 broadcast 10.3.2.255
            nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
            status: active
            vlan: 2 parent interface: bge1
    bge1_vlan3: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
            options=3 <rxcsum,txcsum>ether 00:11:85:bc:a1:3c
            inet6 fe80::211:85ff:febc:a13d%bge1_vlan3 prefixlen 64 scopeid 0xa
            inet 10.3.3.2 netmask 0xffffff00 broadcast 10.3.3.255
            nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
            status: active
            vlan: 3 parent interface: bge1
    bge0_vlan254: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
            options=3 <rxcsum,txcsum>ether 00:11:85:bc:a1:3d
            inet6 fe80::211:85ff:febc:a13d%bge0_vlan254 prefixlen 64 scopeid 0xb
            inet 10.3.254.2 netmask 0xffffff00 broadcast 10.3.254.255
            nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
            status: active
            vlan: 254 parent interface: bge0
    vip1: flags=49 <up,loopback,running>metric 0 mtu 1500
            inet 10.0.0.155 netmask 0xffffff00
            carp: MASTER vhid 1 advbase 1 advskew 0
    vip2: flags=49 <up,loopback,running>metric 0 mtu 1500
            inet 10.3.0.1 netmask 0xffffff00
            carp: MASTER vhid 2 advbase 1 advskew 0
    vip3: flags=49 <up,loopback,running>metric 0 mtu 1500
            inet 10.3.2.1 netmask 0xffffff00
            carp: MASTER vhid 3 advbase 1 advskew 0
    vip4: flags=49 <up,loopback,running>metric 0 mtu 1500
            inet 10.3.3.1 netmask 0xffffff00
            carp: MASTER vhid 4 advbase 1 advskew 0
    vip5: flags=49 <up,loopback,running>metric 0 mtu 1500
            inet 10.3.254.1 netmask 0xffffff00
            carp: MASTER vhid 5 advbase 1 advskew 0

    Slave
    bge0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
            options=8009b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate>ether 00:0b:cd:9d:20:f3
            inet6 fe80::20b:cdff:fe9d:20f3%bge0 prefixlen 64 scopeid 0x1
            inet 10.0.0.157 netmask 0xffffff00 broadcast 10.0.0.255
            nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
            status: active
    bge1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
            options=8009b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate>ether 00:0b:cd:9d:20:f2
            inet6 fe80::20b:cdff:fe9d:20f2%bge1 prefixlen 64 scopeid 0x2
            inet 10.3.0.3 netmask 0xffffff00 broadcast 10.3.0.255
            nd6 options=3 <performnud,accept_rtadv>media: Ethernet 1000baseT <full-duplex>status: active
    em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
            options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>ether 00:11:0a:56:08:80
            inet 10.3.253.2 netmask 0xfffffffc broadcast 10.3.253.3
            inet6 fe80::211:aff:fe56:880%em0 prefixlen 64 scopeid 0x3
            nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
            status: active
    em1: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
            options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>ether 00:11:0a:56:08:81
            media: Ethernet autoselect
            status: no carrier
    lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
            options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000
            inet6 ::1 prefixlen 128
            inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
            nd6 options=3 <performnud,accept_rtadv>pfsync0: flags=41 <up,running>metric 0 mtu 1460
            pfsync: syncdev: em0 syncpeer: 224.0.0.240 maxupd: 128 syncok: 1
    pflog0: flags=100 <promisc>metric 0 mtu 33200
    enc0: flags=0<> metric 0 mtu 1536
    bge1_vlan2: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
            options=3 <rxcsum,txcsum>ether 00:0b:cd:9d:20:f2
            inet6 fe80::20b:cdff:fe9d:20f3%bge1_vlan2 prefixlen 64 scopeid 0x9
            inet 10.3.2.3 netmask 0xffffff00 broadcast 10.3.2.255
            nd6 options=3 <performnud,accept_rtadv>media: Ethernet 1000baseT <full-duplex>status: active
            vlan: 2 parent interface: bge1
    bge0_vlan3: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
            options=3 <rxcsum,txcsum>ether 00:0b:cd:9d:20:f3
            inet6 fe80::20b:cdff:fe9d:20f3%bge0_vlan3 prefixlen 64 scopeid 0xa
            inet 10.3.3.3 netmask 0xffffff00 broadcast 10.3.3.255
            nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
            status: active
            vlan: 3 parent interface: bge0
    bge0_vlan254: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
            options=3 <rxcsum,txcsum>ether 00:0b:cd:9d:20:f3
            inet6 fe80::20b:cdff:fe9d:20f3%bge0_vlan254 prefixlen 64 scopeid 0xb
            inet 10.3.254.3 netmask 0xffffff00 broadcast 10.3.254.255
            nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
            status: active
            vlan: 254 parent interface: bge0
    vip1: flags=49 <up,loopback,running>metric 0 mtu 1500
            inet 10.0.0.155 netmask 0xffffff00
            carp: BACKUP vhid 1 advbase 1 advskew 100
    vip2: flags=49 <up,loopback,running>metric 0 mtu 1500
            inet 10.3.0.1 netmask 0xffffff00
            carp: BACKUP vhid 2 advbase 1 advskew 100
    vip3: flags=49 <up,loopback,running>metric 0 mtu 1500
            inet 10.3.2.1 netmask 0xffffff00
            carp: MASTER vhid 3 advbase 1 advskew 100
    vip4: flags=49 <up,loopback,running>metric 0 mtu 1500
            inet 10.3.3.1 netmask 0xffffff00
            carp: MASTER vhid 4 advbase 1 advskew 100
    vip5: flags=49 <up,loopback,running>metric 0 mtu 1500
            inet 10.3.254.1 netmask 0xffffff00
            carp: BACKUP vhid 5 advbase 1 advskew 100</up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,simplex,multicast></promisc></up,running></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></broadcast,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate></up,broadcast,running,promisc,simplex,multicast></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,simplex,multicast></promisc></up,running></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></broadcast,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate></up,broadcast,running,promisc,simplex,multicast>



  • Make sure you have permit access to both firewall on sync interface to avoid config sync problems.

    The carp check is done in interface you choose to use carp.
    The only situation when both sides belives they are master is when you do not have communication between them.

    Are you using vmware or physical machines?



  • Thanks for the reply. I thought CARP and VIPs were using just the pfSync interface, didn't realize that they're using the actual interfaces as well (makes sense)

    I checked and for the VIPs that work, I'm able to ping across from interface to interface (for example: 10.3.0.2 to 10.3.0.3), for the two that are not working, I'm not able to ping across. Looks like that'd be my issue. Strange, firewalls are open on both boxes, and I'm not seeing anything logged. I'll start looking at the switches.

    To answer your other questions, I have an allow any:any on firewall for the pfSync interface and both boxes are physical.

    Thanks for your help with this, I think I'm on the right track now.



  • Taking a second look through everything, turns out I had the problematic vlans assigned to the wrong interface in pfSense. Once I got that straightened out, everything started working.

    facepalm

    Thanks again for the help.


Log in to reply