Both Server Master



  • Hi,

    i followed the howtos from https://doc.pfsense.org/index.php/Configuring_pfSense_Hardware_Redundancy_(CARP) and https://doc.pfsense.org/index.php/CARP_Configuration_Sync_Troubleshooting . sense02 is syncing everything i set on sense01 but carp status is on both MASTER. the vip for lan (10.13.6.2) is not reachable for the servers behind pfsense but they can ping the real lan ips. the log shows no denies. im sitting here for hours and have no idea what i can test anymore. my setup is

    sense01:

    vtnet0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
            options=80028 <vlan_mtu,jumbo_mtu,linkstate>ether 02:01:70:73:23:27
            inet 10.13.6.11 netmask 0xffffff00 broadcast 10.13.6.255
            inet6 fe80::1:70ff:fe73:2327%vtnet0 prefixlen 64 scopeid 0x1
            inet 10.13.6.2 netmask 0xffffff00 broadcast 10.13.6.255 vhid 10
            nd6 options=21 <performnud,auto_linklocal>media: Ethernet 10Gbase-T <full-duplex>status: active
            carp: MASTER vhid 10 advbase 1 advskew 0
    vtnet1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
            options=80028 <vlan_mtu,jumbo_mtu,linkstate>ether 02:01:7b:c1:0a:36
            inet 185.48.119.xy netmask 0xffffff00 broadcast 185.48.119.255
            inet6 fe80::1:7bff:fec1:a36%vtnet1 prefixlen 64 scopeid 0x2
            inet 185.48.119.xy netmask 0xffffff00 broadcast 185.48.119.255 vhid 1
            nd6 options=21 <performnud,auto_linklocal>media: Ethernet 10Gbase-T <full-duplex>status: active
            carp: MASTER vhid 1 advbase 1 advskew 0
    vtnet2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
            options=80028 <vlan_mtu,jumbo_mtu,linkstate>ether 02:01:9e:18:fb:c3
            inet 10.9.60.12 netmask 0xffffff00 broadcast 10.9.60.255
            inet6 fe80::1:9eff:fe18:fbc3%vtnet2 prefixlen 64 scopeid 0x3
            nd6 options=21 <performnud,auto_linklocal>media: Ethernet 10Gbase-T <full-duplex>status: active
    pflog0: flags=100 <promisc>metric 0 mtu 33144
    pfsync0: flags=41 <up,running>metric 0 mtu 1500
            pfsync: syncdev: vtnet2 syncpeer: 224.0.0.240 maxupd: 128 defer: on
            syncok: 1
    lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
            options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.1 netmask 0xff000000
            inet6 ::1 prefixlen 128
            inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
            nd6 options=21 <performnud,auto_linklocal>enc0: flags=0<> metric 0 mtu 1536
            nd6 options=21 <performnud,auto_linklocal>ovpns1: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500
            options=80000 <linkstate>inet6 fe80::10f1:46e0:3f2e:7c86%ovpns1 prefixlen 64 scopeid 0x8
            inet 192.168.11.1 –> 192.168.11.2 netmask 0xffffffff
            nd6 options=21 <performnud,auto_linklocal>Opened by PID 86746

    sense02

    vtnet0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
            options=80028 <vlan_mtu,jumbo_mtu,linkstate>ether 02:01:04:f1:e8:8e
            inet 10.13.6.14 netmask 0xffffff00 broadcast 10.13.6.255
            inet6 fe80::1:4ff:fef1:e88e%vtnet0 prefixlen 64 scopeid 0x1
            inet 10.13.6.2 netmask 0xffffff00 broadcast 10.13.6.255 vhid 10
            nd6 options=21 <performnud,auto_linklocal>media: Ethernet 10Gbase-T <full-duplex>status: active
            carp: MASTER vhid 10 advbase 1 advskew 254
    vtnet1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
            options=80028 <vlan_mtu,jumbo_mtu,linkstate>ether 02:01:a9:f9:f8:1c
            inet 185.48.119.xy netmask 0xffffff00 broadcast 185.48.119.255
            inet6 fe80::1:a9ff:fef9:f81c%vtnet1 prefixlen 64 scopeid 0x2
            inet 185.48.119.xy netmask 0xffffff00 broadcast 185.48.119.255 vhid 1
            nd6 options=21 <performnud,auto_linklocal>media: Ethernet 10Gbase-T <full-duplex>status: active
            carp: MASTER vhid 1 advbase 1 advskew 254
    vtnet2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
            options=80028 <vlan_mtu,jumbo_mtu,linkstate>ether 02:01:b5:c5:83:82
            inet 10.9.60.11 netmask 0xffffff00 broadcast 10.9.60.255
            inet6 fe80::1:b5ff:fec5:8382%vtnet2 prefixlen 64 scopeid 0x3
            nd6 options=21 <performnud,auto_linklocal>media: Ethernet 10Gbase-T <full-duplex>status: active
    pflog0: flags=100 <promisc>metric 0 mtu 33144
    pfsync0: flags=41 <up,running>metric 0 mtu 1500
            pfsync: syncdev: vtnet2 syncpeer: 224.0.0.240 maxupd: 128 defer: on
            syncok: 1
    lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
            options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.1 netmask 0xff000000
            inet6 ::1 prefixlen 128
            inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
            nd6 options=21 <performnud,auto_linklocal>enc0: flags=0<> metric 0 mtu 1536
            nd6 options=21 <performnud,auto_linklocal>ovpns1: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500
            options=80000 <linkstate>inet6 fe80::bc28:562e:55ca:7204%ovpns1 prefixlen 64 scopeid 0x8
            inet 192.168.11.1 --> 192.168.11.2 netmask 0xffffffff
            nd6 options=21 <performnud,auto_linklocal>Opened by PID 59877

    Im using 2.2 and did this setup on a hoster who is using KVM. virtio drivers are loaded https://doc.pfsense.org/index.php/VirtIO_Driver_Support

    Any tips to debug this are welcome

    Best,

    Mike</performnud,auto_linklocal></linkstate></up,pointopoint,running,multicast></performnud,auto_linklocal></performnud,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></up,running></promisc></full-duplex></performnud,auto_linklocal></vlan_mtu,jumbo_mtu,linkstate></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></vlan_mtu,jumbo_mtu,linkstate></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,auto_linklocal></vlan_mtu,jumbo_mtu,linkstate></up,broadcast,running,promisc,simplex,multicast></performnud,auto_linklocal></linkstate></up,pointopoint,running,multicast></performnud,auto_linklocal></performnud,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></up,running></promisc></full-duplex></performnud,auto_linklocal></vlan_mtu,jumbo_mtu,linkstate></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></vlan_mtu,jumbo_mtu,linkstate></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,auto_linklocal></vlan_mtu,jumbo_mtu,linkstate></up,broadcast,running,promisc,simplex,multicast>



  • I did a tcpdump now on both server

    
    00:00:01.410034 IP 185.48.119.1xy > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36
    .. always the same line
    
    
    
    00:00:01.010122 IP 185.48.119.2xy > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36
    ...always the same line
    
    

    I guess i should see the addvertisement from serv01 at server02?! But i can ping each server from the other one on the real ip. I turned on logging for the rule between these ips and can see the ping but i don't see logs for carp. Is this normal or should i see this too in the logs? If yes, it seems to be a problem on the network (switch).

    Best,

    Mike



  • Since my hoster is also using KVM i found this thread https://forums.freebsd.org/threads/issues-with-carp-under-qemu.22398/
    I tried to set sysctl net.inet.carp.drop_echoed but it is not available. Was it renamed or is it just not part of the pfsense kernel anymore?