Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    PFSense Private network interface disable very frequetly

    Scheduled Pinned Locked Moved General pfSense Questions
    12 Posts 3 Posters 1.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • NollipfSenseN
      NollipfSense @chandranath
      last edited by

      @chandranath You realize that to help us understand what is happening with your network you'll need to provide relevant info such as screen shot of your configuration.

      pfSense+ 23.09 Lenovo Thinkcentre M93P SFF Quadcore i7 dual Raid-ZFS 128GB-SSD 32GB-RAM PCI-Intel i350-t4 NIC, -Intel QAT 8950.
      pfSense+ 23.09 VM-Proxmox, Dell Precision Xeon-W2155 Nvme 500GB-ZFS 128GB-RAM PCIe-Intel i350-t4, Intel QAT-8950, P-cloud.

      1 Reply Last reply Reply Quote 0
      • C
        chandranath
        last edited by chandranath

        We found this information when firewall interface is stop sending traffic , We have Baremetal in IBM Cloud and we manage only Baremetal and backend switch managed by IBM Cloud and they confirmed they did not find any issue.
        Error : laggport: ix0 flags=1c<COLLECTING>

        ifconfig lagg0

        lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=8500b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO>
        ether 0c:c4:7a:8f:7c:fc
        inet6 fe80::ec4:7aff:fe8f:7cfc%lagg0 prefixlen 64 scopeid 0xb
        inet 10.45.30.76 netmask 0xffffffc0 broadcast 10.45.30.127
        inet 10.45.30.67 netmask 0xffffffc0 broadcast 10.45.30.127 vhid 11
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        carp: MASTER vhid 11 advbase 5 advskew 0
        groups: lagg
        laggproto lacp lagghash l2,l3,l4
        laggport: ix0 flags=8<COLLECTING>
        laggport: ix2 flags=8<COLLECTING>
        Collapse

        During an outage out secondary firewall became a master and was exchanging VRRP.
        10.45.30.76 . Primary and 10.45.30.85 secondary firewall

        tcpdump -l -i lagg0 -nn "vrrp"

        tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
        listening on lagg0, link-type EN10MB (Ethernet), capture size 262144 bytes
        15:55:53.130271 IP 10.45.30.76 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 240, authtype none, intvl 5s, length 36
        15:55:53.477548 IP 10.45.30.85 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 100, authtype none, intvl 5s, length 36
        15:55:58.877258 IP 10.45.30.85 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 100, authtype none, intvl 5s, length 36
        15:55:59.109346 IP 10.45.30.76 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 240, authtype none, intvl 5s, length 36
        15:56:04.269434 IP 10.45.30.85 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 100, authtype none, intvl 5s, length 36
        Collapse

        Nov 26 13:52:32 firewall1 kernel: carp: demoted by 240 to 240 (send error 50 on lagg0)
        Nov 26 13:52:32 firewall1 kernel: carp: 10@lagg1: MASTER -> BACKUP (more frequent advertisement received)
        Nov 26 13:52:32 firewall1 kernel: carp: 13@lagg1.816: MASTER -> BACKUP (more frequent advertisement received)
        Nov 26 13:52:32 firewall1 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg1: 3
        Nov 26 13:52:32 firewall1 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg1.816: 3

        Please let us if you need more information.

        1 Reply Last reply Reply Quote 0
        • C
          chandranath
          last edited by

          =================================================
          2 identical hardware/baremetal, used for pfSense HA pair.

          Intel(R) Xeon(R) CPU E3-1270 v3 @ 3.50GHz
          8 CPUs: 1 package(s) x 4 core(s) x 2 hardware threads

          pfSense version:
          2.4.4-RELEASE-p3 (amd64)
          FreeBSD 11.2-RELEASE-p10

          Setup has multiple 2 LACP bonds, VLANs, aliases, NAT, CARP, VPN tunnels, Suricata IPS.
          Bandwidth CPU utilization is around single digit.

          We see CARP being triggered on private interface post LACP bonding errors. Please find error on previous post.
          The CARP switches only private interface traffic, resulting split brain. Tweaking "net.inet.carp.senderr_demotion_factor" value affected complete switchover.
          However we are not able to the root cause of bonding failure.
          The frequency is quite regular and mostly during start of the day.
          The same hardware used to work with other firewall model with no issues for more than 24 months.
          The issue happens on both firewalls.
          Backend switch did not show any errors on interface.

          1 Reply Last reply Reply Quote 0
          • NollipfSenseN
            NollipfSense
            last edited by

            Hopefully, others more advance might be able to help you so I'll give it a bump!

            pfSense+ 23.09 Lenovo Thinkcentre M93P SFF Quadcore i7 dual Raid-ZFS 128GB-SSD 32GB-RAM PCI-Intel i350-t4 NIC, -Intel QAT 8950.
            pfSense+ 23.09 VM-Proxmox, Dell Precision Xeon-W2155 Nvme 500GB-ZFS 128GB-RAM PCIe-Intel i350-t4, Intel QAT-8950, P-cloud.

            C 2 Replies Last reply Reply Quote 0
            • C
              chandranath
              last edited by chandranath

              Two identical hardware/baremetal, used for pfSense HA pair.
              8 CPUs: 1 package(s) x 4 core(s) x 2 hardware threads
              pfSense version:
              2.4.4-RELEASE-p3 (amd64)
              FreeBSD 11.2-RELEASE-p10
              Setup has multiple VLANs, 2 LACP bonds, aliases, NAT, CARP, VPN tunnels, Suricata IPS.
              Bandwidth CPU utilization is around single digit.
              Network Connections Intel® i210 Gigabit Ethernet Controllers: The NIC card is integrated into the motherboard
              • Two (2) i210 LAN controllers for LAN1/LAN2
              • Two (2) RJ-45 rear I/O panel connectors with Link and Activity LEDs

              1 Reply Last reply Reply Quote 0
              • C
                chandranath
                last edited by chandranath

                Please find an attached screen shot of both Primary and secondary firewall basic configuration.

                Firewall 1: Primary
                c290f32c-5fa7-4260-90bc-d587fa04758a-image.png
                62f93df3-1a1a-49cd-9bc5-3ab164fdc01f-image.png
                f42f4e3d-a69b-405e-8633-d67e2a0a4693-image.png
                ![a184b5cc-bb7c-4b66-9bf8-5fcf898b153d-image.png]
                beedf7fe-8e48-4ef1-a840-2cf1a8b75a5d-image.png (/assets/uploads/files/1577985712398-a184b5cc-bb7c-4b66-9bf8-5fcf898b153d-image.png)
                e121ba5d-212e-4732-9ccd-a69aa90ac268-image.png When issue happens: Below is CARP status – Firewall1 -primary
                We found this information when firewall interface is stop sending traffic , We have Baremetal in IBM Cloud and we manage only Baremetal and backend switch managed by IBM Cloud and they confirmed they did not find any issue.
                Error : laggport: ix0 flags=1c<COLLECTING>
                ifconfig lagg0
                lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
                options=8500b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO>
                ether 0c:c4:7a:8f:7c:fc
                inet6 fe80::ec4:7aff:fe8f:7cfc%lagg0 prefixlen 64 scopeid 0xb
                inet 10.45.30.76 netmask 0xffffffc0 broadcast 10.45.30.127
                inet 10.45.30.67 netmask 0xffffffc0 broadcast 10.45.30.127 vhid 11
                nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
                media: Ethernet autoselect
                status: active
                carp: MASTER vhid 11 advbase 5 advskew 0
                groups: lagg
                laggproto lacp lagghash l2,l3,l4
                laggport: ix0 flags=8<COLLECTING>
                laggport: ix2 flags=8<COLLECTING>
                Collapse
                During an outage out secondary firewall became a master and was exchanging VRRP.
                10.45.30.76 . Primary and 10.45.30.85 secondary firewall
                tcpdump -l -i lagg0 -nn "vrrp"
                tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
                listening on lagg0, link-type EN10MB (Ethernet), capture size 262144 bytes
                15:55:53.130271 IP 10.45.30.76 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 240, authtype none, intvl 5s, length 36
                15:55:53.477548 IP 10.45.30.85 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 100, authtype none, intvl 5s, length 36
                15:55:58.877258 IP 10.45.30.85 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 100, authtype none, intvl 5s, length 36
                15:55:59.109346 IP 10.45.30.76 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 240, authtype none, intvl 5s, length 36
                15:56:04.269434 IP 10.45.30.85 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 100, authtype none, intvl 5s, length 36
                Collapse
                Nov 26 13:52:32 firewall1 kernel: carp: demoted by 240 to 240 (send error 50 on lagg0)
                Nov 26 13:52:32 firewall1 kernel: carp: 10@lagg1: MASTER -> BACKUP (more frequent advertisement received)
                Nov 26 13:52:32 firewall1 kernel: carp: 13@lagg1.816: MASTER -> BACKUP (more frequent advertisement received)
                Nov 26 13:52:32 firewall1 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg1: 3
                Nov 26 13:52:32 firewall1 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg1.816: 3
                ca9c5f5d-ace5-4d19-b357-fd87333950fd-image.png

                Firewall2 -Secondary
                5c1cbc30-9b95-4527-8f59-e8a86c0c82e6-image.png
                f1411b1b-26d2-4218-9e34-874b5e4edd31-image.png
                e3c295c2-ec82-4370-9628-b39744e88fa6-image.png
                22b37b4a-c631-49b2-a434-57f93492fbd9-image.png

                When issue happens: Below is CARP status – Firewall2 -Secondary
                59fb8ad2-6e22-4845-8768-2886ee5b3339-image.png

                After few days faced similar issue on firewall2-secondary ( Current Master)
                6f2532f8-721a-44aa-b2de-16354c8ad1d2-image.png

                1 Reply Last reply Reply Quote 0
                • C
                  chandranath @NollipfSense
                  last edited by

                  This post is deleted!
                  1 Reply Last reply Reply Quote 0
                  • C
                    chandranath @NollipfSense
                    last edited by chandranath

                    This post is deleted!
                    C 1 Reply Last reply Reply Quote 0
                    • C
                      chandranath @chandranath
                      last edited by

                      This post is deleted!
                      1 Reply Last reply Reply Quote 0
                      • C
                        chandranath
                        last edited by chandranath

                        This post is deleted!
                        1 Reply Last reply Reply Quote 0
                        • C
                          chetanwa
                          last edited by

                          Hello..
                          If someone gets similar issue, please try disabling LACP strict mode.
                          It worked in our case.

                          All the best

                          1 Reply Last reply Reply Quote 0
                          • First post
                            Last post
                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.