Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    PFSense Private network interface disable very frequetly

    Scheduled Pinned Locked Moved General pfSense Questions
    12 Posts 3 Posters 1.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C
      chandranath
      last edited by chandranath

      We found this information when firewall interface is stop sending traffic , We have Baremetal in IBM Cloud and we manage only Baremetal and backend switch managed by IBM Cloud and they confirmed they did not find any issue.
      Error : laggport: ix0 flags=1c<COLLECTING>

      ifconfig lagg0

      lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
      options=8500b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO>
      ether 0c:c4:7a:8f:7c:fc
      inet6 fe80::ec4:7aff:fe8f:7cfc%lagg0 prefixlen 64 scopeid 0xb
      inet 10.45.30.76 netmask 0xffffffc0 broadcast 10.45.30.127
      inet 10.45.30.67 netmask 0xffffffc0 broadcast 10.45.30.127 vhid 11
      nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
      media: Ethernet autoselect
      status: active
      carp: MASTER vhid 11 advbase 5 advskew 0
      groups: lagg
      laggproto lacp lagghash l2,l3,l4
      laggport: ix0 flags=8<COLLECTING>
      laggport: ix2 flags=8<COLLECTING>
      Collapse

      During an outage out secondary firewall became a master and was exchanging VRRP.
      10.45.30.76 . Primary and 10.45.30.85 secondary firewall

      tcpdump -l -i lagg0 -nn "vrrp"

      tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
      listening on lagg0, link-type EN10MB (Ethernet), capture size 262144 bytes
      15:55:53.130271 IP 10.45.30.76 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 240, authtype none, intvl 5s, length 36
      15:55:53.477548 IP 10.45.30.85 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 100, authtype none, intvl 5s, length 36
      15:55:58.877258 IP 10.45.30.85 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 100, authtype none, intvl 5s, length 36
      15:55:59.109346 IP 10.45.30.76 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 240, authtype none, intvl 5s, length 36
      15:56:04.269434 IP 10.45.30.85 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 100, authtype none, intvl 5s, length 36
      Collapse

      Nov 26 13:52:32 firewall1 kernel: carp: demoted by 240 to 240 (send error 50 on lagg0)
      Nov 26 13:52:32 firewall1 kernel: carp: 10@lagg1: MASTER -> BACKUP (more frequent advertisement received)
      Nov 26 13:52:32 firewall1 kernel: carp: 13@lagg1.816: MASTER -> BACKUP (more frequent advertisement received)
      Nov 26 13:52:32 firewall1 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg1: 3
      Nov 26 13:52:32 firewall1 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg1.816: 3

      Please let us if you need more information.

      1 Reply Last reply Reply Quote 0
      • C
        chandranath
        last edited by

        =================================================
        2 identical hardware/baremetal, used for pfSense HA pair.

        Intel(R) Xeon(R) CPU E3-1270 v3 @ 3.50GHz
        8 CPUs: 1 package(s) x 4 core(s) x 2 hardware threads

        pfSense version:
        2.4.4-RELEASE-p3 (amd64)
        FreeBSD 11.2-RELEASE-p10

        Setup has multiple 2 LACP bonds, VLANs, aliases, NAT, CARP, VPN tunnels, Suricata IPS.
        Bandwidth CPU utilization is around single digit.

        We see CARP being triggered on private interface post LACP bonding errors. Please find error on previous post.
        The CARP switches only private interface traffic, resulting split brain. Tweaking "net.inet.carp.senderr_demotion_factor" value affected complete switchover.
        However we are not able to the root cause of bonding failure.
        The frequency is quite regular and mostly during start of the day.
        The same hardware used to work with other firewall model with no issues for more than 24 months.
        The issue happens on both firewalls.
        Backend switch did not show any errors on interface.

        1 Reply Last reply Reply Quote 0
        • NollipfSenseN
          NollipfSense
          last edited by

          Hopefully, others more advance might be able to help you so I'll give it a bump!

          pfSense+ 23.09 Lenovo Thinkcentre M93P SFF Quadcore i7 dual Raid-ZFS 128GB-SSD 32GB-RAM PCI-Intel i350-t4 NIC, -Intel QAT 8950.
          pfSense+ 23.09 VM-Proxmox, Dell Precision Xeon-W2155 Nvme 500GB-ZFS 128GB-RAM PCIe-Intel i350-t4, Intel QAT-8950, P-cloud.

          C 2 Replies Last reply Reply Quote 0
          • C
            chandranath
            last edited by chandranath

            Two identical hardware/baremetal, used for pfSense HA pair.
            8 CPUs: 1 package(s) x 4 core(s) x 2 hardware threads
            pfSense version:
            2.4.4-RELEASE-p3 (amd64)
            FreeBSD 11.2-RELEASE-p10
            Setup has multiple VLANs, 2 LACP bonds, aliases, NAT, CARP, VPN tunnels, Suricata IPS.
            Bandwidth CPU utilization is around single digit.
            Network Connections Intel® i210 Gigabit Ethernet Controllers: The NIC card is integrated into the motherboard
            • Two (2) i210 LAN controllers for LAN1/LAN2
            • Two (2) RJ-45 rear I/O panel connectors with Link and Activity LEDs

            1 Reply Last reply Reply Quote 0
            • C
              chandranath
              last edited by chandranath

              Please find an attached screen shot of both Primary and secondary firewall basic configuration.

              Firewall 1: Primary
              c290f32c-5fa7-4260-90bc-d587fa04758a-image.png
              62f93df3-1a1a-49cd-9bc5-3ab164fdc01f-image.png
              f42f4e3d-a69b-405e-8633-d67e2a0a4693-image.png
              ![a184b5cc-bb7c-4b66-9bf8-5fcf898b153d-image.png]
              beedf7fe-8e48-4ef1-a840-2cf1a8b75a5d-image.png (/assets/uploads/files/1577985712398-a184b5cc-bb7c-4b66-9bf8-5fcf898b153d-image.png)
              e121ba5d-212e-4732-9ccd-a69aa90ac268-image.png When issue happens: Below is CARP status – Firewall1 -primary
              We found this information when firewall interface is stop sending traffic , We have Baremetal in IBM Cloud and we manage only Baremetal and backend switch managed by IBM Cloud and they confirmed they did not find any issue.
              Error : laggport: ix0 flags=1c<COLLECTING>
              ifconfig lagg0
              lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
              options=8500b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO>
              ether 0c:c4:7a:8f:7c:fc
              inet6 fe80::ec4:7aff:fe8f:7cfc%lagg0 prefixlen 64 scopeid 0xb
              inet 10.45.30.76 netmask 0xffffffc0 broadcast 10.45.30.127
              inet 10.45.30.67 netmask 0xffffffc0 broadcast 10.45.30.127 vhid 11
              nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
              media: Ethernet autoselect
              status: active
              carp: MASTER vhid 11 advbase 5 advskew 0
              groups: lagg
              laggproto lacp lagghash l2,l3,l4
              laggport: ix0 flags=8<COLLECTING>
              laggport: ix2 flags=8<COLLECTING>
              Collapse
              During an outage out secondary firewall became a master and was exchanging VRRP.
              10.45.30.76 . Primary and 10.45.30.85 secondary firewall
              tcpdump -l -i lagg0 -nn "vrrp"
              tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
              listening on lagg0, link-type EN10MB (Ethernet), capture size 262144 bytes
              15:55:53.130271 IP 10.45.30.76 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 240, authtype none, intvl 5s, length 36
              15:55:53.477548 IP 10.45.30.85 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 100, authtype none, intvl 5s, length 36
              15:55:58.877258 IP 10.45.30.85 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 100, authtype none, intvl 5s, length 36
              15:55:59.109346 IP 10.45.30.76 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 240, authtype none, intvl 5s, length 36
              15:56:04.269434 IP 10.45.30.85 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 100, authtype none, intvl 5s, length 36
              Collapse
              Nov 26 13:52:32 firewall1 kernel: carp: demoted by 240 to 240 (send error 50 on lagg0)
              Nov 26 13:52:32 firewall1 kernel: carp: 10@lagg1: MASTER -> BACKUP (more frequent advertisement received)
              Nov 26 13:52:32 firewall1 kernel: carp: 13@lagg1.816: MASTER -> BACKUP (more frequent advertisement received)
              Nov 26 13:52:32 firewall1 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg1: 3
              Nov 26 13:52:32 firewall1 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg1.816: 3
              ca9c5f5d-ace5-4d19-b357-fd87333950fd-image.png

              Firewall2 -Secondary
              5c1cbc30-9b95-4527-8f59-e8a86c0c82e6-image.png
              f1411b1b-26d2-4218-9e34-874b5e4edd31-image.png
              e3c295c2-ec82-4370-9628-b39744e88fa6-image.png
              22b37b4a-c631-49b2-a434-57f93492fbd9-image.png

              When issue happens: Below is CARP status – Firewall2 -Secondary
              59fb8ad2-6e22-4845-8768-2886ee5b3339-image.png

              After few days faced similar issue on firewall2-secondary ( Current Master)
              6f2532f8-721a-44aa-b2de-16354c8ad1d2-image.png

              1 Reply Last reply Reply Quote 0
              • C
                chandranath @NollipfSense
                last edited by

                This post is deleted!
                1 Reply Last reply Reply Quote 0
                • C
                  chandranath @NollipfSense
                  last edited by chandranath

                  This post is deleted!
                  C 1 Reply Last reply Reply Quote 0
                  • C
                    chandranath @chandranath
                    last edited by

                    This post is deleted!
                    1 Reply Last reply Reply Quote 0
                    • C
                      chandranath
                      last edited by chandranath

                      This post is deleted!
                      1 Reply Last reply Reply Quote 0
                      • C
                        chetanwa
                        last edited by

                        Hello..
                        If someone gets similar issue, please try disabling LACP strict mode.
                        It worked in our case.

                        All the best

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post
                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.