PFSense Private network interface disable very frequetly
-
We have a PFSense running on Baremetal and noticed that PFSense private interface is disabled automatically and this is happening very frequently. We had to restart baremetal to recover this issue. Both PFSense and Switch side interface status show UP but no traffic passing through an interface and even local gateway was not reachable. This is happening for Private interface only... We have enabled IPS Suricata
-
@chandranath You realize that to help us understand what is happening with your network you'll need to provide relevant info such as screen shot of your configuration.
-
We found this information when firewall interface is stop sending traffic , We have Baremetal in IBM Cloud and we manage only Baremetal and backend switch managed by IBM Cloud and they confirmed they did not find any issue.
Error : laggport: ix0 flags=1c<COLLECTING>ifconfig lagg0
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=8500b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO>
ether 0c:c4:7a:8f:7c:fc
inet6 fe80::ec4:7aff:fe8f:7cfc%lagg0 prefixlen 64 scopeid 0xb
inet 10.45.30.76 netmask 0xffffffc0 broadcast 10.45.30.127
inet 10.45.30.67 netmask 0xffffffc0 broadcast 10.45.30.127 vhid 11
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
carp: MASTER vhid 11 advbase 5 advskew 0
groups: lagg
laggproto lacp lagghash l2,l3,l4
laggport: ix0 flags=8<COLLECTING>
laggport: ix2 flags=8<COLLECTING>
CollapseDuring an outage out secondary firewall became a master and was exchanging VRRP.
10.45.30.76 . Primary and 10.45.30.85 secondary firewalltcpdump -l -i lagg0 -nn "vrrp"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lagg0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:55:53.130271 IP 10.45.30.76 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 240, authtype none, intvl 5s, length 36
15:55:53.477548 IP 10.45.30.85 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 100, authtype none, intvl 5s, length 36
15:55:58.877258 IP 10.45.30.85 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 100, authtype none, intvl 5s, length 36
15:55:59.109346 IP 10.45.30.76 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 240, authtype none, intvl 5s, length 36
15:56:04.269434 IP 10.45.30.85 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 100, authtype none, intvl 5s, length 36
Collapse
Nov 26 13:52:32 firewall1 kernel: carp: demoted by 240 to 240 (send error 50 on lagg0)
Nov 26 13:52:32 firewall1 kernel: carp: 10@lagg1: MASTER -> BACKUP (more frequent advertisement received)
Nov 26 13:52:32 firewall1 kernel: carp: 13@lagg1.816: MASTER -> BACKUP (more frequent advertisement received)
Nov 26 13:52:32 firewall1 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg1: 3
Nov 26 13:52:32 firewall1 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg1.816: 3Please let us if you need more information.
-
=================================================
2 identical hardware/baremetal, used for pfSense HA pair.Intel(R) Xeon(R) CPU E3-1270 v3 @ 3.50GHz
8 CPUs: 1 package(s) x 4 core(s) x 2 hardware threadspfSense version:
2.4.4-RELEASE-p3 (amd64)
FreeBSD 11.2-RELEASE-p10Setup has multiple 2 LACP bonds, VLANs, aliases, NAT, CARP, VPN tunnels, Suricata IPS.
Bandwidth CPU utilization is around single digit.We see CARP being triggered on private interface post LACP bonding errors. Please find error on previous post.
The CARP switches only private interface traffic, resulting split brain. Tweaking "net.inet.carp.senderr_demotion_factor" value affected complete switchover.
However we are not able to the root cause of bonding failure.
The frequency is quite regular and mostly during start of the day.
The same hardware used to work with other firewall model with no issues for more than 24 months.
The issue happens on both firewalls.
Backend switch did not show any errors on interface. -
Hopefully, others more advance might be able to help you so I'll give it a bump!
-
Two identical hardware/baremetal, used for pfSense HA pair.
8 CPUs: 1 package(s) x 4 core(s) x 2 hardware threads
pfSense version:
2.4.4-RELEASE-p3 (amd64)
FreeBSD 11.2-RELEASE-p10
Setup has multiple VLANs, 2 LACP bonds, aliases, NAT, CARP, VPN tunnels, Suricata IPS.
Bandwidth CPU utilization is around single digit.
Network Connections Intel
i210 Gigabit Ethernet Controllers: The NIC card is integrated into the motherboard
• Two (2) i210 LAN controllers for LAN1/LAN2
• Two (2) RJ-45 rear I/O panel connectors with Link and Activity LEDs -
Please find an attached screen shot of both Primary and secondary firewall basic configuration.
Firewall 1: Primary
![a184b5cc-bb7c-4b66-9bf8-5fcf898b153d-image.png]
(/assets/uploads/files/1577985712398-a184b5cc-bb7c-4b66-9bf8-5fcf898b153d-image.png)
When issue happens: Below is CARP status – Firewall1 -primary
We found this information when firewall interface is stop sending traffic , We have Baremetal in IBM Cloud and we manage only Baremetal and backend switch managed by IBM Cloud and they confirmed they did not find any issue.
Error : laggport: ix0 flags=1c<COLLECTING>
ifconfig lagg0
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=8500b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO>
ether 0c:c4:7a:8f:7c:fc
inet6 fe80::ec4:7aff:fe8f:7cfc%lagg0 prefixlen 64 scopeid 0xb
inet 10.45.30.76 netmask 0xffffffc0 broadcast 10.45.30.127
inet 10.45.30.67 netmask 0xffffffc0 broadcast 10.45.30.127 vhid 11
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
carp: MASTER vhid 11 advbase 5 advskew 0
groups: lagg
laggproto lacp lagghash l2,l3,l4
laggport: ix0 flags=8<COLLECTING>
laggport: ix2 flags=8<COLLECTING>
Collapse
During an outage out secondary firewall became a master and was exchanging VRRP.
10.45.30.76 . Primary and 10.45.30.85 secondary firewall
tcpdump -l -i lagg0 -nn "vrrp"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lagg0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:55:53.130271 IP 10.45.30.76 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 240, authtype none, intvl 5s, length 36
15:55:53.477548 IP 10.45.30.85 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 100, authtype none, intvl 5s, length 36
15:55:58.877258 IP 10.45.30.85 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 100, authtype none, intvl 5s, length 36
15:55:59.109346 IP 10.45.30.76 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 240, authtype none, intvl 5s, length 36
15:56:04.269434 IP 10.45.30.85 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 100, authtype none, intvl 5s, length 36
Collapse
Nov 26 13:52:32 firewall1 kernel: carp: demoted by 240 to 240 (send error 50 on lagg0)
Nov 26 13:52:32 firewall1 kernel: carp: 10@lagg1: MASTER -> BACKUP (more frequent advertisement received)
Nov 26 13:52:32 firewall1 kernel: carp: 13@lagg1.816: MASTER -> BACKUP (more frequent advertisement received)
Nov 26 13:52:32 firewall1 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg1: 3
Nov 26 13:52:32 firewall1 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg1.816: 3
Firewall2 -Secondary
When issue happens: Below is CARP status – Firewall2 -Secondary
After few days faced similar issue on firewall2-secondary ( Current Master)
-
This post is deleted! -
This post is deleted! -
This post is deleted! -
This post is deleted! -
Hello..
If someone gets similar issue, please try disabling LACP strict mode.
It worked in our case.All the best
