2.4.1: pfSense lockup with CARP on bridge interface

bkraptor

Want to bump this thread as I had another attempt at updating both boxes. I thought this was related to the pfBlockerNG issue that everyone seems to be having, which should have been fixed with pfSense 2.4.1 and the latest pfBlockerNG (that I had installed, but not activated). What I observed: the APU2C4 box was upgraded, but left in a CARP backup state for 24h. It did not show any signs of locking up for the whole duration. The moment it became CARP master it only took ~5 minutes to get it to lock up. Same thing then happened for the SG-4860.

I believe this issue is different from the pfBlockerNG issue, as the processess in this case get stuck in the L state, compared to the D state for the pfBlockerNG issue.

Guest

The main features I use:

bridges that contain:
VLAN tagged interfaces (no PPPoE)
wireless interfaces with multiple (3) virtual SSIDs
CARP running on bridge interfaces
NAT in various flavors
OpenVPN
OpenBGPd
no explicit shaping/policing/queueing configured

In version 2.4.0 some VLAN labeling (to long names) problems occurs if I was reading it right here through the forum.
In Version 2.4.1 are some hard problems using VLANs at the PPPoE connection!
In the early version 2.4.2 this problems are gone, but this must not be meaning now that the version is stable as others!

Across the whole forum therre are many problems updating or upgrading to a 2.4.x version, but often or many users
were installing it fresh and full on an storage and played back their config xml file and all was right then. If I am in
your situation I would try out installing 2.4.0 ADI image on the SG-4860 and the CE Edition on the APU2C4 and
in front of that I would proof the firmware images too and/or update them both if needed. And then play back the
config xml file.

bkraptor

Thanks for the tip, but I think this is a basic 2.4 bug. I can easily replicate the issue on a freshly installed pfSense VM by sending traffic via a CARP IP on a bridge interface.

https://redmine.pfsense.org/issues/8056

wwwdrich

I hate to make a "me too" post, but I'm seeing the same thing on 2.4.1 with a clean install and a config.xml loaded from my old system. The only way I have found to keep the firewall up is to turn all of my CARP VIPs into IP Aliases and turn off my secondary firewall.

When I get the hang, hitting ctrl-t on the console gives me variations on:

load: 7.22  cmd: ifconfig 88749 [*carp_if] 11.76r 0.00u 0.00s 0% 2704k

so it looks like it is spinning in carp_if.

webwiz

Another me too, as well.

All our pfSense firewalls that are using Bridged interfaces and CARP will freeze as soon as traffic starts passing across the Bridge Interface.

Had to reinstall 2.3 to get firewalls working again.

bkraptor

Hoping this gets some traction, but so far no activity on the linked bug report…

webwiz

Does anyone know if this bug has been fixed in 2.4.2?

bkraptor

I re-tested with 2.4.2 and I still see the same behavior.

Until the pfSense team acknowledges https://redmine.pfsense.org/issues/8056 I don't see how we'll get a fix.

bkraptor

Bumping this thread in the hope that someone on the pfSense team acknowledges this issue.

webwiz

Looking through the following bug report it looks to be an bug in Free BSD when an interface used as a bridge member has a CARP IP;

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200319

Has anyone tested to see if the problem persists if CARP IP's are removed from the interfaces that are members of the bridge?

gtoso

Hi,
I have a similar problem but not with BRIDGE but LAGG (with LACP enabled).
the problem started after the upgrade of one of two firewall in CARP from 2.3.4p1 to 2.4.2p1.

Could it be related to this bug?
ASAP I will try to better describe my problem.

gtoso

Hi,
I have this scenario:
2 firewall Dell PowerEdge R310 with these network adapters:
2 embedded Broadcom NetXtreme II Gigabit Ethernet
1 Intel(R) Gigabit ET Quad Port Server Adapter
firewall 1: PfSense 2.3.4-RELEASE-p1 (amd64) installed on HDD, the only package that is installed is FTP_Client_Proxy, this firewall normally has CARP status MASTER on all interfaces, now it's in carp persistent mantenance mode.
firewall 2: same hardware, but reinstalled with pfSense-CE-2.4.2-RELEASE-amd64 ZFS auto 4GB swap, during the installation I recoverd the previous config.
Just after installation I upgraded it to 2.4.2 p1
I have 2 LAGGs (whith LACP): igb0,igb1 and igb2,igb3.
One LAGG is assigned to an interface, the other has some VLANs.
One Broadcom is directly connected to the other firewall (sync).

On all interfaces except for the sync one we have one or more CARP IPs.
Firewall 2 with 2.4.2 p1 version works for less than an hour (about 30 minutes), than the other firewall becames master and this firewall gets stuck:
on console (DRAC) does not respond anymore but, at least in 1 case, something seems to be working; all 4 nics in LACP have link up, but only one PortChannel is up with an interface only and browser shows certificate warning but then it does't load the login page.

The system.log file contains rows referring to the stuck status time,
a part from "pfr_update_stats: assertion failed." pre-existing and recurring errors, also new errors showed up like
"sonewconn: pcb 0xfffff8007394e1d0: Listen queue overflow: 2 already in queue awaiting acceptance (12 occurrences)"

Any suggestion would be appreciated.

gtoso

@gtoso:

Hi,
I have a similar problem but not with BRIDGE but LAGG (with LACP enabled).
the problem started after the upgrade of one of two firewall in CARP from 2.3.4p1 to 2.4.2p1.

Could it be related to this bug?
ASAP I will try to better describe my problem.

Sorry, I forgot a BRIDGE between an OpenVPN TAP and an interface.
Now I'm trying after removing the bridge.

Thanks.

EDIT: I confirm more than 2 hours whitout problems.
So even a bridge little used not assigned as interface, that include an interface with an IP CARP triggers the problem.