It appears there's a known issue with Broadcom BCM57810 adapters in FreeBSD (LACP bonding is not working well): https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213606
Today I tried to make some tests thru the HAProxy running on the firewall and the server has just screwed up after reaching ~140000 connections. Log contained:
Aug 9 05:20:17 pfSense kernel: bxe0: ERROR: ECORE: timeout waiting for state 1
Aug 9 05:20:17 pfSense kernel: bxe0: ERROR: Queue(3) SETUP failed (rc = -4)
Aug 9 05:20:17 pfSense kernel: bxe0: ERROR: Queue(3) setup failed rc = -4
Aug 9 05:20:18 pfSense rc.gateway_alarm[19058]: >>> Gateway alarm: WANGW (Addr:a.b.c.d Alarm:1 RTT:2000271ms RTTsd:3249226ms Loss:21%)
...
Aug 9 05:20:28 pfSense kernel: bxe1: ERROR: TX watchdog timeout on fp[01], resetting!
Aug 9 05:20:34 pfSense kernel: bxe1: ERROR: ECORE: timeout waiting for state 7
Aug 9 05:21:02 pfSense kernel: bxe0: ERROR: FW failed to respond!
Aug 9 05:21:02 pfSense kernel: bxe0: ERROR: Initialization failed, stack notified driver is NOT running!
Aug 9 05:21:17 pfSense rc.gateway_alarm[45717]: >>> Gateway alarm: WANGW (Addr:a.b.c.d Alarm:1 RTT:0ms RTTsd:0ms Loss:100%)
...
Aug 9 05:21:31 pfSense kernel: bxe2: Interface stopped DISTRIBUTING, possible flapping
Aug 9 05:21:42 pfSense sshd[82110]: Timeout, client not responding.
Aug 9 05:21:54 pfSense sshd[19888]: Timeout, client not responding.
Aug 9 05:21:55 pfSense kernel: bxe0: Interface stopped DISTRIBUTING, possible flapping
Aug 9 05:22:43 pfSense kernel: bxe1: ERROR: ECORE: timeout waiting for state 1
Aug 9 05:22:43 pfSense kernel: bxe1: ERROR: Queue(0) SETUP failed (rc = -4)
Aug 9 05:22:43 pfSense kernel: bxe1: ERROR: Setup leading failed! rc = -4
Aug 9 05:23:14 pfSense kernel: bxe1: ERROR: Initialization failed, stack notified driver is NOT running!
Aug 9 05:23:36 pfSense kernel: bxe3: Interface stopped DISTRIBUTING, possible flapping
Aug 9 05:24:23 pfSense kernel: bxe1: Interface stopped DISTRIBUTING, possible flapping
Going to change the adapters to Intel.