Netgate 6100 — LACP interface loses ability to originate new traffic after upgrade to pfSense Plus 25.11.1

[CyberCloud_Consulting]

Hardware

• Model: Netgate 6100
• Serial: 2027233525
• Netgate Device ID (NDI): eeaa02e1813578cedecd

Summary

After upgrading from pfSense Plus 24.11-RELEASE to 25.11.1-RELEASE, the LACP lagg0 interface (member NICs ix0+ix1, plus its VLAN children — networks named UNIFI/GUEST/HOME/SERVER/KIDS/DOCKER/IOT/PROXY/VOIP/VIDEO) can no longer originate any new outbound traffic, while pre-existing traffic on the same interface continues to flow normally.

Symptom detail

• arp -a shows the interface's own IP as the only entry (permanent), with no dynamically-learned neighbours — even though those neighbours are actively sending ARP requests that arrive at the firewall (confirmed via tcpdump).

• ping <any host on lagg0 or its VLANs> fails immediately (no ARP timeout delay) with:

  ping: sendto: Invalid argument
  

• Root syscall confirmed via truss:

  sendto(3,"\b\0...",64,0,NULL,0) ERR#22 'Invalid argument'
  

  This is a raw ICMP socket, already connect()-ed to the destination, failing at the kernel sendto() call itself — not an ARP resolution timeout, not a routing table miss.

• Interfaces ix0/ix1 both report clean state: media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>), status: active, and mac_stats.rx_errs/mac_stats.crc_errs are 0 on both — ruling out a physical/cable/SFP problem.

• dmesg shows ix1 (one LACP member) flapped twice during boot before stabilizing (UP→DOWN→UP→DOWN→UP), but this appears unrelated: forcing LACP renegotiation via ifconfig lagg0 down; ifconfig lagg0 up does not resolve the issue.

• Disabling hardware offloads on both member NICs (-tso -lro -txcsum -rxcsum) does not resolve the issue.

• Interfaces not part of the LACP group (igc3/MGT, ix2/WAN2) are completely unaffected.

Reproducibility

Confirmed on two independent, separate upgrade attempts from a verified-clean 24.11-RELEASE state (config.xml byte-identical to a pre-upgrade backup both times) — this is fully reproducible, not a one-off corruption event.

Confirmed fix (workaround only)

Rolling back to the 24.11-RELEASE ZFS boot environment (bectl activate) immediately and completely restores normal ARP learning and connectivity on the same hardware/config, with zero other changes.

Question for Netgate

Is this a known regression in 25.11.1 (or its successors 26.03/26.03.1/26.07 — none of which mention a lagg/LACP fix in their release notes)? Is there a patch, or a recommended interface/driver workaround short of staying on 24.11?

---

Prepared by CyberCloud · 6 September 2026

[stephenw10]

@CyberCloud_Consulting said in Netgate 6100 — LACP interface loses ability to originate new traffic after upgrade to pfSense Plus 25.11.1:

while pre-existing traffic on the same interface continues to flow normally

What exactly do you mean there? You can open inbound connections to the interfaces?

Does ifconfig -vvm lagg0 show lacp connected correctly like:

[admin@5100.stevew.lan]/root: ifconfig -v lagg0
lagg0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	description: OPT1
	options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
	ether 00:90:0b:76:8e:55
	hwaddr 00:00:00:00:00:00
	inet 192.168.221.1 netmask 0xffffff00 broadcast 192.168.221.255
	inet6 fe80::290:bff:fe76:8e55%lagg0 prefixlen 64 scopeid 0xb
	laggproto lacp lagghash l2,l3,l4
	lagg options:
		flags=14<USE_NUMA,LACP_STRICT>
		flowid_shift: 16
	lagg statistics:
		active ports: 2
		flapping: 2
	lag id: [(8000,00-90-0B-76-8E-55,0163,0000,0000),
		 (8000,00-08-A2-0C-0B-BA,0183,0000,0000)]
	laggport: ix2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
		[(8000,00-90-0B-76-8E-55,0163,8000,0005),
		 (8000,00-08-A2-0C-0B-BA,0183,8000,0001)]
	laggport: ix3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
		[(8000,00-90-0B-76-8E-55,0163,8000,0006),
		 (8000,00-08-A2-0C-0B-BA,0183,8000,0008)]
	groups: lagg
	media: Ethernet autoselect
	status: active
	nd6 options=821<PERFORMNUD,AUTO_LINKLOCAL,STABLEADDR>
	drivername: lagg0

No there's no known issue in 25.11.1 AFAIK. Have you tested 26.07?

[CyberCloud_Consulting]

@stephenw10 Thanks for the quick response, Stephen.

Re: "pre-existing traffic continues to flow" — I mean already-established sessions (e.g. an existing SSH session to the firewall itself over lagg0, or an ongoing TCP connection) kept working with no interruption. What broke was originating new traffic: new ARP requests, new pings, new connections. The kernel sendto() call itself returned EINVAL on a raw socket for any new outbound packet on lagg0/its VLANs, confirmed via truss. So it's not "can I open inbound connections" — it's specifically that the interface stopped being able to originate anything new, while existing flows were untouched.

Re: ifconfig -vvm lagg0 — we've since rolled back to 24.11-RELEASE to restore production connectivity, so I don't have that output from the 25.11.1 state captured. I can reproduce on 25.11.1 again to get it if that's genuinely needed, but given the effort/risk of a repeat outage, would it help more if I instead test 26.07 directly and report back with the full ifconfig -vvm lagg0 + arp -a + truss output from that release? That would answer both this and your other question in one pass.

[stephenw10]

firewall state persisted across an upgrade? I would not have expected that. I would expect any ssh sessions to be terminated.

Yes test in 26.07. There's really no point testing 25.11.1. If there is a bug here it won't be fixed there.

I'm unable to replicate it here though. LACP LAGG has worked fine in every version I've tested. What is it connected to?

[CyberCloud_Consulting]

@stephenw10 Agreed — no point re-testing 25.11.1. We'll do a fresh install
of 26.07 (skipping 25.11.1 entirely, landing from the 24.11 backup) and
capture the full set you asked for:

• ifconfig -vvm lagg0
• arp -a
• truss ping <host-on-lagg>
• dmesg

Re: what it's connected to — the LACP uplink (ix0 + ix1, 10G SR SFP+)
goes to a UniFi USG XG-16 (v7.4.1) on ports 13 + 14 (Aggregating /
LACP, 20G, Port Mode Infrastructure, Native VLAN Server Network, Tagged Allow
All). For reference, on the working 24.11 box ifconfig -vvm lagg0 shows the
LACP negotiation fully up:

• laggproto lacp lagghash l2,l3,l4, active ports: 2, flapping 0
• pfSense system: 90:EC:77:88:FB:AB, priority 0x01B2 (434)
• switch partner: 74:AC:B9:AD:1E:DE, priority 0x001B (27)
• both members ix0/ix1 in ACTIVE,COLLECTING,DISTRIBUTING /
  ACTIVITY,AGGREGATION,SYNC

I'll capture the same on the 26.07 install and diff it against this.

Re: pre-existing flows surviving — you're right that's unexpected. We'll
re-examine that during the 26.07 test and report back with evidence either
way. Will post the full output here once the fresh install is done.

[CyberCloud_Consulting]

pfSense Plus - LACP regression investigation working notes

Status: current as of 19 September 2026 (updated after the 26.07 fresh-install test window - see "26.07 fresh-install findings" below, which supersedes the "Options under consideration" section).

Subject device

• Netgate 6100 firewall (pfsense.unraid.cloud)
• Serial: 2027233525; Netgate Device ID (NDI): XXXXXXXXXXXXXXXXX
• Current OS: pfSense Plus 24.11-RELEASE (rolled back; see below)
• Access: ssh -i %USERPROFILE%\.ssh\id_ed25519_pfsense root@172.16.1.1; console settings 115200/8/N/1

The defect (confirmed)

Upgrading 24.11-RELEASE to 25.11.1-RELEASE breaks lagg0 (LACP, ix0+ix1, 10G SFP+ fibre):

• Cannot originate NEW outbound traffic (ARP requests, pings, new connections); established flows continue.
• arp -a: only the interface's own IP; ping: sendto: Invalid argument returned immediately.
• truss: sendto(3,"\b\0...",64,0,NULL,0) ERR#22 'Invalid argument' - kernel-level send failure on a connected raw socket.
• NIC error counters clean; interfaces outside the LAG unaffected; offload changes and LACP renegotiation do not help.
• Reproduced twice from a verified-clean 24.11 state (config.xml byte-identical to pre-upgrade backup both times).
• Workaround: bectl activate the 24.11 boot environment - service restores immediately. Applied 6 and 7 September; system has run on 24.11 since.

Netgate forum thread: https://forum.netgate.com/topic/201321
Staff response asked for ifconfig -vvm lagg0 output from the broken state and whether 26.07 has been tested.

Switch-side audit (19 September 2026) - NOT the cause

Checked the UniFi Aggregate Switch (US-16-XG, 10.10.30.252) via the UDM Pro controller database:

• Firewall = SFP+ ports 1+2, one LACP aggregate: both up at 10 Gbps, forced speed with autonegotiation off, full-VLAN trunk.
• Optics healthy: 10GBase-SR multimode modules; Rx power about -3.0 dBm (receiver sensitivity about -10 dBm); normal temperatures.
• Second aggregate (ports 13+14) = unRAID tower (10GBASE-T). Port 6 = UDM Pro. Port 16 = gaming PC (normally off). No alarms, no anomalies.
• Conclusion: switch configuration and optics are conventional and were unchanged across the working and broken states; the failure tracks the pfSense version only.

Re-run command (on the UDM Pro): mongo --quiet --port 27117 ace, query the newest document in db.time_machine_ports_snapshot for device_id 68184976d740ef6974a2b30c (about 25 hours of retention).

Recurring pkg/ABI bug after boot-environment switches

After each boot-environment rollback, config.xml keeps the stale <pkg_repo_conf_path> value (that file lives outside the boot environment), so pkg points at the wrong repository/ABI. Fix (verified twice):

1. Set pkg_repo_conf_path to the running release (for example 24_11).
2. Run /usr/local/sbin/pfSense-repo-setup.
3. Run pkg bootstrap -f; verify pkg -vv shows the matching ABI and pkg update runs clean.

Re-check this after any future boot-environment switch or after a 26.07 config restore.

Options under consideration (superseded - see below)

• Option A: Boot the retained 25.11.1 boot environment (default_20260906070339) once more to capture live diagnostics. Superseded - see "26.07 fresh-install findings".
• Option B: Fresh install of 26.07 via the Netgate Installer USB. Completed 19 September 2026 - see below.

26.07 fresh-install findings (19 September 2026) - TAC update

Summary for TAC

A completely fresh install of pfSense Plus 26.07-RELEASE (not an in-place upgrade; all boot environments wiped, config restored only from the USB External Config Loader after install) reproduces the identical lagg0 defect seen on 25.11.1. This rules out an upgrade-path or stale-boot-environment cause. Further testing today shows the defect is independent of the lagg protocol (reproduced under lacp, loadbalance, and failover alike), narrowing the fault to the lagg(4) pseudo-interface / ix driver interaction itself, not the LACP negotiation layer.

Test 1 - fresh install reproduces the defect

• Root cause of the original in-place-upgrade hang ("Updating configuration........." never completing) was isolated to a base-OS/hardware-level defect in the boot-time config-version-migration step, confirmed independent of config content via three controlled tests (full config; config minus pfBlockerNG/Suricata; blank/renamed config.xml). Not the subject of this TAC update, but recorded here for completeness - fresh install was adopted specifically to work around it.

• Fresh install to 26.07-RELEASE completed successfully (Updating configuration.............done.), WAN interface manually corrected to igc0 (installer default was wrong), production config auto-restored via the USB ECL.

• Immediately after boot, with lagg0 on its default lacp protocol:

  # ping -c 3 10.10.30.253
  ping: sendto: Invalid argument
  ping: sendto: Invalid argument
  ping: sendto: Invalid argument
  # ping -c 3 10.10.30.252
  ping: sendto: Invalid argument
  ping: sendto: Invalid argument
  ping: sendto: Invalid argument
  # arp -a
  pfsense.unraid.cloud (10.10.30.1) at 90:ec:77:88:fb:ab on lagg0 permanent [ethernet]
  

  Only pfSense's own address is ever present in the ARP table; new/unresolved hosts on lagg0 and its VLAN sub-interfaces cannot be reached. Existing/established flows (WAN NAT sessions, DNS to self, etc.) continue unaffected - confirmed via pfctl -ss, which showed active ESTABLISHED/SINGLE:MULTIPLE states on lagg0.2 throughout.

• Workaround (as previously found on 25.11.1): arp -s <ip> <mac> immediately restores full bidirectional connectivity to that specific host (0% packet loss). Confirmed for the UDM Pro (10.10.30.253), the Aggregate Switch (10.10.30.252), and two Access Points (10.10.30.242, 10.10.30.244) today.

• Loop hypothesis explicitly ruled out: sendto: EINVAL is a local kernel-side error raised before the packet reaches the wire, so an L2 loop cannot produce it; the identical cabling/switch config works without fault on 24.11 (a loop would not be version-dependent); no STP topology-change/blocking indicators were observed on the switch during any test window.

Test 2 - protocol independence (new evidence, 19 September 2026)

To test whether the defect was specific to LACP, lagg0's protocol was changed live (switch-side aggregate group changed in lock-step each time: Aggregating for lacp, Switching/independent ports for loadbalance/failover):

lagg0 protocol	Switch port mode	Link/negotiation result	New-host ARP resolution
lacp (original)	Aggregating	Both ports ACTIVE,COLLECTING,DISTRIBUTING	Fails (sendto: Invalid argument)
loadbalance	Switching (independent)	Came up, but mismatched pairing (loadbalance requires a static trunk, not independent switch ports) - not a valid comparison point, superseded by the failover test	Not conclusively tested in isolation
failover	Switching (independent)	ix0 MASTER,ACTIVE; ix1 standby (by design)	Fails identically - after protocol-change churn flushed the ARP table (all prior static entries lost), ping -c 3 10.10.30.253 returned sendto: Invalid argument until the static ARP entry was re-seeded, at which point it immediately returned 0% packet loss
lacp (reverted)	Aggregating	Both ports ACTIVE,COLLECTING,DISTRIBUTING	Fails again for any host not re-seeded via static ARP (confirmed by re-adding entries for the switch and both APs, each restoring 0% packet loss immediately)

Note: changing lagg0's protocol reinitializes the interface and flushes its dynamic/static ARP table each time. This is expected pfSense/FreeBSD lagg(4) behaviour, not evidence of anything new; it required re-seeding the static ARP workaround entries after each protocol change.

Conclusion: the defect reproduces under both lacp and failover (the two protocols fully validated end-to-end today). Combined with the pre-existing 25.11.1 evidence, this indicates the fault is not in the LACP negotiation state machine specifically, but in the lagg(4) pseudo-interface's ARP-origination path (or the underlying ix driver it wraps), independent of which aggregation protocol is configured on top of it.

Downstream impact confirmed today

• Static ARP entries only help fixed/known infrastructure IPs (UDM, switch, APs) that can be pre-seeded. They do not and cannot help arbitrary WiFi client devices, since each client is, by definition, a new/unknown host to pfSense on first association - it hits the identical sendto: Invalid argument failure regardless of whether it reaches pfSense via a wired port or bridged through an AP.
• Confirmed directly: after statically ARPing both on-site Access Points (10.10.30.242, 10.10.30.244 - management IPs, now both reachable at 0% packet loss), wireless client devices associating to those same APs still cannot reach the Internet. This isolates the fault to per-client ARP origination, not anything about the AP's own reachability or the switch/VLAN trunk configuration.

Causes ruled out (cumulative, 25.11.1 + 26.07 testing)

• DNS resolution failure - ruled out.
• pkg-repo-slot naming bug (pfSense-repo-0001.name overwritten) - real finding, confirmed not causal.
• USB/ECL config overwrite - real risk, ruled out as sole cause (hang persisted with USB removed; bug present on 26.07 despite USB ECL only supplying the standard production config).
• Config content (packages, size) - ruled out via the three blank/reduced/full config tests during the original hang investigation.
• Switch-side configuration/optics - audited 19 September, healthy and unchanged across working (24.11) and broken (25.11.1/26.07) states.
• L2 loop - ruled out on kernel-error-mechanics grounds (see Test 1) and absence of STP indicators.
• LACP protocol specifically - ruled out today; failover reproduces identically.
• UDM Pro's own routing table - reviewed, contains only standard subnet routes plus three pre-existing host-specific static routes (for 172.16.1.100, 172.16.2.100/101/201) pointing via pfSense; no mismatch with pfSense's own routing table (which has no custom static routes at all, only standard per-VLAN interface routes). Not a contributing factor.

Suggested next TAC ask

Request Netgate engineering specifically test whether lagg(4) on 26.07/25.11.1 can originate an ARP request for a new host on an ix-backed lagg interface in failover mode (single active physical port, no LACP negotiation involved at all) on equivalent Netgate 6100 hardware. Because the defect reproduces even with LACP entirely out of the picture, it should be reproducible in a minimal single-active-link test, which removes any question of switch-side LACP partner behaviour from the equation.

Rollback to 24.11 (19 September 2026) - current state

After the protocol-independence testing above and confirmation that the AP static-ARP workaround does not extend to wireless client devices, the decision was made to roll the firewall back to 24.11-RELEASE (boot environment switch, not a fresh install) rather than continue operating with the defect present.

Verified directly post-rollback (not just inferred from the user's report):

# cat /etc/version
24.11-RELEASE
# ifconfig lagg0
laggproto lacp lagghash l2,l3,l4
laggport: ix0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: ix1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
status: active
# arp -d 10.10.30.242
10.10.30.242 (10.10.30.242) deleted
# ping -c 3 10.10.30.242
3 packets transmitted, 3 packets received, 0.0% packet loss

The static ARP entry for the AP (10.10.30.242) was explicitly deleted before the ping test, so this confirms fresh dynamic ARP origination to a previously-unresolved host succeeds on 24.11 with no workaround in place - the same operation that fails with sendto: Invalid argument on both 25.11.1 and the fresh 26.07 install. This is the strongest available confirmation that the defect is genuinely absent on 24.11 and not merely masked by leftover static ARP entries.

Current state: firewall running 24.11-RELEASE, LACP lagg0 fully bundled, dynamic ARP resolution confirmed working. Any manually-added static ARP entries from today's testing (UDM, switch, both APs) are now redundant on 24.11 and can be left in place harmlessly or removed via arp -d at leisure - they do not need to be cleared for connectivity to work.

[stephenw10]

I still can't replicate this. Testing here on a 8200, same NICs, in 26.07:

[26.07-RELEASE][admin@8200-2.stevew.lan]/root: ifconfig -vm lagg0 
lagg0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	description: LAGG0
	options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
	capabilities=4f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
	ether 00:08:a2:12:e2:ca
	hwaddr 00:00:00:00:00:00
	inet 10.2.45.1 netmask 0xffffff00 broadcast 10.2.45.255
	inet6 fe80::208:a2ff:fe12:e2ca%lagg0 prefixlen 64 scopeid 0xe
	laggproto lacp lagghash l2,l3,l4
	lagg options:
		flags=14<USE_NUMA,LACP_STRICT>
		flowid_shift: 16
	lagg statistics:
		active ports: 2
		flapping: 0
	lag id: [(8000,00-08-A2-12-E2-CA,01C6,0000,0000),
		 (8000,00-07-43-48-1C-80,0112,0000,0000)]
	laggport: ix0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
		[(8000,00-08-A2-12-E2-CA,01C6,8000,0005),
		 (8000,00-07-43-48-1C-80,0112,8000,0002)]
	laggport: ix1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
		[(8000,00-08-A2-12-E2-CA,01C6,8000,0006),
		 (8000,00-07-43-48-1C-80,0112,8000,0001)]
	groups: lagg
	media: Ethernet autoselect
	status: active
	supported media:
		media autoselect
	nd6 options=821<PERFORMNUD,AUTO_LINKLOCAL,STABLEADDR>
	drivername: lagg0

[26.07-RELEASE][admin@8200-2.stevew.lan]/root: arp -a | grep lagg0
? (10.2.45.1) at 00:08:a2:12:e2:ca on lagg0 permanent [ethernet]
? (10.2.45.11) at 00:07:43:48:1c:80 on lagg0 expires in 894 seconds [ethernet]
[26.07-RELEASE][admin@8200-2.stevew.lan]/root: arp -d 10.2.45.11
10.2.45.11 (10.2.45.11) deleted
[26.07-RELEASE][admin@8200-2.stevew.lan]/root: arp -a | grep lagg0
? (10.2.45.1) at 00:08:a2:12:e2:ca on lagg0 permanent [ethernet]
? (10.2.45.11) at 00:07:43:48:1c:80 on lagg0 expires in 1194 seconds [ethernet]
[26.07-RELEASE][admin@8200-2.stevew.lan]/root: ping -c 2 10.2.45.11
PING 10.2.45.11 (10.2.45.11): 56 data bytes
64 bytes from 10.2.45.11: icmp_seq=0 ttl=64 time=0.190 ms
64 bytes from 10.2.45.11: icmp_seq=1 ttl=64 time=0.131 ms

--- 10.2.45.11 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.131/0.161/0.190/0.029 ms

Do you have any custom sysctls set? Edit: Nope you still see it on a fresh install.

Have you tried connecting the lagg to something other than the UniFi switch?

[CyberCloud_Consulting]

@stephenw10 Thanks Stephen — understood it's not reproducing on the 8200. Updates from our side plus some targeted asks, since we still reproduce this reliably on the 6100.

Answers to your questions

Custom sysctls: none. The 26.07 test was a clean installer install with defaults; it fails on first boot before any tuning. (We'll capture a sysctl diff vs 24.11 next window regardless.)
Connected to: UniFi US-16-XG aggregation switch, SFP+ ports 1+2 (minor correction to my earlier post — 13+14 is our unRAID box), 10GBase-SR optics, full VLAN trunk.
Since the last post we've also confirmed it is protocol-independent: it reproduces identically under laggproto failover (single active port, no LACP negotiation at all). arp -s <ip> <mac> still restores connectivity to any host immediately.
One environment difference worth testing on your side: our lagg0 carries 10 VLAN sub-interfaces; your 8200 output looks like a flat lagg0 with no VLAN children. If feasible, could you re-test with several VLANs on the lagg? We'll run the reverse bisect here (minimal lagg, no VLANs, then add them back).

Related report on identical hardware/OS base: topic 200214 documents another 6100 under 25.11.1/FreeBSD 16 with a kernel-level sendto failure where the packet never reaches pf, traced there to the pf/pfil error path returning the error to userspace (pf_map_addr_sn has failed → NAT proxy port allocation failed in dmesg; worked around with an outbound NAT exclusion), and still present on 26.07. Our EINVAL-on-sendto for any newly-originated traffic on lagg0 has the same shape. Can engineering check whether a common FreeBSD 16 pfil/pf error-propagation path is involved?

Next window from us (with full captures): (1) minimal config on 26.07 — one lagg, no VLANs, then add VLANs back; (2) pfctl -d on the broken state — does the EINVAL clear; (3) dmesg scan for pf/lagg/iflib lines at failure time (we didn't capture this on 25.11.1); (4) ifconfig -vvm lagg0/ix0/ix1 + arp -a + truss ping from the broken state; (5) alternate peer if we can source a suitable module.

[stephenw10]

I'll test it on the 6100 tomorrow but I'll be amazed if it makes any difference since it's the same platform. The difference is only the CPU cores.

I'll also test with VLANs over the lagg but again that's a very common setup.

It looked like you were testing with arp on the lagg directly but it's hard to tell when all your test output has been put through some LLM.

Direct output of the arp and ifconfig commands would be very helpful to compare here.

I could believe, for example, that VLAN hardware offloading is causing a problem for traffic in the VLANs but would not for anything on the LAGG directly.
I wouldn't expect that in a default install though since the off-loading options would be the same.

[CyberCloud_Consulting]

@stephenw10 > Both, but direct was the one we hit first. At the time of the original report lagg0 itself had an IP (10.10.30.1, straight on the lagg rather than a VLAN child), and new pings from it failed exactly as reported. Raw output from the 26.07 install, before any static ARP entries were added:

# ping -c 3 10.10.30.253
ping: sendto: Invalid argument
ping: sendto: Invalid argument
ping: sendto: Invalid argument
# ping -c 3 10.10.30.252
ping: sendto: Invalid argument
ping: sendto: Invalid argument
ping: sendto: Invalid argument
# arp -a
pfsense.unraid.cloud (10.10.30.1) at 90:ec:77:88:fb:ab on lagg0 permanent [ethernet]

The VLAN networks behaved the same way - every new destination gave sendto: Invalid argument until we seeded a static entry, then it worked straight away.

On the offload theory: we tried disabling the member NIC offloads early on (-tso -lro -txcsum -rxcsum) and it made no difference. The VLAN hw offload flags we haven't touched yet - I'll add that to the next window and grab the flag state while I'm there. One thing that makes me doubt it's offload: the direct-lagg path above fails too, so whatever this is, it isn't only happening in the VLAN path.

Raw output from our box as it stands today (24.11, working). Heads up the layout has changed since the original report - 10.10.30.0/24 now routes via the UDM over a transit VLAN and lagg0 itself no longer has an IP. Same 10 VLAN children on it as before.

# ifconfig -vvm lagg0
lagg0: flags=1008b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: TRUNK
        options=48138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,HWSTATS,MEXTPG>
        capabilities=4f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        ether 90:ec:77:88:fb:ab
        hwaddr 00:00:00:00:00:00
        inet6 fe80::92ec:77ff:fe88:fbab%lagg0 prefixlen 64 scopeid 0xd
        laggproto lacp lagghash l2,l3,l4
        lagg options:
                flags=14<USE_NUMA,LACP_STRICT>
                flowid_shift: 16
        lagg statistics:
                active ports: 2
                flapping: 0
        lag id: [(8000,90-EC-77-88-FB-AB,01B2,0000,0000),
                 (8000,74-AC-B9-AD-1E-DE,001B,0000,0000)]
        laggport: ix0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
                [(8000,90-EC-77-88-FB-AB,01B2,8000,0005),
                 (8000,74-AC-B9-AD-1E-DE,001B,0080,0001)]
        laggport: ix1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
                [(8000,90-EC-77-88-FB-AB,01B2,8000,0006),
                 (8000,74-AC-B9-AD-1E-DE,001B,0080,0002)]
        groups: lagg
        media: Ethernet autoselect
        status: active
        supported media:
                media autoselect
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        drivername: lagg0

# ifconfig ix0
ix0: flags=1008b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=48138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,HWSTATS,MEXTPG>
        ether 90:ec:77:88:fb:ab
        inet6 fe80::92ec:77ff:fe88:fbad%ix0 prefixlen 64 scopeid 0x5
        media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
# ifconfig ix1
ix1: flags=1008b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=48138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,HWSTATS,MEXTPG>
        ether 90:ec:77:88:fb:ab
        hwaddr 90:ec:77:88:fb:ac
        inet6 fe80::92ec:77ff:fe88:fbad%ix1 prefixlen 64 scopeid 0x6
        media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

# arp -a (trimmed to the lagg0 entries - full table is long)
? (10.10.23.1) at 90:ec:77:88:fb:ab on lagg0.23 permanent [vlan]
? (10.10.20.1) at 90:ec:77:88:fb:ab on lagg0.20 permanent [vlan]
? (10.10.21.16) at 5a:98:f0:2a:5b:70 on lagg0.21 expires in 190 seconds [vlan]
? (172.16.3.1) at 90:ec:77:88:fb:ab on lagg0.4 permanent [vlan]
? (172.16.2.74) at 80:0a:80:5d:4e:03 on lagg0.3 expires in 1102 seconds [vlan]
? (172.16.2.32) at (incomplete) on lagg0.3 expired [vlan]
LivingRoom.unraid.cloud (172.16.1.82) at 90:56:82:00:d2:d7 on lagg0.2 expires in 1184 seconds [vlan]
pfsense.unraid.cloud (172.16.1.1) at 90:ec:77:88:fb:ab on lagg0.2 permanent [vlan]
? (10.10.99.2) at 78:45:58:e6:da:fb on lagg0.99 expires in 222 seconds [vlan]
? (10.10.99.1) at 90:ec:77:88:fb:ab on lagg0.99 permanent [vlan]

# sysctl net.link.lagg
net.link.lagg.default_flowid_shift: 16
net.link.lagg.default_use_numa: 1
net.link.lagg.default_use_flowid: 0
net.link.lagg.failover_rx_all: 0
net.link.lagg.lacp.default_strict_mode: 1
net.link.lagg.lacp.debug: 0

Plan for our next window:

• minimal config on 26.07: single lagg, no VLANs, then add VLANs back one at a time
• pfctl -d on the broken state to see if the EINVAL clears
• dmesg at failure time (didn't capture this on 25.11.1)
• full set from the broken state: ifconfig -vvm lagg0/ix0/ix1 + the VLAN children, arp -a, truss ping
• VLAN hw offload toggled, to see if it changes anything
• a different peer if we can find a free port and optic

Good luck with the 6100 test - keen to hear how it goes either way.

[stephenw10]

That's great data thanks. OK I'll setup the 6100 tomorrow and see what happens.

[stephenw10]

Same result here on a 6100 and with VLANs:

[26.07-RELEASE][admin@6100-3.stevew.lan]/root: ifconfig -v lagg0
lagg0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	description: LAGG0
	options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
	ether 90:ec:77:1c:80:ef
	hwaddr 00:00:00:00:00:00
	inet 10.2.46.1 netmask 0xffffff00 broadcast 10.2.46.255
	inet6 fe80::92ec:77ff:fe1c:80ef%lagg0 prefixlen 64 scopeid 0xd
	laggproto lacp lagghash l2,l3,l4
	lagg options:
		flags=14<USE_NUMA,LACP_STRICT>
		flowid_shift: 16
	lagg statistics:
		active ports: 2
		flapping: 0
	lag id: [(8000,90-EC-77-1C-80-EF,01A6,0000,0000),
		 (8000,00-07-43-48-1C-80,0112,0000,0000)]
	laggport: ix0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
		[(8000,90-EC-77-1C-80-EF,01A6,8000,0005),
		 (8000,00-07-43-48-1C-80,0112,8000,0001)]
	laggport: ix1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
		[(8000,90-EC-77-1C-80-EF,01A6,8000,0006),
		 (8000,00-07-43-48-1C-80,0112,8000,0002)]
	groups: lagg
	media: Ethernet autoselect
	status: active
	nd6 options=821<PERFORMNUD,AUTO_LINKLOCAL,STABLEADDR>
	drivername: lagg0

Testing the arp on each:

[26.07-RELEASE][admin@6100-3.stevew.lan]/root: arp -a | grep lagg0
? (10.2.46.1) at 90:ec:77:1c:80:ef on lagg0 permanent [ethernet]
? (10.2.46.10) at 00:07:43:48:1c:80 on lagg0 expires in 211 seconds [ethernet]
? (10.3.100.1) at 90:ec:77:1c:80:ef on lagg0.100 permanent [vlan]
? (10.3.100.2) at 00:07:43:48:1c:80 on lagg0.100 expires in 564 seconds [vlan]
? (10.3.101.1) at 90:ec:77:1c:80:ef on lagg0.101 permanent [vlan]
? (10.3.101.2) at 00:07:43:48:1c:80 on lagg0.101 expires in 575 seconds [vlan]
? (10.3.102.2) at 00:07:43:48:1c:80 on lagg0.102 expires in 918 seconds [vlan]
? (10.3.102.1) at 90:ec:77:1c:80:ef on lagg0.102 permanent [vlan]
[26.07-RELEASE][admin@6100-3.stevew.lan]/root: arp -da
10.3.102.2 (10.3.102.2) deleted
10.3.101.2 (10.3.101.2) deleted
10.3.100.2 (10.3.100.2) deleted
10.2.46.10 (10.2.46.10) deleted
172.21.16.8 (172.21.16.8) deleted
172.21.16.238 (172.21.16.238) deleted
172.21.16.1 (172.21.16.1) deleted
172.21.16.4 (172.21.16.4) deleted
[26.07-RELEASE][admin@6100-3.stevew.lan]/root: arp -a | grep lagg0
? (10.2.46.1) at 90:ec:77:1c:80:ef on lagg0 permanent [ethernet]
? (10.2.46.10) at 00:07:43:48:1c:80 on lagg0 expires in 1185 seconds [ethernet]
? (10.3.100.1) at 90:ec:77:1c:80:ef on lagg0.100 permanent [vlan]
? (10.3.101.1) at 90:ec:77:1c:80:ef on lagg0.101 permanent [vlan]
? (10.3.102.1) at 90:ec:77:1c:80:ef on lagg0.102 permanent [vlan]
[26.07-RELEASE][admin@6100-3.stevew.lan]/root: ping -c 1 10.3.100.2 ; ping -c 1 10.3.101.2 ; ping -c 1 10.3.102.2
PING 10.3.100.2 (10.3.100.2): 56 data bytes
64 bytes from 10.3.100.2: icmp_seq=0 ttl=64 time=0.400 ms

--- 10.3.100.2 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.400/0.400/0.400/0.000 ms
PING 10.3.101.2 (10.3.101.2): 56 data bytes
64 bytes from 10.3.101.2: icmp_seq=0 ttl=64 time=0.332 ms

--- 10.3.101.2 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.332/0.332/0.332/0.000 ms
PING 10.3.102.2 (10.3.102.2): 56 data bytes
64 bytes from 10.3.102.2: icmp_seq=0 ttl=64 time=0.324 ms

--- 10.3.102.2 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.324/0.324/0.324/0.000 ms
[26.07-RELEASE][admin@6100-3.stevew.lan]/root: arp -a | grep lagg0
? (10.2.46.1) at 90:ec:77:1c:80:ef on lagg0 permanent [ethernet]
? (10.2.46.10) at 00:07:43:48:1c:80 on lagg0 expires in 1175 seconds [ethernet]
? (10.3.100.1) at 90:ec:77:1c:80:ef on lagg0.100 permanent [vlan]
? (10.3.100.2) at 00:07:43:48:1c:80 on lagg0.100 expires in 1198 seconds [vlan]
? (10.3.101.1) at 90:ec:77:1c:80:ef on lagg0.101 permanent [vlan]
? (10.3.101.2) at 00:07:43:48:1c:80 on lagg0.101 expires in 1198 seconds [vlan]
? (10.3.102.2) at 00:07:43:48:1c:80 on lagg0.102 expires in 1198 seconds [vlan]
? (10.3.102.1) at 90:ec:77:1c:80:ef on lagg0.102 permanent [vlan]

The fact it works for you if you add static ARP entries seems to rule out VLAN hardware off-loading. Something there is affecting only arp traffic. Hmm..

[stephenw10]

The error you see when trying to ping is interesting. Do you have the full truss output from the failing ping?

You might also try installing the arping pkg and testing with that too. See what error, if any, that gives.

[stephenw10]

Did you ever try 25.07(.1)? That was built on FreeBSD 15 like 24.11 was. But from 25.11 it was built on FreeBSD 16 which seems like a clue.

[CyberCloud_Consulting]

@stephenw10 > Thanks for setting up the 6100 and running it with VLANs over the lagg, and for the raw output. Good to have the lab box result on file - the flush and relearn cycle completes there, while on ours a new destination doesn't come back without a static entry, so the delta looks like it's in our unit or its exact environment rather than the platform itself. And agreed on the static ARP reading - it's new ARP origination that fails; static entries skip that resolution path, which is why everything downstream works once they're in.

Re the truss: the single line in the original report is all we captured on the failing state - we didn't save the full trace, so it needs the broken boot to reproduce. It's on the list for the next run: full output of truss ping -c 1 against an on-link neighbour from the broken build, plus the same trace from the working 24.11 build for contrast. Arping we'll add too - install the pkg on the broken boot and post what it gives; nothing to reproduce on 24.11, so both sides get the same commands in the one window.

Re 25.07(.1): no - we skipped it, straight from 24.11 to 25.11.1. Good lead though, and it fits your FreeBSD 15 vs 16 note: we'll stand up 25.07.1 on the unit in the same run and repeat the test. If it's clean, the regression sits in the 15 to 16 jump; if it fails, it goes back further than that. Not somewhere we can stay long-term, but as a bisect it's exactly right.

We need to move off 24.11 regardless, so the window is covering two things at once: evidence for this thread, and finding something we can run with in the meantime. Plan, all pasted raw:

• full truss of a failing ping + arping test (and the working-24.11 counterparts)
• 25.07.1 stood up on the unit - the same ping/ARP test again
• the sequence you ran - arp -da, then a ping to each VLAN neighbour - on ours, so it sits next to your output
• pfctl -d on the broken state - if the EINVAL clears, that would point at the pf/pfil error-propagation shape from topic 200214, and possibly a workaround we can run with
• a plain ix port on 26.07 with no lagg at all - if a former lagg member works on its own, that's both an isolation point and an interim path
• minimal config: single lagg, no VLANs, then add VLANs back one at a time
• full set from the broken state: ifconfig -vvm for lagg0/ix0/ix1, arp -a, dmesg at the point of failure, sysctl diff against 24.11, and the VLAN offload flags toggled while we're in there
• a different peer/optic for the LAG if we can source the kit

One thing that would help if it's quick: what is the lab 6100's LAG connected to - switch model and ports, optics or DAC? Whether it's close to ours (UniFi US-16-XG, 10GBase-SR) or quite different, both are useful.

If it helps, we can also get a sanitised config.xml to you for the lab box - if the 6100 fails with our config restored, the trigger is config-side and we can bisect it together from there. And if a support case is the better route to get this in front of engineering, happy to raise one with the full package; otherwise we'll keep posting the raw output here.

[stephenw10]

In my test setup that 6100 is connected to another pfSense instance directly. Its using Chelsio NICs and DAC cables.

[26.03-RELEASE][root@mark.stevew.lan]/root: ifconfig -v lagg0
lagg0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: LAN
        options=6ea00bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,NV,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,HWRXTSTMP,MEXTPG>
        ether 00:07:43:48:1c:80
        hwaddr 00:00:00:00:00:00
        inet 10.2.46.10 netmask 0xffffff00 broadcast 10.2.46.255
        inet6 fe80::207:43ff:fe48:1c80%lagg0 prefixlen 64 scopeid 0x8
        laggproto lacp lagghash l2,l3,l4
        lagg options:
                flags=14<USE_NUMA,LACP_STRICT>
                flowid_shift: 16
        lagg statistics:
                active ports: 2
                flapping: 0
        lag id: [(8000,00-07-43-48-1C-80,0112,0000,0000),
                 (8000,90-EC-77-1C-80-EF,01A6,0000,0000)]
        laggport: cxl0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
                [(8000,00-07-43-48-1C-80,0112,8000,0001),
                 (8000,90-EC-77-1C-80-EF,01A6,8000,0005)]
        laggport: cxl1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
                [(8000,00-07-43-48-1C-80,0112,8000,0002),
                 (8000,90-EC-77-1C-80-EF,01A6,8000,0006)]
        groups: lagg
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        drivername: lagg0
[26.03-RELEASE][root@mark.stevew.lan]/root: ifconfig -v cxl0
cxl0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=6ea00bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,NV,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,HWRXTSTMP,MEXTPG>
        ether 00:07:43:48:1c:80
        inet6 fe80::6ef0:49ff:fecb:407a%cxl0 prefixlen 64 scopeid 0x1
        media: Ethernet 10Gbase-Twinax <full-duplex,rxpause,txpause>
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        drivername: cxl0
        plugged: SFP/SFP+/SFP28 1X Copper Active (Copper pigtail)
        vendor: BROCADE PN: 58-1000026-01 SN: CAX116410001093 DATE: 2016-10-07
[26.03-RELEASE][root@mark.stevew.lan]/root: ifconfig -v cxl1
cxl1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=6ea00bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,NV,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,HWRXTSTMP,MEXTPG>
        ether 00:07:43:48:1c:80
        hwaddr 00:07:43:48:1c:88
        inet6 fe80::6ef0:49ff:fecb:407a%cxl1 prefixlen 64 scopeid 0x2
        media: Ethernet 10Gbase-Twinax <full-duplex,rxpause,txpause>
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        drivername: cxl1
        plugged: SFP/SFP+/SFP28 1X Copper Active (Copper pigtail)
        vendor: BROCADE PN: 58-1000026-01 SN: CAX112240004092 DATE: 2012-06-16

[CyberCloud_Consulting]

@stephenw10 Thanks for the patience on this one. We have it understood now, and the outcome is a slightly embarrassing one from our side: this is not a lagg or FreeBSD problem at all, it is the DHCP server's Static ARP option in our own configuration. Unless something below looks off to you, we will call the case closed from our side.

Short version: several of our interfaces have Static ARP enabled (the trunk lagg itself, plus HOME, IOT, SERVER and VIDEO). The genuinely curious part is that the option was not actually in force on 24.11 on our box, and the newer builds apply it faithfully - so what we experienced as "the lagg can't originate new traffic" was really those ARP restrictions switching on for the first time.

Side by side, same config in both:

24.11 (23 Sep capture):
  lagg0 and its VLANs - no STATICARP flag; only the box's own addresses are permanent;
  neighbours learned dynamically, new destinations resolve fine.

26.07 (26 Sep, both boots):
  lagg0, lagg0.2, lagg0.3, lagg0.21 - STATICARP; 35 permanent entries (the static mappings);
  a destination with no entry fails "sendto: Invalid argument" (connect() succeeds, sendto() ERR#22);
  ifconfig lagg0.21 -staticarp - the next send resolves and creates an (incomplete) entry;
  flag back on - the error returns for entry-less destinations.

That is the documented staticarp behaviour - "will only reply to requests for its addresses, and will never send any requests" - and it accounts for everything that looked strange in our reports: static entries making things work (there is nothing to originate), LACP and failover failing identically (it is per-interface, not the lagg), established flows untouched, and the immediate kernel EINVAL on a new destination. Your read that something was "affecting only arp traffic" was exactly right - we just never suspected our own option, because on our build it had never actually done anything. A stock lab config will not show any of it either, which is why neither lab box could replicate our reports.

For completeness: we could not reproduce the original full-failure state (empty ARP table, nothing outbound) on the fresh install from the 26th - the operation that used to fail ran clean there. What we can reproduce every time is the per-interface behaviour above, and it accounts for every symptom we originally reported. So we are confident this is fully explained on our side, with no platform defect.

One footnote, in case it is useful to Netgate engineering: on our unit, 24.11 shows no trace of the option being applied at all - no flags, no mapping entries, dynamic ARP normal - while current builds apply it as documented. We are not chasing it further, but it seemed worth a mention.

Thanks again for the two lab setups and the thread time, and sorry for the long trail of evidence when the answer was sitting in our own DHCP settings all along. We are staying on 26.07 here, so this closes it out from our side - happy to post the raw window captures if you ever want them for the record.

[stephenw10]

Aha, yup that would do it. That is interesting though. As you say the interfaces are not flagged as staticarp in your output from 24.11. So it looks like there was a bug there preventing it being set correctly that was fixed in 25.11 (and possibly 25.07).
Are you using Kea for DHCP perhaps? If so that option was added there for 25.07 so it may have been ignored until then:
https://redmine.pfsense.org/issues/15654

I usually recommend not using staticarp if possible because it can make debugging extremely confusing!

Anyway glad you were able to resolve it cleanly.

[CyberCloud_Consulting]

@stephenw10 Hi Stephen,

Thanks - no, we are not using Kea, however we are moving to it soon though. Our active backend is ISC DHCP;
Kea is installed on the box but has never been selected. So the change referenced in #15654 is not what flipped this for us - it happened on the ISC path, which still fits your read that the option was not being applied before.

It lines up with our two captures. On 24.11 the same staticarp options were present in the configuration, along with 16 mapping-level static ARP entry flags, but the ARP table held zero permanent entries and no interface carried the STATICARP flag. On 26.07 we see 35 permanent entries and the flag applied. So the option was configured all along, and the newer builds are the first ones honouring it - which is the documented staticarp behaviour of never sending requests, and explains the immediate sendto failure, the static entries
fixing it instantly, and established flows being untouched.

Agreed on your recommendation - we do not need staticarp here and are taking it out.

One note for the record: on our unit 24.11 applied the option not at all, while current builds apply it faithfully. If that was a fixed bug, it is a sharp behaviour change to land quietly, given it turned a dormant config option into an outage on upgrade.

Thanks again for the lab setups and the time on this thread.