pfSense 2.8.1 / Intel I226-V LAN Interface Stops Passing Traffic Until Reboot (Protectli V1410)

[ahole4sure]

I'm hoping someone can help me narrow down an intermittent issue before I RMA the appliance.

Hardware
Protectli V1410
Intel N5105
4 × Intel I226-V NICs
Kingston NV3 500GB NVMe
BIOS: coreboot/Dasharo v0.9.3 (Sept 2024)
pfSense CE 2.8.1 (previously 2.7.2)
Network
WAN: Frontier Fiber (igc0)
LAN: igc2
Multiple WireGuard tunnels
HAProxy exposed on WAN for remote access
UniFi switches
Eero APs in bridge mode
The Problem

Twice now I've had the LAN essentially "die."

Symptoms:

LAN clients lose connectivity.
WireGuard tunnels become unreachable.
Anything requiring LAN access stops working.
I can still reach the pfSense GUI remotely through an HAProxy service exposed on WAN.
The firewall itself does not appear to crash.
CPU and memory look normal.
WAN remains online.

A reboot restores everything immediately.

The first reboot did not restore LAN, but the second reboot did.

Interface Status During Normal Operation

LAN interface shows:

UP
RUNNING
ACTIVE
2500Base-T Full Duplex

No obvious errors.

What I've Checked

The following all report zero:

watchdog_timeouts
rx_overruns
CRC errors
receive errors
missed packets

The I226-V EEPROM reports:

EEPROM V2.13
eTrack 0x80000286

All four NICs are detected correctly.

dmesg

Earlier I observed:

igc2: link state changed to DOWN
igc2: link state changed to UP

along with repeated link events on igc1.

I have not seen watchdog timeout messages.

ARP Messages

I did notice these in dmesg:

arp: 192.168.1.20 moved from xx:xx to yy:yy
arp: 192.168.1.20 moved back

and similar messages for one other LAN IP.

Not sure whether those are significant or simply a bridged device/Eero behavior.

What I've Already Done
Updated from pfSense 2.7.2 to 2.8.1.
Verified BIOS is current.
Verified NIC firmware.
Verified all interfaces negotiate at 2500Base-T Full Duplex.
Verified no CRC/RX/watchdog errors.
Rebooted (issue cleared).
My Question

Does this sound like:

an igc/I226 driver issue?
a hardware issue with the Protectli?
a switch/L2 problem?
something else I should investigate?

Are there additional logs or commands I should collect before the next occurrence so the root cause can be identified?

Thanks in advance.

[patient0]

@ahole4sure did you have the same issue while you were running pfSense 2.7.2?

• is a UniFi switch connected to igc2, what model? And if yes, what does the switch think of that connection at that time?
• you write repeated igc1 events, what is on igc1? You only mention igc0 (WAN) and igc2 (LAN)
• if LAN stops working, can you connect to the console (HDMI or serial, whatever you set up) and see if you can ping anything on the LAN and/or WAN? And what is the status of the interfaces (ifconfig -mv igc2, ifconfig -mv igc0)
• where are the WireGuard tunnels going that you loose them when you loose LAN?

[stephenw10]

Do you still see link/activity LEDs on the LAN NIC in that state?

I would connect to the console, try to ping out to something on the LAN and see what error is shown.

[ahole4sure]

@stephenw10 I have not connected to console

But from the UI command prompt or ping - pinging from LAN to anywhere FAILS
The only reason I have access to the UI is becasue I get "external access" back to the device via the WAN and haproxy, but local connections fail

Of course I initially thought the LAN NIC was failing , switch form igc2 to igc3 and the 12 days later it happened again

[stephenw10]

@ahole4sure said in pfSense 2.8.1 / Intel I226-V LAN Interface Stops Passing Traffic Until Reboot (Protectli V1410):

pinging from LAN to anywhere FAILS

Do you know how it fails though, what error is shown?

[ahole4sure]

@stephenw10
TBH I don’t remember the exact failure but I think I recall that it was just the typical error
Transmitted but nothing received, 100% packet loss

I do have a screenshot that shows ALL of the LAN dhcp leases down (this was when it happened the second time

Just not sure what to do? The device was flawless for a year. No real pfsense changes
I do recall that first time it failed was in fact right after a lightning storm
Second time was just random

Because of this I actually assumed maybe hardware NIC failure but when it happened a second time with LAN moved to a different igc

Attached is what the dhcp table looked like. Everything suddenly down
But weirdly was ONLY able to access remotely through the WAN and haproxy (even plugging directly from PC into LAN port did not allow access to pfsense)!

[stephenw10]

Mmm, everything showing down like that means they had all expired from the ARP table.

When you connected to the LAN directly did the LAN itself show as UP? Did you see link LEDs on the LAN port?

[ahole4sure]

@stephenw10
When connected directly it did NOT show as up

Don’t recall the lights though

[stephenw10]

Hmm, if it showed as down it can never pass traffic. Was there any sort of error logged when you connected to it?

[Gertjan]

@ahole4sure said in pfSense 2.8.1 / Intel I226-V LAN Interface Stops Passing Traffic Until Reboot (Protectli V1410):

Of course I initially thought the LAN NIC was failing , switch form igc2 to igc3 and the 12 days later it happened again

A LAN (pfSense ?) NIC failing, that possible.
But you moved the pfSense LAN from igc2 to some other pfSense NIC, like igc3.
And the issue repeats ? ( ! )
Don't look any further, you found the issue.
It's not the pfSense device, but everything, starting from the cable leaving pfSense igc2 ( or igc3) up to (other cables, switches whatever) and including the device you use to connect to LAN.

[slu]

We had some years ago a faulty nic in our pfSense, the interface stops random on load.
The management interface show some pci-e errors followed by a pci-e reset for this card.

We replaced this nic and the problem was solved...

[ahole4sure]

@stephenw10
@slu

This is a repsonse form Protectli (screenshot)

You think it is worth changing the bios to AMI ?

[Gertjan]

@ahole4sure

A couple of things noteworthy :

Netgate isn't just a FreeBSD 'consumer', they are a FreeBSD - the OS - contributor/developer.
I would like to advise you to install native "FreeBSD" but I can't find the 16.x version of FreeBSD.

The NIC used here is "igc" which is 'stock' and developed by the the creator itself : Intel.
I'm using the igc driver myself on a 4100, for years now.

Last but not least : How many Protecli devices are there out there used with pfSense CE or Plus ?
I can't tell, but lets make it a safe bet : many thousands. It's probably more.

What is easy to test and it's easy to get back to the current situation :

Make a copy of the current setup first : Diagnostics > Backup & Restore and then :

As soon as the system reboot, with the console, set up WAN, and LAN.
WAN : might be kept on DHCP which is the default.
LAN : accept the 192.168.1.1/24 with a pool like 192.168.1.10 -> 192.168.1.254
Change the password.
Do not enter / change / modify / add any DNS related stuff ( !! ) just use the resolver as it was meant to be.
Do not import the config file you've backup moments ago.
Now, test for some time.
At this very moment how have the same binary, bit-for-bit software and settings as millions of others. If it still fails, you have the 99,9+% guarantee it's hardware based.

For this test, hook up you PC directly to a LAN pfSense port, so you also exclude all switch (or switch power brick) issues. Take also a know good network cable.

If you don't find any issues now, keep on testing like this as long as possible.
Then import the backup up config back in.
If issues start again, you know now where the issue is ^^

[stephenw10]

Mmm, I would still be looking at the console for errors. If it's some low level PCI issue there will be errors shown and/or logged.