Proxmox: WAN NIC passthrough issue with 2.5G connection to fiber router

lp.descamps

Hi all,

I'm encountering an issue with my Proxmox setup where a WAN NIC passed through to a pfSense VM stops working after around 2000 pings. To restore connectivity, I need to access the pfSense interface, navigate to the NIC settings, and click "Apply" without making any changes. This seems to reset the port, and it starts working again.

Here’s a summary of my setup:

Proxmox version: 8.3.1
pfSense version:2.7.2
NIC details: 02:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03) with passthrough configured for the WAN interface
WAN connection: 2.5G Ethernet connected to a fiber router
Behaviour: WAN works fine initially but stops responding after ~2000 pings. The issue resolves temporarily after applying changes to the NIC in pfSense.

I've checked the following:

Proxmox passthrough configuration seems correct, as the NIC is recognized and functional in the pfSense VM.
pfSense logs show no clear indication of why the WAN connection drops.
The issue is reproducible consistently, and other NICs on the Proxmox server don’t exhibit this behaviour.

My questions:

Has anyone encountered a similar issue with Intel I225-V NICs or passthrough configurations?
Could this be related to Proxmox, pfSense, or potential compatibility issues with the 2.5G fiber router?
Are there specific troubleshooting steps or logs I should look into on the Proxmox side to narrow down the cause?

Any advice or pointers would be greatly appreciated. Let me know if more details are needed.

Thanks in advance!

stephenw10

So the WAN gateway stops responding? What is the gateway monitoring set to?

It sounds like something loses an ARP entry. Does the gateway disappear from the ARP table in pfSense? (Diag > ARP).

If you run a pcap on the WAN when it fails, is it still sending the ping packets?

Steve

lp.descamps

@stephenw10

thanks for pointing me out to look at arp
i dont have gw monitoring. the issue remains when it s enabled
my arp table has 2 entries but internet stops when the entries have grown.
it goes up to 90+ entries

my isp is using cgnat

the original 2 entries are still there when it breaks

stephenw10

Hmm, how are testing then? Where are you pinging from?

But, yes, run pcap and see what's being sent and if anything is coming back.

lp.descamps

@stephenw10
the test is that i just lose internet
I did a pcap I can see traffic going out but nothing back
it seems that I get arp entries from every other customers on the ISP
I 'm trying to block them on the firewall but no luck
any suggestions on preventing other customers to flood me with arp?

stephenw10

Seeing broadcast ARPs from other customers is only an indication of a poor setup at the ISP IMO. It should not affect your connectivity unless maybe here is a conflicting MAC address or IP. But pfSense would log that and complain loudly if there were.

You said it stops after ~2000 pings so I assumed you were running a ping test from somewhere against something?

We have seen similar reports of ISP gateways losing ARP entries and instead of ARPing to renew it just dropping traffic which does sound similar. However it normally comes back after some time in those situations. Are you seeing that?

A workaround for that is to set the pfSense ARP renewal time much shorter so it keeps the upstream device current. To test that run at the command line:
sysctl net.link.ether.inet.max_age=300

whosmatt

@lp-descamps said in Proxmox: WAN NIC passthrough issue with 2.5G connection to fiber router:

Has anyone encountered a similar issue with Intel I225-V NICs or passthrough configurations?

I have encountered similar behavior with multiple I225 class NICs, yes. No passthrough in my case. The only thing that showed up in the pfSense logs was repeating entries like:

kernel: arpresolve: can't allocate llinfo for <wan gateway ip > on igc0

Like you, simply "reconfiguring" the interface without any changes would bring back the connection until next time it happened.

I had this problem with 3 different igc NICs, including one OEM Intel part and two cheap chinese parts.

I eventually gave up on igc and got a different NIC, in my case an X710-T4L.

lp.descamps

@whosmatt said in Proxmox: WAN NIC passthrough issue with 2.5G connection to fiber router:

arpresolve

I do get this error too
kernel arpresolve: can't allocate llinfo for 192.168.1.1 on igc0

but I don't use 192.168.1.0 network on pfsense?!?

thanks

lp.descamps

@stephenw10

i m pinging from a laptop to 1.1.1.1

I ve now enable the gw monitoring to ping 8.8.8.8 instead of ISP gw. will how it goes

the internet connection works again after the lease time of the arp expires and disappears from the table and I m back with 2 arp entries from the isp. the gw IP and wan interface

I will try sysctl net.link.ether.inet.max_age=300 if the gw monitoring didn't do the trick. what is the default value for max_age in case I want to roll this back?

thanks

lp.descamps

@lp-descamps
lost internet again

changed the sysctl net.link.ether.inet.max_age from 1200 to 300

I didn't get kernel arpresolve: can't allocate llinfo for 192.168.1.1 on igc0 errors this time

stephenw10

@lp-descamps said in Proxmox: WAN NIC passthrough issue with 2.5G connection to fiber router:

but I don't use 192.168.1.0 network on pfsense?!?

Might be a clue there though. Your WAN is DHCP I'm assuming? Perhaps something else is giving it a lease with that IP as the gateway.

lp.descamps

@stephenw10
there is a conflicting MAC

when internet is working I can see this in arp table,

WAN	100.68.0.1	42:9e:a4:3c:8e:2b		Expires in 1156 seconds	ethernet	  
WAN	100.69.245.16	7c:83:34:bd:4b:21		Permanent	        ethernet

when internet breaks, I get this

WAN	100.68.0.64	42:9e:a4:3c:8e:2b		Expires in 1186 seconds	ethernet	  
WAN	100.68.0.32	42:9e:a4:3c:8e:2b		Expires in 1186 seconds	ethernet	  
WAN	100.68.0.65	42:9e:a4:3c:8e:2b		Expires in 1186 seconds	ethernet	  
WAN	100.69.245.16	7c:83:34:bd:4b:21		Permanent	ethernet	  
WAN	100.68.0.69	42:9e:a4:3c:8e:2b		Expires in 1186 seconds	ethernet	  
WAN	100.68.0.31	42:9e:a4:3c:8e:2b		Expires in 1186 seconds	ethernet

stephenw10

Which of those, if any, is the WAN gateway? If it is it's a long way from the WAN IP or a huge subnet.

That doesn't look like it's necessarily a conflict though. One MAC address could be associated with all those IPs.

lp.descamps

@stephenw10
the wan gw is 100.68.0.1 42:9e:a4:3c:8e:2b
i can see it in the arp table before and after I lose internet
but sometimes I get incomplete instead of MAC

stephenw10

Hmm, if it shows 'incomplete' then it's unable to get the gateway MAC which implies a layer2 failure.

Where is the gateway in this case? Remote, at the ISP?

It does seem like you might be hitting an issue with the NIC though.

lp.descamps

@stephenw10
i believe the gw is remotely at the isp.
all I have at home is a fiber box that seems unmanaged
is there a way to set static arp?
how would I validate a hw nic failure?

stephenw10

I've yet to see the i225 disconnect issues that are relatively well known but as I understand it they usually show as the NIC losing link which I don't think you're seeing?

I would run a pcap on it in the failed state and see if it's actually passing packets at all.

lp.descamps

@stephenw10 that s one of the capture packetcapture-igc0-20241212183144.pcap

stephenw10

Hmm, so no incoming traffic at all.

Have you tried running it as not pass-through so pfSense just sees it as vtnet?

lp.descamps

@stephenw10 i was non-passthrough and had issues hence moving to passthrough