For anyone interested in the exciting conclusions... it worked fine in the 16x slot for 2 weeks and is still in there now
I put an I340-T4 in the 1x slot at the same time and left that running and that has been perfectly fine as well

It seems to be an incompatibility between the 1x slot and the I350 specifically but i'm not sure why. In either case, the issue seems to be resolved

It may be something specific to AM5 and the I350 in the 1x, or just the I350 and the 1x alone but if anyone else for some reason tries the same, at least you know what symptoms manifest and what the cause was

Thanks again for those that helped and commented

Probably just that then. But you should see the set options and capabilities for those NICs like:

[2.8.0-RELEASE][admin@t70.stevew.lan]/root: ifconfig -vm igb0 igb0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: WAN options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG> capabilities=4f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>

So there you can see the NIC is both checksum offload and TSO capable but only checksum is enabled.

@stephenw10

I want to use 10GBIT DAC inside my rack and also directly attach to my ISPs Fiber. It'd be the perfect successor for the SG6100 with the SFP+ Addon installed.

Hmm, interesting. Hard to see how that driver could ever have made in difference when passed through like that.

If you don't pass it through you could use the modified driver in Proxmox and then present it to pfSense as a virtual NIC until we get the patched driver in pfSense. Except you're saying the modified driver no longer loads in Proxmox.... 🤔

@stephenw10
ok i sure will do thanks

So initially it booted and then failed after 48hrs and now doesn't boot at all?

Do you have the actual boot logs showing the errors?

@jwt that is great news, thank you!

@bttech56 Hey, this was the only single tim IDs changed. Though I did move from LTE to optical cable, with terminal-only unit, so at the moment it is via ethernet port. Now I have about 3 months without reboot. No root cause identified.

@planedrop yep its an older thread, i arrived here through googling the issue i was having and this was one of the top results.

whilst it is a virtualised environment, the commonality between them exists and my setup is largely indistinguishable from a non-virtualised setup. i'm using dedicated cores, binding IRQs to those cores, isolating them, using hardware passthrough.

theres a variety of other threads who have had similar experiences with SR-IOV too.

just to add this was with the default MTU of 1500 and rx / tx descriptors of 4k on the linux side

improving this on the freebsd side (by placing into the loader.conf.local)

#enable iflib overrides for ixv interfaces dev.ixv.0.iflib.override_qs_enable="1" dev.ixv.1.iflib.override_qs_enable="1" # Set 4K descriptors dev.ixv.0.iflib.override_nrxds="4096" dev.ixv.0.iflib.override_ntxds="4096" dev.ixv.1.iflib.override_nrxds="4096" dev.ixv.1.iflib.override_ntxds="4096" # Enable iflib overrides for ix interface dev.ix.0.iflib.override_qs_enable="1" # Set 4K descriptors for ix interface dev.ix.0.iflib.override_nrxds="4096" dev.ix.0.iflib.override_ntxds="4096"

seems to have helped out with regard to rx_no_dma_resources issues, you can generally check if you are having those same issues by looking at

sysctl dev.ix | grep -E "(r_drops|r_stalls|r_restarts|no_desc_avail|credits|override_n[rt]xds)"

@jra9511 where does it fail in the process.. What error does it present? I just created a new account and didn't have any issues.

creation.jpg

I think I figured it out my mistake to over commit CPU in a VM . Now seens to not get stuck at 100%

You might be able to get OpenWRT to run on there if you can get access. You should see something on the console though. I'd make sure your terminal setup is good, RJ-45 serial lead is correct etc.

@zeroepoch Thank you. That worked like a charm on a 2.5 GBE Realtek adapter. I had to modify the install command as the affected interface was the WAN-side, so I had no internet connection:

pkg install --no-repo-update ./realtek-re-kmod198-198.00.150029.pkg

Hope that Netgate will include this driver in future releases as it is a pain to get this running on a dedicated server at Hetzner.com when the have a 2.5 GBE as a default LAN adapter.

Well, it pretty much has to be bad hardware and almost certainly bad RAM.

You don't need 20GB so I'd try removing some until the error goes away.

@stephenw10 Thank you Stephen!

@stephenw10 Grateful Sir for the information and thank you for the response.

The only way this is possible currently is to use the syslogng package. There you can set a location to use for log storage which can be a different drive or partition. If you then set /var as a ram drive system logging does not write to the drive.

BUT.... there is no built in management of different drives in pfSense so you need to use custom scripts to do that. Hence it's not supportable.

Both will work with VLANs. It may not have some offloading but you can do VLANs entirely in software if needed.

@matyi-szabolcs Hey, I also have the N100. You can monitor CPU freq via pfSense command line using this command:

while ( 1 ) clear sysctl dev.cpu | grep freq: sleep 1 end

Paste that as-is directly into the pfSense shell (using the terminal not UI).

It'll refresh every second and show you all core frequencies like:

dev.cpu.0.freq: 1800 dev.cpu.1.freq: 1600 ...

Then you can do something CPU heavy like restart Suricata or something.