Hmm, interesting. Hard to see how that driver could ever have made in difference when passed through like that.

If you don't pass it through you could use the modified driver in Proxmox and then present it to pfSense as a virtual NIC until we get the patched driver in pfSense. Except you're saying the modified driver no longer loads in Proxmox.... 🤔

@stephenw10
ok i sure will do thanks

So initially it booted and then failed after 48hrs and now doesn't boot at all?

Do you have the actual boot logs showing the errors?

Time for an update, sadly not a good one, i know this is getting a little off topic for pfSense directly but it may still help others

Less than 48 hours and the same issues has happened on the host, spot the quad nic
d8105e6b-78ab-43b7-ab02-cbf4166731dc-image.png

b6386497-d0a1-4317-b43d-afed61305b9d-image.png

33c6e351-c653-4b5f-86f8-6b0a301a13fe-image.png

db3b4058-ab83-4209-a480-81a0883729f7-image.png

enp4s0f0 and f1 are my intel 10gb NICs but they use ixgbe. I tried reloading igb and had no luck
Looking through dmesg i get the following:

[168683.759683] igb 0000:05:00.3 enp5s0f3: malformed Tx packet detected and dropped, LVMMC:0xffffffff [168685.039818] igb 0000:05:00.0 enp5s0f0: malformed Tx packet detected and dropped, LVMMC:0xffffffff [168685.807653] igb 0000:05:00.1 enp5s0f1: malformed Tx packet detected and dropped, LVMMC:0xffffffff [168687.087693] igb 0000:05:00.2 enp5s0f2: malformed Tx packet detected and dropped, LVMMC:0xffffffff [168687.855687] igb 0000:05:00.3 enp5s0f3: malformed Tx packet detected and dropped, LVMMC:0xffffffff [168688.879965] igb 0000:05:00.0 enp5s0f0: malformed Tx packet detected and dropped, LVMMC:0xffffffff [168689.903630] igb 0000:05:00.1 enp5s0f1: malformed Tx packet detected and dropped, LVMMC:0xffffffff [168690.927699] igb 0000:05:00.2 enp5s0f2: malformed Tx packet detected and dropped, LVMMC:0xffffffff [168691.951708] igb 0000:05:00.3 enp5s0f3: malformed Tx packet detected and dropped, LVMMC:0xffffffff [168692.975871] igb 0000:05:00.0 enp5s0f0: malformed Tx packet detected and dropped, LVMMC:0xffffffff [168693.999640] igb 0000:05:00.1 enp5s0f1: malformed Tx packet detected and dropped, LVMMC:0xffffffff [168695.023693] igb 0000:05:00.2 enp5s0f2: malformed Tx packet detected and dropped, LVMMC:0xffffffff [168695.791663] igb 0000:05:00.3 enp5s0f3: malformed Tx packet detected and dropped, LVMMC:0xffffffff [168696.815772] igb 0000:05:00.0 enp5s0f0: malformed Tx packet detected and dropped, LVMMC:0xffffffff [168697.839653] igb 0000:05:00.1 enp5s0f1: malformed Tx packet detected and dropped, LVMMC:0xffffffff [168698.863696] igb 0000:05:00.2 enp5s0f2: malformed Tx packet detected and dropped, LVMMC:0xffffffff [168699.887665] igb 0000:05:00.3 enp5s0f3: malformed Tx packet detected and dropped, LVMMC:0xffffffff [168700.437650] igb 0000:05:00.3: removed PHC on enp5s0f3 [168700.911656] igb 0000:05:00.0 enp5s0f0: malformed Tx packet detected and dropped, LVMMC:0xffffffff [168701.935637] igb 0000:05:00.1 enp5s0f1: malformed Tx packet detected and dropped, LVMMC:0xffffffff [168702.179403] igb 0000:05:00.3 enp5s0f3 (unregistering): left allmulticast mode [168702.179406] igb 0000:05:00.3 enp5s0f3 (unregistering): left promiscuous mode [168702.468717] igb 0000:05:00.2: removed PHC on enp5s0f2 [168702.959649] igb 0000:05:00.2 enp5s0f2: malformed Tx packet detected and dropped, LVMMC:0xffffffff [168704.702443] igb 0000:05:00.2 enp5s0f2 (unregistering): left allmulticast mode [168704.702447] igb 0000:05:00.2 enp5s0f2 (unregistering): left promiscuous mode [168704.952707] igb 0000:05:00.1: removed PHC on enp5s0f1 [168705.007729] igb 0000:05:00.0 enp5s0f0: malformed Tx packet detected and dropped, LVMMC:0xffffffff [168705.775759] igb 0000:05:00.1 enp5s0f1: malformed Tx packet detected and dropped, LVMMC:0xffffffff [168707.776784] igb 0000:05:00.0: removed PHC on enp5s0f0 [168708.847740] igb 0000:05:00.0 enp5s0f0: malformed Tx packet detected and dropped, LVMMC:0xffffffff [168710.590634] igb 0000:05:00.0 enp5s0f0 (unregistering): left allmulticast mode [168710.590637] igb 0000:05:00.0 enp5s0f0 (unregistering): left promiscuous mode [168712.799086] igb: Intel(R) Gigabit Ethernet Network Driver [168712.799089] igb: Copyright (c) 2007-2014 Intel Corporation. [168712.799121] igb 0000:05:00.0: enabling device (0000 -> 0002) [168712.799290] igb 0000:05:00.0 0000:05:00.0 (uninitialized): PCIe link lost [168712.799522] igb: Failed to read reg 0x18! [168712.799610] WARNING: CPU: 2 PID: 1106121 at drivers/net/ethernet/intel/igb/igb_main.c:745 igb_rd32+0x7c/0x90 [igb] [168712.799706] Modules linked in: igb(E+) vhost_net(E) tun(E) vhost(E) vhost_iotlb(E) macvtap(E) macvlan(E) tap(E) xt_nat(E) xt_tcpudp(E) veth(E) xt_conntrack(E) nft_chain_nat(E) xt_MASQUERADE(E) nf_nat(E) nf_conntrack_netlink(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) xfrm_user(E) xt_addrtype(E) nft_compat(E) nf_tables(E) libcrc32c(E) crc32c_generic(E) nfnetlink(E) br_netfilter(E) bridge(E) stp(E) llc(E) nvme_fabrics(E) overlay(E) sunrpc(E) binfmt_misc(E) ntb_netdev(E) ntb_transport(E) ntb_split(E) ntb(E) ioatdma(E) ib_core(E) edac_mce_amd(E) kvm_amd(E) kvm(E) irqbypass(E) ghash_clmulni_intel(E) sha512_ssse3(E) sha256_ssse3(E) sha1_ssse3(E) amdgpu(E) aesni_intel(E) crypto_simd(E) cryptd(E) snd_hda_codec_hdmi(E) rapl(E) drm_exec(E) amdxcp(E) drm_buddy(E) gpu_sched(E) drm_suballoc_helper(E) drm_display_helper(E) snd_hda_intel(E) cec(E) snd_usb_audio(E) snd_intel_dspcfg(E) snd_usbmidi_lib(E) rc_core(E) snd_hda_codec(E) snd_rawmidi(E) snd_seq_device(E) snd_hda_core(E) mc(E) snd_hwdep(E) drm_ttm_helper(E) [168712.799722] snd_pcm(E) wmi_bmof(E) ttm(E) snd_timer(E) sp5100_tco(E) pcspkr(E) drm_kms_helper(E) snd(E) k10temp(E) watchdog(E) soundcore(E) ccp(E) cdc_mbim(E) joydev(E) cdc_wdm(E) sg(E) button(E) evdev(E) loop(E) drm(E) efi_pstore(E) configfs(E) ip_tables(E) x_tables(E) autofs4(E) zfs(POE) spl(OE) efivarfs(E) sr_mod(E) cdrom(E) hid_generic(E) cdc_ncm(E) cdc_ether(E) usbnet(E) usbhid(E) uas(E) hid(E) usb_storage(E) mii(E) sd_mod(E) ahci(E) nvme(E) ahciem(E) xhci_pci(E) libahci(E) xhci_hcd(E) ixgbe(E) nvme_core(E) libata(E) xfrm_algo(E) t10_pi(E) crc32_pclmul(E) mdio_devres(E) usbcore(E) scsi_mod(E) crc32c_intel(E) crc64_rocksoft(E) i2c_piix4(E) usb_common(E) scsi_common(E) crc64(E) crc_t10dif(E) i2c_algo_bit(E) libphy(E) dca(E) crct10dif_generic(E) mdio(E) crct10dif_pclmul(E) crct10dif_common(E) video(E) wmi(E) gpio_amdpt(E) gpio_generic(E) [last unloaded: igb(E)] [168712.800964] RIP: 0010:igb_rd32+0x7c/0x90 [igb] [168712.803497] ? igb_rd32+0x7c/0x90 [igb] [168712.803880] ? igb_rd32+0x7c/0x90 [igb] [168712.805033] ? igb_rd32+0x7c/0x90 [igb] [168712.805228] ? igb_rd32+0x7c/0x90 [igb] [168712.805422] igb_get_invariants_82575+0xa6/0xf00 [igb] [168712.805620] igb_probe+0x3be/0x1520 [igb] [168712.807796] ? __pfx_igb_init_module+0x10/0x10 [igb] [168713.136674] igb 0000:05:00.0: PHY reset is blocked due to SOL/IDER session. [168714.866546] igb 0000:05:00.0: The NVM Checksum Is Not Valid [168715.129000] igb: probe of 0000:05:00.0 failed with error -5 [168715.129479] igb 0000:05:00.1: enabling device (0000 -> 0002) [168715.129619] igb 0000:05:00.1 0000:05:00.1 (uninitialized): PCIe link lost [168715.130133] igb: Failed to read reg 0x18! [168715.130381] WARNING: CPU: 10 PID: 1106121 at drivers/net/ethernet/intel/igb/igb_main.c:745 igb_rd32+0x7c/0x90 [igb] [168715.130642] Modules linked in: igb(E+) [168715.130922] snd_timer(E) sp5100_tco(E) pcspkr(E) drm_kms_helper(E) snd(E) k10temp(E) watchdog(E) soundcore(E) ccp(E) cdc_mbim(E) joydev(E) cdc_wdm(E) sg(E) button(E) evdev(E) loop(E) drm(E) efi_pstore(E) configfs(E) ip_tables(E) x_tables(E) autofs4(E) zfs(POE) spl(OE) efivarfs(E) sr_mod(E) cdrom(E) hid_generic(E) cdc_ncm(E) cdc_ether(E) usbnet(E) usbhid(E) uas(E) hid(E) usb_storage(E) mii(E) sd_mod(E) ahci(E) nvme(E) ahciem(E) xhci_pci(E) libahci(E) xhci_hcd(E) ixgbe(E) nvme_core(E) libata(E) xfrm_algo(E) t10_pi(E) crc32_pclmul(E) mdio_devres(E) usbcore(E) scsi_mod(E) crc32c_intel(E) crc64_rocksoft(E) i2c_piix4(E) usb_common(E) scsi_common(E) crc64(E) crc_t10dif(E) i2c_algo_bit(E) libphy(E) dca(E) crct10dif_generic(E) mdio(E) crct10dif_pclmul(E) crct10dif_common(E) video(E) wmi(E) gpio_amdpt(E) gpio_generic(E) [last unloaded: igb(E)] [168715.133023] RIP: 0010:igb_rd32+0x7c/0x90 [igb] [168715.137255] ? igb_rd32+0x7c/0x90 [igb] [168715.137508] ? igb_rd32+0x7c/0x90 [igb] [168715.138247] ? igb_rd32+0x7c/0x90 [igb] [168715.138250] ? igb_rd32+0x7c/0x90 [igb] [168715.139694] igb_get_invariants_82575+0xa6/0xf00 [igb] [168715.139700] igb_probe+0x3be/0x1520 [igb] [168715.142610] ? __pfx_igb_init_module+0x10/0x10 [igb] [168715.472667] igb 0000:05:00.1: PHY reset is blocked due to SOL/IDER session. [168717.202621] igb 0000:05:00.1: The NVM Checksum Is Not Valid [168717.400883] igb: probe of 0000:05:00.1 failed with error -5 [168717.401370] igb 0000:05:00.2: enabling device (0000 -> 0002) [168717.401509] igb 0000:05:00.2 0000:05:00.2 (uninitialized): PCIe link lost [168717.402106] igb: Failed to read reg 0x18! [168717.402357] WARNING: CPU: 2 PID: 1106121 at drivers/net/ethernet/intel/igb/igb_main.c:745 igb_rd32+0x7c/0x90 [igb] [168717.402619] Modules linked in: igb(E+) vhost_net(E) tun(E) vhost(E) vhost_iotlb(E) macvtap(E) macvlan(E) tap(E) xt_nat(E) xt_tcpudp(E) veth(E) xt_conntrack(E) nft_chain_nat(E) xt_MASQUERADE(E) nf_nat(E) nf_conntrack_netlink(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) xfrm_user(E) xt_addrtype(E) nft_compat(E) nf_tables(E) libcrc32c(E) crc32c_generic(E) nfnetlink(E) br_netfilter(E) bridge(E) stp(E) llc(E) nvme_fabrics(E) overlay(E) sunrpc(E) binfmt_misc(E) ntb_netdev(E) ntb_transport(E) ntb_split(E) ntb(E) ioatdma(E) ib_core(E) edac_mce_amd(E) kvm_amd(E) kvm(E) irqbypass(E) ghash_clmulni_intel(E) sha512_ssse3(E) sha256_ssse3(E) sha1_ssse3(E) amdgpu(E) aesni_intel(E) crypto_simd(E) cryptd(E) snd_hda_codec_hdmi(E) rapl(E) drm_exec(E) amdxcp(E) drm_buddy(E) gpu_sched(E) drm_suballoc_helper(E) drm_display_helper(E) snd_hda_intel(E) cec(E) snd_usb_audio(E) snd_intel_dspcfg(E) snd_usbmidi_lib(E) rc_core(E) snd_hda_codec(E) snd_rawmidi(E) snd_seq_device(E) snd_hda_core(E) mc(E) snd_hwdep(E) drm_ttm_helper(E) [168717.402644] snd_pcm(E) wmi_bmof(E) ttm(E) snd_timer(E) sp5100_tco(E) pcspkr(E) drm_kms_helper(E) snd(E) k10temp(E) watchdog(E) soundcore(E) ccp(E) cdc_mbim(E) joydev(E) cdc_wdm(E) sg(E) button(E) evdev(E) loop(E) drm(E) efi_pstore(E) configfs(E) ip_tables(E) x_tables(E) autofs4(E) zfs(POE) spl(OE) efivarfs(E) sr_mod(E) cdrom(E) hid_generic(E) cdc_ncm(E) cdc_ether(E) usbnet(E) usbhid(E) uas(E) hid(E) usb_storage(E) mii(E) sd_mod(E) ahci(E) nvme(E) ahciem(E) xhci_pci(E) libahci(E) xhci_hcd(E) ixgbe(E) nvme_core(E) libata(E) xfrm_algo(E) t10_pi(E) crc32_pclmul(E) mdio_devres(E) usbcore(E) scsi_mod(E) crc32c_intel(E) crc64_rocksoft(E) i2c_piix4(E) usb_common(E) scsi_common(E) crc64(E) crc_t10dif(E) i2c_algo_bit(E) libphy(E) dca(E) crct10dif_generic(E) mdio(E) crct10dif_pclmul(E) crct10dif_common(E) video(E) wmi(E) gpio_amdpt(E) gpio_generic(E) [last unloaded: igb(E)] [168717.405253] RIP: 0010:igb_rd32+0x7c/0x90 [igb] [168717.408900] ? igb_rd32+0x7c/0x90 [igb] [168717.409393] ? igb_rd32+0x7c/0x90 [igb] [168717.410845] ? igb_rd32+0x7c/0x90 [igb] [168717.411088] ? igb_rd32+0x7c/0x90 [igb] [168717.411330] igb_get_invariants_82575+0xa6/0xf00 [igb] [168717.411577] igb_probe+0x3be/0x1520 [igb] [168717.414240] ? __pfx_igb_init_module+0x10/0x10 [igb] [168717.744687] igb 0000:05:00.2: PHY reset is blocked due to SOL/IDER session. [168719.474678] igb 0000:05:00.2: The NVM Checksum Is Not Valid [168719.664847] igb: probe of 0000:05:00.2 failed with error -5 [168719.665336] igb 0000:05:00.3: enabling device (0000 -> 0002) [168719.665475] igb 0000:05:00.3 0000:05:00.3 (uninitialized): PCIe link lost [168719.666111] igb: Failed to read reg 0x18! [168719.666361] WARNING: CPU: 10 PID: 1106121 at drivers/net/ethernet/intel/igb/igb_main.c:745 igb_rd32+0x7c/0x90 [igb] [168719.666623] Modules linked in: igb(E+) vhost_net(E) tun(E) vhost(E) vhost_iotlb(E) macvtap(E) macvlan(E) tap(E) xt_nat(E) xt_tcpudp(E) veth(E) xt_conntrack(E) nft_chain_nat(E) xt_MASQUERADE(E) nf_nat(E) nf_conntrack_netlink(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) xfrm_user(E) xt_addrtype(E) nft_compat(E) nf_tables(E) libcrc32c(E) crc32c_generic(E) nfnetlink(E) br_netfilter(E) bridge(E) stp(E) llc(E) nvme_fabrics(E) overlay(E) sunrpc(E) binfmt_misc(E) ntb_netdev(E) ntb_transport(E) ntb_split(E) ntb(E) ioatdma(E) ib_core(E) edac_mce_amd(E) kvm_amd(E) kvm(E) irqbypass(E) ghash_clmulni_intel(E) sha512_ssse3(E) sha256_ssse3(E) sha1_ssse3(E) amdgpu(E) aesni_intel(E) crypto_simd(E) cryptd(E) snd_hda_codec_hdmi(E) rapl(E) drm_exec(E) amdxcp(E) drm_buddy(E) gpu_sched(E) drm_suballoc_helper(E) drm_display_helper(E) snd_hda_intel(E) cec(E) snd_usb_audio(E) snd_intel_dspcfg(E) snd_usbmidi_lib(E) rc_core(E) snd_hda_codec(E) snd_rawmidi(E) snd_seq_device(E) snd_hda_core(E) mc(E) snd_hwdep(E) drm_ttm_helper(E) [168719.666648] snd_pcm(E) wmi_bmof(E) ttm(E) snd_timer(E) sp5100_tco(E) pcspkr(E) drm_kms_helper(E) snd(E) k10temp(E) watchdog(E) soundcore(E) ccp(E) cdc_mbim(E) joydev(E) cdc_wdm(E) sg(E) button(E) evdev(E) loop(E) drm(E) efi_pstore(E) configfs(E) ip_tables(E) x_tables(E) autofs4(E) zfs(POE) spl(OE) efivarfs(E) sr_mod(E) cdrom(E) hid_generic(E) cdc_ncm(E) cdc_ether(E) usbnet(E) usbhid(E) uas(E) hid(E) usb_storage(E) mii(E) sd_mod(E) ahci(E) nvme(E) ahciem(E) xhci_pci(E) libahci(E) xhci_hcd(E) ixgbe(E) nvme_core(E) libata(E) xfrm_algo(E) t10_pi(E) crc32_pclmul(E) mdio_devres(E) usbcore(E) scsi_mod(E) crc32c_intel(E) crc64_rocksoft(E) i2c_piix4(E) usb_common(E) scsi_common(E) crc64(E) crc_t10dif(E) i2c_algo_bit(E) libphy(E) dca(E) crct10dif_generic(E) mdio(E) crct10dif_pclmul(E) crct10dif_common(E) video(E) wmi(E) gpio_amdpt(E) gpio_generic(E) [last unloaded: igb(E)] [168719.669264] RIP: 0010:igb_rd32+0x7c/0x90 [igb] [168719.672935] ? igb_rd32+0x7c/0x90 [igb] [168719.673429] ? igb_rd32+0x7c/0x90 [igb] [168719.674884] ? igb_rd32+0x7c/0x90 [igb] [168719.675127] ? igb_rd32+0x7c/0x90 [igb] [168719.675369] igb_get_invariants_82575+0xa6/0xf00 [igb] [168719.675616] igb_probe+0x3be/0x1520 [igb] [168719.678284] ? __pfx_igb_init_module+0x10/0x10 [igb] [168720.008929] igb 0000:05:00.3: PHY reset is blocked due to SOL/IDER session. [168721.738903] igb 0000:05:00.3: The NVM Checksum Is Not Valid [168721.916849] igb: probe of 0000:05:00.3 failed with error -5

My last port of call now is to pull the nic and try it in a difference PCI-e slot
The current slot is a gen 3 1x slot so should be fine but i will try it in a gen 5 16x on the same system just in case its causing issues. I doubt it but it is indeed looking like some kind of hardware issue now

Regards,
Jamie

@jwt that is great news, thank you!

@bttech56 Hey, this was the only single tim IDs changed. Though I did move from LTE to optical cable, with terminal-only unit, so at the moment it is via ethernet port. Now I have about 3 months without reboot. No root cause identified.

@planedrop yep its an older thread, i arrived here through googling the issue i was having and this was one of the top results.

whilst it is a virtualised environment, the commonality between them exists and my setup is largely indistinguishable from a non-virtualised setup. i'm using dedicated cores, binding IRQs to those cores, isolating them, using hardware passthrough.

theres a variety of other threads who have had similar experiences with SR-IOV too.

just to add this was with the default MTU of 1500 and rx / tx descriptors of 4k on the linux side

improving this on the freebsd side (by placing into the loader.conf.local)

#enable iflib overrides for ixv interfaces dev.ixv.0.iflib.override_qs_enable="1" dev.ixv.1.iflib.override_qs_enable="1" # Set 4K descriptors dev.ixv.0.iflib.override_nrxds="4096" dev.ixv.0.iflib.override_ntxds="4096" dev.ixv.1.iflib.override_nrxds="4096" dev.ixv.1.iflib.override_ntxds="4096" # Enable iflib overrides for ix interface dev.ix.0.iflib.override_qs_enable="1" # Set 4K descriptors for ix interface dev.ix.0.iflib.override_nrxds="4096" dev.ix.0.iflib.override_ntxds="4096"

seems to have helped out with regard to rx_no_dma_resources issues, you can generally check if you are having those same issues by looking at

sysctl dev.ix | grep -E "(r_drops|r_stalls|r_restarts|no_desc_avail|credits|override_n[rt]xds)"

@jra9511 where does it fail in the process.. What error does it present? I just created a new account and didn't have any issues.

creation.jpg

I think I figured it out my mistake to over commit CPU in a VM . Now seens to not get stuck at 100%

You might be able to get OpenWRT to run on there if you can get access. You should see something on the console though. I'd make sure your terminal setup is good, RJ-45 serial lead is correct etc.

@zeroepoch Thank you. That worked like a charm on a 2.5 GBE Realtek adapter. I had to modify the install command as the affected interface was the WAN-side, so I had no internet connection:

pkg install --no-repo-update ./realtek-re-kmod198-198.00.150029.pkg

Hope that Netgate will include this driver in future releases as it is a pain to get this running on a dedicated server at Hetzner.com when the have a 2.5 GBE as a default LAN adapter.

Well, it pretty much has to be bad hardware and almost certainly bad RAM.

You don't need 20GB so I'd try removing some until the error goes away.

@stephenw10 Thank you Stephen!

@stephenw10 Grateful Sir for the information and thank you for the response.

The only way this is possible currently is to use the syslogng package. There you can set a location to use for log storage which can be a different drive or partition. If you then set /var as a ram drive system logging does not write to the drive.

BUT.... there is no built in management of different drives in pfSense so you need to use custom scripts to do that. Hence it's not supportable.

Both will work with VLANs. It may not have some offloading but you can do VLANs entirely in software if needed.

@matyi-szabolcs Hey, I also have the N100. You can monitor CPU freq via pfSense command line using this command:

while ( 1 ) clear sysctl dev.cpu | grep freq: sleep 1 end

Paste that as-is directly into the pfSense shell (using the terminal not UI).

It'll refresh every second and show you all core frequencies like:

dev.cpu.0.freq: 1800 dev.cpu.1.freq: 1600 ...

Then you can do something CPU heavy like restart Suricata or something.

@stephenw10 i have strange fetish for this kind of devices,i'd like to get my hands on them,but still,i'll have to use some bigger pc-like or passive and keep those for tinker purposes...
i didn't even installed windows 10/11 there yet..just for fun

addon card eats about 2.5W idle unconnected.i'm now down to ~23.8W

OK, Thanks Stephen