Errors out going up of pfsense VM when saturating LAN interface

[netnerdy]

I'm trying to diagnose an issue that I have on my proxmox host.

I have a pfsense 2.6 VM, running on virtio (vtnet) adapter.

I also have ubuntu(22.04), truenas (core 13), freebsd (12.3-Release) VMs on the same host with virtio network adapters.

Here is the issue: I see "errors out" increasing on pfsense LAN interface when I run iperf3 between pfsense and each host.

I tried running between pfsense <-> ubuntu host (errors out increases)
I tried running between pfsense <-> truenas host (errors out increases)
I tried running between pfsense <-> freebsd host (errors out increases)

I tried running between truenas <-> freebsd host (no errors out)
I tried running between truenas <-> ubuntu host (no errors out)

Proxmox is version 7.2-3. Kernel version is 5.15.30-2-pve.

Any help is greatly appreciated.
Thanks!

[Cool_Corona]

@netnerdy Thats normal because all traffic gets bogged down when the link is saturated.

[netnerdy]

@cool_corona It does not seem normal from my tests. i have used pfsense for years on different platforms. I didn’t see this behavior when I ran pfsense on bare metal or esxi or older versions of proxmox.

Thats why I also created a vanilla freebsd vm to do the same tests. Freebsd does not show the same kind of errors when running iperf.

[viragomann]

@netnerdy
Did you enable hardware checksum offloading in the system advanced settings?

[stephenw10]

Do you have flow control set the same on each?

[netnerdy]

@viragomann Yes I have tried enabling/disabling both tcp segmentation and hw checksum, but no avail.

below is ifconfig output. vtnet0 is WAN, vtnet1 is LAN. My WAN and LAN both use vlans btw.

vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=900b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,LINKSTATE>

vtnet1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=900b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,LINKSTATE>

[stephenw10]

Are you actually seeing traffic dropped?

Can you see what type of errors they are in the output of sysctl dev.vtnet.0?

[netnerdy]

@stephenw10 Flow control is off for Hypervisor raw interface. does vtnet have its own flow control setting?

Right now, error count is at 967 (i had to restart my machine). here is sysctl:

dev.vtnet.1.txq0.rescheduled: 0
dev.vtnet.1.txq0.tso: 0
dev.vtnet.1.txq0.csum: 0
dev.vtnet.1.txq0.omcasts: 103
dev.vtnet.1.txq0.obytes: 7095877990
dev.vtnet.1.txq0.opackets: 14070402
dev.vtnet.1.rxq0.rescheduled: 143
dev.vtnet.1.rxq0.csum_failed: 0
dev.vtnet.1.rxq0.csum: 2228565
dev.vtnet.1.rxq0.ierrors: 0
dev.vtnet.1.rxq0.iqdrops: 0
dev.vtnet.1.rxq0.ibytes: 16575625620
dev.vtnet.1.rxq0.ipackets: 12832966
dev.vtnet.1.tx_task_rescheduled: 0
dev.vtnet.1.tx_tso_offloaded: 0
dev.vtnet.1.tx_csum_offloaded: 0
dev.vtnet.1.tx_defrag_failed: 0
dev.vtnet.1.tx_defragged: 0
dev.vtnet.1.tx_tso_not_tcp: 0
dev.vtnet.1.tx_tso_bad_ethtype: 0
dev.vtnet.1.tx_csum_bad_ethtype: 0
dev.vtnet.1.rx_task_rescheduled: 0
dev.vtnet.1.rx_csum_offloaded: 0
dev.vtnet.1.rx_csum_failed: 0
dev.vtnet.1.rx_csum_bad_proto: 0
dev.vtnet.1.rx_csum_bad_offset: 0
dev.vtnet.1.rx_csum_bad_ipproto: 0
dev.vtnet.1.rx_csum_bad_ethtype: 0
dev.vtnet.1.rx_mergeable_failed: 0
dev.vtnet.1.rx_enq_replacement_failed: 0
dev.vtnet.1.rx_frame_too_large: 0
dev.vtnet.1.mbuf_alloc_failed: 0
dev.vtnet.1.act_vq_pairs: 1
dev.vtnet.1.requested_vq_pairs: 0
dev.vtnet.1.max_vq_pairs: 1
dev.vtnet.1.%parent: virtio_pci3
dev.vtnet.1.%pnpinfo:
dev.vtnet.1.%location:
dev.vtnet.1.%driver: vtnet
dev.vtnet.1.%desc: VirtIO Networking Adapter

[stephenw10]

Hmm, nothing shown there...

Are you seeing that only on vtnet1?

Are you actually seeing connection dropped when the error rate is increasing?

[netnerdy]

@stephenw10

[stephenw10]

Ah, that looks like LAN is actually vtnet0. Can we see the sysctl output for that?

[netnerdy]

@stephenw10

nope LAN is on vtnet1.

this is the sysctl output for vtnet0

dev.vtnet.0.txq0.rescheduled: 0
dev.vtnet.0.txq0.tso: 0
dev.vtnet.0.txq0.csum: 0
dev.vtnet.0.txq0.omcasts: 1139
dev.vtnet.0.txq0.obytes: 10781607078
dev.vtnet.0.txq0.opackets: 16092681
dev.vtnet.0.rxq0.rescheduled: 21
dev.vtnet.0.rxq0.csum_failed: 0
dev.vtnet.0.rxq0.csum: 16804156
dev.vtnet.0.rxq0.ierrors: 0
dev.vtnet.0.rxq0.iqdrops: 0
dev.vtnet.0.rxq0.ibytes: 29382393539
dev.vtnet.0.rxq0.ipackets: 26088829
dev.vtnet.0.tx_task_rescheduled: 0
dev.vtnet.0.tx_tso_offloaded: 0
dev.vtnet.0.tx_csum_offloaded: 0
dev.vtnet.0.tx_defrag_failed: 0
dev.vtnet.0.tx_defragged: 0
dev.vtnet.0.tx_tso_not_tcp: 0
dev.vtnet.0.tx_tso_bad_ethtype: 0
dev.vtnet.0.tx_csum_bad_ethtype: 0
dev.vtnet.0.rx_task_rescheduled: 0
dev.vtnet.0.rx_csum_offloaded: 0
dev.vtnet.0.rx_csum_failed: 0
dev.vtnet.0.rx_csum_bad_proto: 0
dev.vtnet.0.rx_csum_bad_offset: 0
dev.vtnet.0.rx_csum_bad_ipproto: 0
dev.vtnet.0.rx_csum_bad_ethtype: 0
dev.vtnet.0.rx_mergeable_failed: 0
dev.vtnet.0.rx_enq_replacement_failed: 0
dev.vtnet.0.rx_frame_too_large: 0
dev.vtnet.0.mbuf_alloc_failed: 0
dev.vtnet.0.act_vq_pairs: 1
dev.vtnet.0.requested_vq_pairs: 0
dev.vtnet.0.max_vq_pairs: 1
dev.vtnet.0.%parent: virtio_pci2
dev.vtnet.0.%pnpinfo:
dev.vtnet.0.%location:
dev.vtnet.0.%driver: vtnet
dev.vtnet.0.%desc: VirtIO Networking Adapter

[stephenw10]

Hmm, interesting the in/out bytes in your screenshot for WAN lined up exactly with the sysctl output for vtnet1. Just coincidence perhaps? Were those takes at approximately the same time?

You still haven't said if you're actually seeing connections drop when you see those errors?

Steve

[netnerdy]

@stephenw10 They were taken at different times. It must have been a coincidence.

How would I know which connections have dropped? Tcps auto retry anyway. Only weirdness i feel i have is my dns queries sometimes don’t resolve intermittently.

[stephenw10]

Ah, OK do you see anything in the sysctls that looks like the same error count shown in the interface status?