10%-40% Packet Loss When Somebody On Network Streams Video
-
TL;DR When somebody on the internal network starts streaming video, pfSense reports 10-40% gateway packet loss. Anybody seen this issue before?
This is a very strange fault.
Our hardware router died last week, so I replaced it temporarily with an Intel NUC running Proxmox Virtual Environment (v8.2.2) running the latest pfSense CE.
Long story short, everything works fine until somebody on the network starts streaming video (e.g. Zoom, Teams etc).
At that exact time:
- A continuous ping starts to time out 10-40% of the time.
- Large file downloads start-stop, start-stop etc.
- The Gateway (Dashboard) reports 10%-40% packet loss.
- The Traffic Graph (Dashboard) shows LAN traffic in, but intermittent WAN traffic out during stream.
The issue is repeatable: stop the video stream, everything returns to normal, start the video stream, packet loss begins again, rinse and repeat.
Audio seems to be unaffected.
Default firewall and NAT rules i.e. nothing allowed in, everything allowed out.
Two interfaces, WAN and LAN. No VLANs.
LAN: In-built NUC Intel NIC, port eno1 bridged to vmbr0 (default proxmox config), static IP 192.168.xx.xx/24, bandwidth: 1Gbs Ethernet.
WAN: External USB Adapter, port enx00e04c9689bc passed through to pfSense appearing as ue0, static IP 51.xx.xx.xx/30, bandwidth: 100Mbs FTTC.Troubleshooting already done:
- No memory or CPU issues showing in Proxmox or pfSense. Multiple GB memory free, during packet loss CPU at 2-5%.
- Updated Proxmox and pfSense
- Disabled / re-enabled hardware Crypto support
- Disabled / re-enabled IOMMU (and passthrough)
- Re-installed pfSense
- Re-installed Proxmox
- Connected directly to WAN, no issues.
- pfSense Monitoring shows correlation between Quality > Packet Loss and System > Interrupts & System Util
Has anybody experienced this before?
Any help would be warmly received - I've been troubleshooting this issue now for 4 days straight
-
@tictag for starters set your gateway action not to happen on packet loss.. It is quite common to to loose some pings with high load on your bandwidth..
How does it work now?
What is your isp bandwidth?
-
@johnpoz Thanks for replying.
RE gateway monitoring ... I've confirmed 0% packet loss even when our WAN connection is saturated, downloading at full bandwidth capacity (100Mbs) raises CPU to 41%, but still multiple GB memory free and zero packet loss. I've also stress tested this downloading 10GB files continuously for around 20 minutes.
RE ISP bandwidth ... 100MBs FTTC
-
@tictag said in 10%-40% Packet Loss When Somebody On Network Streams Video:
Connected directly to WAN, no issues.
Important to know.
If you can do this : what happens if you remove - for me the most obvious - : proxmox.
pfSense, on a a bare bone device like a 'close to 10 years old' stripped down "PC" can do 100 of Megas/sec.
For me its a 'like here' or 'change that' in proxmox and pfSense will be fine. -
@Gertjan It's a fair point and I was close to doing that at 23:30 last night after I had pulled out what hair I had left!!
The things is, I also need to run Unifi Network and PaperCut (these are both disabled at the moment), hence the Proxmox solution. But that said, Internet Access is more important so if I can't find a resolution, that will be my next step.
-
@tictag said in 10%-40% Packet Loss When Somebody On Network Streams Video:
I've confirmed 0% packet loss even when our WAN connection is saturated
That is not what this is showing
Nor your monitoring graphs showing packet loss..
Turn off the gateway action option..
An intelligent man is sometimes forced to be drunk to spend time with his fools
If you get confused: Listen to the Music Play
Please don't Chat/PM me for help, unless mod related
SG-4860 24.03 | Lab VMs 2.7.2, 24.03 -
@johnpoz said in 10%-40% Packet Loss When Somebody On Network Streams Video:
Turn off the gateway action option..
What does Gateway Monitoring Action do?
The documentation is not very clear. Ive submitted a Redmine to give a bit more clarity to the documentation on why one would want to turn off gateway monitoring action and when it would be appropriate to do so. -
@michmoor The action is taken when pfsense believe the internet connection is down, ie what your monitoring is not answering ping.. If you disable the action then high ping times or what pfsense believe is packet loss because the monitoring IP is not answering pings don't take action..
I have had this set pretty much since day 1.. Because I quite often saturate my links for short bursts and pfsense thinks since the ping times to the monitoring go way up, or showing packet loss it should take some action - when it it shouldn't in my use case.
This is mostly has use only if your doing failover of some sort, where pfsense thinks well this internet connection is bad should switch over to another connection... If you don't have multiple wan interfaces, then you really have no use of taking any action like killing states, etc. Just disable it - if your internet is actually down your not going anywhere anyway - there is nothing for pfsense to actually do if you only have 1 wan interface.
-
@johnpoz said in 10%-40% Packet Loss When Somebody On Network Streams Video:
e had this set pretty much since day 1.. Because I quite often saturate my links for short bursts and pfsense thinks since the ping times to the monitoring go way up, or showing packet loss it should take some action - when it it shouldn't in my use case.
But what I'm asking is what does this do?
What action(s) does pfsense take when there is a loss of pings? The blurb is not clear on why someone would or should disable this. So you asked the OP to disable it but why?
-
@johnpoz said in 10%-40% Packet Loss When Somebody On Network Streams Video:
This is mostly has use only if your doing failover of some sort, where pfsense thinks well this internet connection is bad should switch over to another connection... If you don't have multiple wan interfaces, then you really have no use of taking any action like killing states, etc. Just disable it - if your internet is actually down your not going anywhere anyway - there is nothing for pfsense to actually do if you only have 1 wan interface.
Ok so this action specifically is for MultiWAN set ups but what happens when its only a single Internet connection? What does pfsense do?
Firewall: NetGate,Palo Alto-VM,Juniper SRX
Routing: Juniper, Arista, Cisco
Switching: Juniper, Arista, Cisco
Wireless: Unifi, Aruba IAP
JNCIP,CCNP Enterprise -
@johnpoz said in 10%-40% Packet Loss When Somebody On Network Streams Video:
Turn off the gateway option..
Everybody has gone home now, disabled this option, re-tested, unfortunately, no change.
-
-
@tictag In general this symptom is what traffic shaping is for. If you can identify particular streams then you can lower their priority. Otherwise something like CoDel is supposed to be more automatic.
https://docs.netgate.com/pfsense/en/latest/recipes/codel-limiters.html@michmoor said in 10%-40% Packet Loss When Somebody On Network Streams Video:
Ok so this action specifically is for MultiWAN set ups but what happens when its only a single Internet connection? What does pfsense do?
It's not very useful in that case, no. Basically some form of "consider this WAN to be offline."
-
@SteveITS said in 10%-40% Packet Loss When Somebody On Network Streams Video:
It's not very useful in that case
exactly - which is why I have it disabled ;)
@tictag
Doesn't look your states are resetting now - don't see time with no traffic like you were seeing before, as @SteveITS mentions for something like you don't want something that can saturate your line to effect something like real time traffic like video, qos is something you prob want to look into. -
@johnpoz said in 10%-40% Packet Loss When Somebody On Network Streams Video:
qos is something you prob want to look into.
RE Traffic shaping / QoS... ironically, if you look closely at one of the images of pfSense, the 2nd tab from the left if pfSense's documentation on Traffic Shaping! Definitely on the cards, but just need to solve this problem first.
@johnpoz said in 10%-40% Packet Loss When Somebody On Network Streams Video:
Doesn't look your states are resetting now
The previous screenshot was showing the start-stop behaviour of a large file download, the most recent actually shows the problem I'm facing. You can see the pretty stable LAN OUT and WAN IN, but look at LAN IN, this is my video stream being received on pfSense's LAN interface, now look at what pfSense is actually sending out to the Internet over WAN OUT - every traffic drop shown there is where the pings time-out, the video at the receiving end freezes and the Gateway reports massive packet loss.
When adding a large file download i.e. to saturate the WAN pipe, it masks this underlying problem - when a video stream appears on the LAN, pfSense is reporting massive packet loss. Stop the video stream, and everything returns to normal.
I've been given some advice that the likely culprit for all my woes is the USB Ethernet adapter.
Do you guys think this could be causing the interrupt correlation?
-
@tictag usb interfaces can be problematic for sure.
-
Just a quick update, tried swapping over the USB Ethernet adapter to service the LAN and the inbuilt Ethernet NIC to service the WAN...
Let's just say, that did not end well 🤮
The instant I started pfSense, I lost all connectivity to Proxmox and a load of kernel errors relating to Realtek (the USB Ethernet adpater's chipset) were recorded in the System Log.
I'm really starting to believe this USB Ethernet adapter might be the root cause.
-
@tictag Realtek is also often cited here. :)
-
Yeah realtek not known for being all that great, but that being said I have some usb2.5ge interfaces and work fine but one is windows and the other end is also a usb2.5 in a synology nas with a 3rd party driver written for it.
And I pump lots and lots of data over them.. lots!!
But lots of issues seen here on forums with realtek or just usb in general not something would choose to use at all.
-
@johnpoz
Do not kill states on gateway failure (Default) setting. Do not sure that’s in play here
