10%-40% Packet Loss When Somebody On Network Streams Video

tictag

@Gertjan It's a fair point and I was close to doing that at 23:30 last night after I had pulled out what hair I had left!!

The things is, I also need to run Unifi Network and PaperCut (these are both disabled at the moment), hence the Proxmox solution. But that said, Internet Access is more important so if I can't find a resolution, that will be my next step.

johnpoz

@tictag said in 10%-40% Packet Loss When Somebody On Network Streams Video:

I've confirmed 0% packet loss even when our WAN connection is saturated

That is not what this is showing

Nor your monitoring graphs showing packet loss..

Turn off the gateway action option..

michmoor

@johnpoz said in 10%-40% Packet Loss When Somebody On Network Streams Video:

Turn off the gateway action option..

What does Gateway Monitoring Action do?
The documentation is not very clear. Ive submitted a Redmine to give a bit more clarity to the documentation on why one would want to turn off gateway monitoring action and when it would be appropriate to do so.

johnpoz

@michmoor The action is taken when pfsense believe the internet connection is down, ie what your monitoring is not answering ping.. If you disable the action then high ping times or what pfsense believe is packet loss because the monitoring IP is not answering pings don't take action..

I have had this set pretty much since day 1.. Because I quite often saturate my links for short bursts and pfsense thinks since the ping times to the monitoring go way up, or showing packet loss it should take some action - when it it shouldn't in my use case.

This is mostly has use only if your doing failover of some sort, where pfsense thinks well this internet connection is bad should switch over to another connection... If you don't have multiple wan interfaces, then you really have no use of taking any action like killing states, etc. Just disable it - if your internet is actually down your not going anywhere anyway - there is nothing for pfsense to actually do if you only have 1 wan interface.

michmoor

@johnpoz said in 10%-40% Packet Loss When Somebody On Network Streams Video:

e had this set pretty much since day 1.. Because I quite often saturate my links for short bursts and pfsense thinks since the ping times to the monitoring go way up, or showing packet loss it should take some action - when it it shouldn't in my use case.

But what I'm asking is what does this do?
What action(s) does pfsense take when there is a loss of pings? The blurb is not clear on why someone would or should disable this. So you asked the OP to disable it but why?

michmoor

@johnpoz said in 10%-40% Packet Loss When Somebody On Network Streams Video:

This is mostly has use only if your doing failover of some sort, where pfsense thinks well this internet connection is bad should switch over to another connection... If you don't have multiple wan interfaces, then you really have no use of taking any action like killing states, etc. Just disable it - if your internet is actually down your not going anywhere anyway - there is nothing for pfsense to actually do if you only have 1 wan interface.

Ok so this action specifically is for MultiWAN set ups but what happens when its only a single Internet connection? What does pfsense do?

tictag

@johnpoz said in 10%-40% Packet Loss When Somebody On Network Streams Video:

Turn off the gateway option..

Everybody has gone home now, disabled this option, re-tested, unfortunately, no change.

johnpoz

@michmoor it can reset the states..

Which isnt going to make for a good day.

SteveITS

@tictag In general this symptom is what traffic shaping is for. If you can identify particular streams then you can lower their priority. Otherwise something like CoDel is supposed to be more automatic.
https://docs.netgate.com/pfsense/en/latest/recipes/codel-limiters.html

@michmoor said in 10%-40% Packet Loss When Somebody On Network Streams Video:

Ok so this action specifically is for MultiWAN set ups but what happens when its only a single Internet connection? What does pfsense do?

It's not very useful in that case, no. Basically some form of "consider this WAN to be offline."

johnpoz

@SteveITS said in 10%-40% Packet Loss When Somebody On Network Streams Video:

It's not very useful in that case

exactly - which is why I have it disabled ;)

@tictag
Doesn't look your states are resetting now - don't see time with no traffic like you were seeing before, as @SteveITS mentions for something like you don't want something that can saturate your line to effect something like real time traffic like video, qos is something you prob want to look into.

tictag

@johnpoz said in 10%-40% Packet Loss When Somebody On Network Streams Video:

qos is something you prob want to look into.

RE Traffic shaping / QoS... ironically, if you look closely at one of the images of pfSense, the 2nd tab from the left if pfSense's documentation on Traffic Shaping! Definitely on the cards, but just need to solve this problem first.

@johnpoz said in 10%-40% Packet Loss When Somebody On Network Streams Video:

Doesn't look your states are resetting now

The previous screenshot was showing the start-stop behaviour of a large file download, the most recent actually shows the problem I'm facing. You can see the pretty stable LAN OUT and WAN IN, but look at LAN IN, this is my video stream being received on pfSense's LAN interface, now look at what pfSense is actually sending out to the Internet over WAN OUT - every traffic drop shown there is where the pings time-out, the video at the receiving end freezes and the Gateway reports massive packet loss.

When adding a large file download i.e. to saturate the WAN pipe, it masks this underlying problem - when a video stream appears on the LAN, pfSense is reporting massive packet loss. Stop the video stream, and everything returns to normal.

I've been given some advice that the likely culprit for all my woes is the USB Ethernet adapter.

Do you guys think this could be causing the interrupt correlation?

johnpoz

@tictag usb interfaces can be problematic for sure.

tictag

Just a quick update, tried swapping over the USB Ethernet adapter to service the LAN and the inbuilt Ethernet NIC to service the WAN...

Let's just say, that did not end well 🤮

The instant I started pfSense, I lost all connectivity to Proxmox and a load of kernel errors relating to Realtek (the USB Ethernet adpater's chipset) were recorded in the System Log.

I'm really starting to believe this USB Ethernet adapter might be the root cause.

SteveITS

@tictag Realtek is also often cited here. :)

johnpoz

Yeah realtek not known for being all that great, but that being said I have some usb2.5ge interfaces and work fine but one is windows and the other end is also a usb2.5 in a synology nas with a 3rd party driver written for it.

And I pump lots and lots of data over them.. lots!!

But lots of issues seen here on forums with realtek or just usb in general not something would choose to use at all.

michmoor

@johnpoz
Do not kill states on gateway failure (Default) setting. Do not sure that’s in play here

tictag

@michmoor
I'll try that tonight (when everybody has gone home). I'm also going to try a bare-metal install as a hail-Mary.

Does anybody have any ideas as to why this would only happen with video streaming data? I mean packets are packets to a firewall, right?

The only differences I can think of between video data and, say, web data is a different TCP port and such traffic is likely tagged with higher priority QoS markers. But http and https both have different TCP ports and audio traffic would likely have even higher QoS markers and that is unaffected.

This fault just boggles my mind 🤯

Oh, I found something else out last night, incoming video does NOT cause this issue, only outgoing, so, for example, you could be viewing 10 people's video streams in a Zoom/Teams meeting just fine, but as soon as you turn your camera on, it blows up.

tictag

Update: so I bought a new SSD and installed pfSence onto bare metal and ... it works perfectly!! No issues at all, video streaming working fine in both directions, CPU usage doesn't move off 1%, Memory doesn't move off 4%, zero packet loss, RTT 0.6ms, RTTsd 0.1ms. Even maxing out my WAN at 100Mbs, ping times only increase by 10ms.

So this is without any doubt whatsoever, a Proxmox issue. Maybe a driver issue with the external USB Ethernet adapter?

Can you install device drivers in Linux or are they all built into the kernel?

johnpoz

@tictag you can for sure add drivers to linux.. modprobe, etc. You can always normally recompile the kernal.. Keep in mind pfsense isn't linux.. So take your talking about adding or changing the driver(s) in proxmox.

I use to be a big fan of running pfsense as vm many years ago, but once I went to its own hardware - it is much nicer and easier.. Sure you could take snapshots for backups and switch versions real easy with a vm.. But having its own hardware is better solution to be honest just all the way around.

Antibiotic

@johnpoz Are you using SG-4860 right? But where you get this. I do not see in pfsense shop this one, only 2100; 4200; 6100 and etc.