10%-40% Packet Loss When Somebody On Network Streams Video

SteveITS

@tictag In general this symptom is what traffic shaping is for. If you can identify particular streams then you can lower their priority. Otherwise something like CoDel is supposed to be more automatic.
https://docs.netgate.com/pfsense/en/latest/recipes/codel-limiters.html

@michmoor said in 10%-40% Packet Loss When Somebody On Network Streams Video:

Ok so this action specifically is for MultiWAN set ups but what happens when its only a single Internet connection? What does pfsense do?

It's not very useful in that case, no. Basically some form of "consider this WAN to be offline."

johnpoz

@SteveITS said in 10%-40% Packet Loss When Somebody On Network Streams Video:

It's not very useful in that case

exactly - which is why I have it disabled ;)

@tictag
Doesn't look your states are resetting now - don't see time with no traffic like you were seeing before, as @SteveITS mentions for something like you don't want something that can saturate your line to effect something like real time traffic like video, qos is something you prob want to look into.

tictag

@johnpoz said in 10%-40% Packet Loss When Somebody On Network Streams Video:

qos is something you prob want to look into.

RE Traffic shaping / QoS... ironically, if you look closely at one of the images of pfSense, the 2nd tab from the left if pfSense's documentation on Traffic Shaping! Definitely on the cards, but just need to solve this problem first.

@johnpoz said in 10%-40% Packet Loss When Somebody On Network Streams Video:

Doesn't look your states are resetting now

The previous screenshot was showing the start-stop behaviour of a large file download, the most recent actually shows the problem I'm facing. You can see the pretty stable LAN OUT and WAN IN, but look at LAN IN, this is my video stream being received on pfSense's LAN interface, now look at what pfSense is actually sending out to the Internet over WAN OUT - every traffic drop shown there is where the pings time-out, the video at the receiving end freezes and the Gateway reports massive packet loss.

When adding a large file download i.e. to saturate the WAN pipe, it masks this underlying problem - when a video stream appears on the LAN, pfSense is reporting massive packet loss. Stop the video stream, and everything returns to normal.

I've been given some advice that the likely culprit for all my woes is the USB Ethernet adapter.

Do you guys think this could be causing the interrupt correlation?

johnpoz

@tictag usb interfaces can be problematic for sure.

tictag

Just a quick update, tried swapping over the USB Ethernet adapter to service the LAN and the inbuilt Ethernet NIC to service the WAN...

Let's just say, that did not end well 🤮

The instant I started pfSense, I lost all connectivity to Proxmox and a load of kernel errors relating to Realtek (the USB Ethernet adpater's chipset) were recorded in the System Log.

I'm really starting to believe this USB Ethernet adapter might be the root cause.

SteveITS

@tictag Realtek is also often cited here. :)

johnpoz

Yeah realtek not known for being all that great, but that being said I have some usb2.5ge interfaces and work fine but one is windows and the other end is also a usb2.5 in a synology nas with a 3rd party driver written for it.

And I pump lots and lots of data over them.. lots!!

But lots of issues seen here on forums with realtek or just usb in general not something would choose to use at all.

michmoor

@johnpoz
Do not kill states on gateway failure (Default) setting. Do not sure that’s in play here

tictag

@michmoor
I'll try that tonight (when everybody has gone home). I'm also going to try a bare-metal install as a hail-Mary.

Does anybody have any ideas as to why this would only happen with video streaming data? I mean packets are packets to a firewall, right?

The only differences I can think of between video data and, say, web data is a different TCP port and such traffic is likely tagged with higher priority QoS markers. But http and https both have different TCP ports and audio traffic would likely have even higher QoS markers and that is unaffected.

This fault just boggles my mind 🤯

Oh, I found something else out last night, incoming video does NOT cause this issue, only outgoing, so, for example, you could be viewing 10 people's video streams in a Zoom/Teams meeting just fine, but as soon as you turn your camera on, it blows up.

tictag

Update: so I bought a new SSD and installed pfSence onto bare metal and ... it works perfectly!! No issues at all, video streaming working fine in both directions, CPU usage doesn't move off 1%, Memory doesn't move off 4%, zero packet loss, RTT 0.6ms, RTTsd 0.1ms. Even maxing out my WAN at 100Mbs, ping times only increase by 10ms.

So this is without any doubt whatsoever, a Proxmox issue. Maybe a driver issue with the external USB Ethernet adapter?

Can you install device drivers in Linux or are they all built into the kernel?

johnpoz

@tictag you can for sure add drivers to linux.. modprobe, etc. You can always normally recompile the kernal.. Keep in mind pfsense isn't linux.. So take your talking about adding or changing the driver(s) in proxmox.

I use to be a big fan of running pfsense as vm many years ago, but once I went to its own hardware - it is much nicer and easier.. Sure you could take snapshots for backups and switch versions real easy with a vm.. But having its own hardware is better solution to be honest just all the way around.

Antibiotic

@johnpoz Are you using SG-4860 right? But where you get this. I do not see in pfsense shop this one, only 2100; 4200; 6100 and etc.

johnpoz

@Antibiotic the 4860 is no longer sold.. It went eol back in 2020 I think.. Still kicking it!! solid piece of gear for sure..

I have a ssd sitting on my self waiting for me to install it, which I will do here soon - prob when I get back from my trip to ireland.. I told myself I would switch to ssd when I deployed 24.03.. the built in emmc prob nearing end of life, no real way to tell though how much has been written to it over the years.. So better to just take it out of the picture and hope to get a couple more years out of the old girl..

Antibiotic

@johnpoz Ok . than any replace for this. I mean was any new hardware after SG-4860, except public 2100;4200;6100 and etc?

Antibiotic

@johnpoz said in 10%-40% Packet Loss When Somebody On Network Streams Video:

So better to just take it out of the picture and hope to get a couple more years out of the old girl.

Man in love)))

SteveITS

@Antibiotic said in 10%-40% Packet Loss When Somebody On Network Streams Video:

@johnpoz Ok . than any replace for this. I mean was any new hardware after SG-4860, except public 2100;4200;6100 and etc?

They are listed at https://www.netgate.com/support/product-lifecycle

Antibiotic

@SteveITS said in 10%-40% Packet Loss When Somebody On Network Streams Video:

https://www.netgate.com/support/product-lifecycle

Ok , than I try to see SG-5100 as replacement but this link not working!from here

https://shop.netgate.com/products/5100-pfsense

What the price and technical details?

johnpoz

@Antibiotic I don't think the 5100 is for sale any more.. They prob should/need to update the product cycle page..

Yeah from here

https://www.netgate.com/support/product-lifecycle

The 5100 was EOL in dec 2023

tictag

@johnpoz said in 10%-40% Packet Loss When Somebody On Network Streams Video:

So take your talking about adding or changing the driver(s) in proxmox.

Yeah, I'm thinking if pfSense works fine on its own (bare metal) then the fault must lie with Proxmox. I think Proxmox runs on Debian, but I could be wrong.

I've had many comments about this Realtek-based USB adapter so I figured I'd start with that.

Actually, you know what, I'm going to take the win today! It's working, my users are going to be happy tomorrow. I'm gonna leave this for another day.

Win, win, win!! It's only bloody well working!

Thanks for all your help chaps!

Antibiotic

@johnpoz yEa but according this almost all EOL, only 4200 up to 26 year. What the reason to buy EOL. I do not understand. Netgate planning to produce new line of hardware?