10%-40% Packet Loss When Somebody On Network Streams Video

tictag

@johnpoz said in 10%-40% Packet Loss When Somebody On Network Streams Video:

qos is something you prob want to look into.

RE Traffic shaping / QoS... ironically, if you look closely at one of the images of pfSense, the 2nd tab from the left if pfSense's documentation on Traffic Shaping! Definitely on the cards, but just need to solve this problem first.

@johnpoz said in 10%-40% Packet Loss When Somebody On Network Streams Video:

Doesn't look your states are resetting now

The previous screenshot was showing the start-stop behaviour of a large file download, the most recent actually shows the problem I'm facing. You can see the pretty stable LAN OUT and WAN IN, but look at LAN IN, this is my video stream being received on pfSense's LAN interface, now look at what pfSense is actually sending out to the Internet over WAN OUT - every traffic drop shown there is where the pings time-out, the video at the receiving end freezes and the Gateway reports massive packet loss.

When adding a large file download i.e. to saturate the WAN pipe, it masks this underlying problem - when a video stream appears on the LAN, pfSense is reporting massive packet loss. Stop the video stream, and everything returns to normal.

I've been given some advice that the likely culprit for all my woes is the USB Ethernet adapter.

Do you guys think this could be causing the interrupt correlation?

johnpoz

@tictag usb interfaces can be problematic for sure.

tictag

Just a quick update, tried swapping over the USB Ethernet adapter to service the LAN and the inbuilt Ethernet NIC to service the WAN...

Let's just say, that did not end well 🤮

The instant I started pfSense, I lost all connectivity to Proxmox and a load of kernel errors relating to Realtek (the USB Ethernet adpater's chipset) were recorded in the System Log.

I'm really starting to believe this USB Ethernet adapter might be the root cause.

SteveITS

@tictag Realtek is also often cited here. :)

johnpoz

Yeah realtek not known for being all that great, but that being said I have some usb2.5ge interfaces and work fine but one is windows and the other end is also a usb2.5 in a synology nas with a 3rd party driver written for it.

And I pump lots and lots of data over them.. lots!!

But lots of issues seen here on forums with realtek or just usb in general not something would choose to use at all.

michmoor

@johnpoz
Do not kill states on gateway failure (Default) setting. Do not sure that’s in play here

tictag

@michmoor
I'll try that tonight (when everybody has gone home). I'm also going to try a bare-metal install as a hail-Mary.

Does anybody have any ideas as to why this would only happen with video streaming data? I mean packets are packets to a firewall, right?

The only differences I can think of between video data and, say, web data is a different TCP port and such traffic is likely tagged with higher priority QoS markers. But http and https both have different TCP ports and audio traffic would likely have even higher QoS markers and that is unaffected.

This fault just boggles my mind 🤯

Oh, I found something else out last night, incoming video does NOT cause this issue, only outgoing, so, for example, you could be viewing 10 people's video streams in a Zoom/Teams meeting just fine, but as soon as you turn your camera on, it blows up.

tictag

Update: so I bought a new SSD and installed pfSence onto bare metal and ... it works perfectly!! No issues at all, video streaming working fine in both directions, CPU usage doesn't move off 1%, Memory doesn't move off 4%, zero packet loss, RTT 0.6ms, RTTsd 0.1ms. Even maxing out my WAN at 100Mbs, ping times only increase by 10ms.

So this is without any doubt whatsoever, a Proxmox issue. Maybe a driver issue with the external USB Ethernet adapter?

Can you install device drivers in Linux or are they all built into the kernel?

johnpoz

@tictag you can for sure add drivers to linux.. modprobe, etc. You can always normally recompile the kernal.. Keep in mind pfsense isn't linux.. So take your talking about adding or changing the driver(s) in proxmox.

I use to be a big fan of running pfsense as vm many years ago, but once I went to its own hardware - it is much nicer and easier.. Sure you could take snapshots for backups and switch versions real easy with a vm.. But having its own hardware is better solution to be honest just all the way around.

Antibiotic

@johnpoz Are you using SG-4860 right? But where you get this. I do not see in pfsense shop this one, only 2100; 4200; 6100 and etc.

johnpoz

@Antibiotic the 4860 is no longer sold.. It went eol back in 2020 I think.. Still kicking it!! solid piece of gear for sure..

I have a ssd sitting on my self waiting for me to install it, which I will do here soon - prob when I get back from my trip to ireland.. I told myself I would switch to ssd when I deployed 24.03.. the built in emmc prob nearing end of life, no real way to tell though how much has been written to it over the years.. So better to just take it out of the picture and hope to get a couple more years out of the old girl..

Antibiotic

@johnpoz Ok . than any replace for this. I mean was any new hardware after SG-4860, except public 2100;4200;6100 and etc?

Antibiotic

@johnpoz said in 10%-40% Packet Loss When Somebody On Network Streams Video:

So better to just take it out of the picture and hope to get a couple more years out of the old girl.

Man in love)))

SteveITS

@Antibiotic said in 10%-40% Packet Loss When Somebody On Network Streams Video:

@johnpoz Ok . than any replace for this. I mean was any new hardware after SG-4860, except public 2100;4200;6100 and etc?

They are listed at https://www.netgate.com/support/product-lifecycle

Antibiotic

@SteveITS said in 10%-40% Packet Loss When Somebody On Network Streams Video:

https://www.netgate.com/support/product-lifecycle

Ok , than I try to see SG-5100 as replacement but this link not working!from here

https://shop.netgate.com/products/5100-pfsense

What the price and technical details?

johnpoz

@Antibiotic I don't think the 5100 is for sale any more.. They prob should/need to update the product cycle page..

Yeah from here

https://www.netgate.com/support/product-lifecycle

The 5100 was EOL in dec 2023

tictag

@johnpoz said in 10%-40% Packet Loss When Somebody On Network Streams Video:

So take your talking about adding or changing the driver(s) in proxmox.

Yeah, I'm thinking if pfSense works fine on its own (bare metal) then the fault must lie with Proxmox. I think Proxmox runs on Debian, but I could be wrong.

I've had many comments about this Realtek-based USB adapter so I figured I'd start with that.

Actually, you know what, I'm going to take the win today! It's working, my users are going to be happy tomorrow. I'm gonna leave this for another day.

Win, win, win!! It's only bloody well working!

Thanks for all your help chaps!

Antibiotic

@johnpoz yEa but according this almost all EOL, only 4200 up to 26 year. What the reason to buy EOL. I do not understand. Netgate planning to produce new line of hardware?

johnpoz

@Antibiotic the life cycle for hardware isn't very long.. But they do support the software aspect for quite some time.. As you can see my 4860 is still running and its viable for 24.03, etc. I haven't heard of them stopping putting out new pfsense versions for it.. Only thing I heard is like the 3100 might not be viable for updates soon.. cpu not going to be supported any longer with freebsd? But don't quote me on that.. I don't really pay a lot of attention to that stuff unless its my hardware ;) heheh

SteveITS

@Antibiotic they’ve just lasted that long:) we got our 4860 in 2017ish.

The 5100 is EOL too. That page I expect is the replacement “at that time” not currently.

If you’re curious all specs are on this page.
https://docs.netgate.com/pfsense/en/latest/solutions/#end-of-life-appliances