Acceptable packet loss on WAN
-
I have two pfSense boxes - both Supermicro A1SRi based with onboard Intel I354 NICs.
One is connected to a 76Mb ADSL 2 line the other to a FTTC (fibre to the cabinet) 300Mb line.
During downloads which saturate the line I see packet loss on both of them up to 16% (in the Quality section of pfSense).
Externally monitoring the WAN IP of both also shows packet loss of up to 20%.
ADSL 2 (76Mb):
FTTC (300Mb):
Is this level of packet loss normal when maxing out a PPPOE connection in pfSense?
-
I have likely the same as you theoretical 80Mbps vdsl FTTC connection here in the UK and I do not see packet loss.
Do you see that loss reflected in the pfSense quality graphs? (edit: I see you do) What IP are you monitoring there? What is the Firebrick box monitoring?
Steve
-
Stephen what do you use as a modem?
The make up of DSL services in the UK, has a lot of potential issues when trying to manage clean QoS of packets.
Connection from ISP to exchange by isp backhaul, ISP's are required by BT wholesale to shape this connection for each end user,they have to rate limit no higher than the profile speed set by BT wholesale, this is set based on modem sync speed.
Connection from BTw BRAS to exchange shaped based on modem sync speed.
Connection from exchange to end user, shaped, openreach has profiles which provide assured speeds, congestion on this link is extremely unlikely, but there may well still be queuing mechanics in use.
Most user's of pfsense will have a terminating device in front of it, common one's are cheap consumer routers running in bridge mode. As standalone modem's aren't really a thing anymore.So lots of places of potential bottlenecks and dropped packets. With that said tho I seen this issue on pfsense as well, it was really bad on my old kit, when I replaced with qotom its a lot better but not completely gone away. There is of course many users using pfsense who have no such problems, so its tough to nail down a cause, my speculation is its potentially a driver issue with some nic's not behaving properly. My previous pfsense device using expensive i350 ports was barely able to sustain 40mbit/sec without packet loss. I have by far had the best performance out of virtual machines which I expect is because the virtual networking drivers are extremely well developed and maintained. On both pfsense and opnsense I can easily get 900+mbit both directions without packet loss on the virtual networking drivers.
pfSense CE 2.7.2
-
@stephenw10 said in Acceptable packet loss on WAN:
What IP are you monitoring there? What is the Firebrick box monitoring?
Thanks, I'm monitoring the gateway's external facing WAN address.
The firebrick box monitoring is from thinkbroadband's BQM service here:
https://www.thinkbroadband.com/broadband/monitoring/quality -
Fascinating info on how BT throttle broadband here. The engineer showed me the speed my line was capable of before he left - 400Mbps down, 100Mbps up! Sadly it's capped at 300/50 but guess I better not look a gift horse in the mouth!
Most user's of pfsense will have a terminating device in front of it, common one's are cheap consumer routers running in bridge mode. As standalone modem's aren't really a thing anymore.
Thankfully BT gave me a Huawei MT992 Openreach modem which is going straight to my pfSense box.
I have by far had the best performance out of virtual machines which I expect is because the virtual networking drivers are extremely well developed and maintained.
Will try a virtual pfSense device soon as but seems odd a pretty well supported Intel I354 NIC has such issues. I believe one of pfSense's previous Atom C2000 gateways used the exact Supermicro A1SRi board in my box.
-
@chrcoluk said in Acceptable packet loss on WAN:
Stephen what do you use as a modem?
I'm using the Openreach ECI modem on both connections I have. I was using the Huawei HG612B which is slightly faster, but the one I have became very unreliable.
@leonroy said in Acceptable packet loss on WAN:
Thankfully BT gave me a Huawei MT992 Openreach modem which is going straight to my pfSense box.
Nice. Those are highly sought after!
Steve
-
Downloading an Ubuntu ISO. Gateway monitoring against 8.8.8.8:
So ping time goes up but I see zero packet loss. No shaping in pfSense.
Steve
-
@stephenw10 said in Acceptable packet loss on WAN:
@leonroy said in Acceptable packet loss on WAN:
Thankfully BT gave me a Huawei MT992 Openreach modem which is going straight to my pfSense box.
Nice. Those are highly sought after!
I cannot believe how much they go for on ebay...
So literally ran the same test as you did just now with an Ubuntu ISO no less
. Zero packet loss in the Monitoring graph in pfSense and zero loss in the BQM graphs.
The only thing I can think of is that the Zen gateway IP which my router is connected to has changed since I started researching this. Perhaps the new gateway is more reliable?
Either way, think I'll be leaving things alone for now since no serious packet loss problems occurring since yesterday...
-
Actually...just ran another test.
So seems if I download a single big file like an ISO from a single host I get zero packet loss.
However downloading a bunch of little files across many connections seems to cause 9-16% packet loss...
-
Do either of those tests saturate your upload? That is where we more commonly see packet loss and latency issues.
-
@stephenw10 yep, all fully saturated the line with 74Mbps sustained load on the VDSL line and 300Mbps sustained on the G.Fast line.
-
That's the download though, did it saturate the upload?
-
@stephenw10 neither scenario (ISO download or multiple concurrent downloads) saturated the upload. Only the download.
-
I expect a steam download will cause carnage.
I consider that the ultimate test, steam downloads (uncapped in client) are almost like DDOS'ing your own line 30+ download threads.
Basically when a line is saturated you will get packet loss, there has to be, TCP flow is controlled by the fact packets get dropped. However the issue is which packets are getting dropped and the amount been dropped.
Ingress shaping that prioritises small packets e.g. if working perfect would ensure these pings make it through on the monitoring, so presenting a loss free line, and "all" the dropped packets would be from the TCP downloads having practically no impact on throughput. That's the ideal world scenario.
It is a mystery why some people don't see this issue at all, some see it but can fix it via ingress shaping, others see it but cannot fix it at all (without a massive downstream throttle to prevent it). My theory remains a driver/nic issue been possible, but also hard to rule out the intermediate modem.
If possible the first test would be to swap out pfSense for something else temporarily, and if the behaviour is the same, then it suggests an issue either with modem or ISP side queuing. Certainly I think on the modern internet ingress is far more difficult to manage than egress, egress for me has only ever really compromised QoS in the days when it was tiny like on ADSL. Even then it tended to just skyrocket latency rather than cause significant packet loss. Buffer bloat is bad, but its a lesser evil than a mass of indiscriminate dropped packets.
