Low throughput and packloss with pfsense and cable modem in bridge mode.
-
I have a 1000/100 cable connection on a local cable ISP in Denmark. Running an ISP supplied Arris TG3442 modem/router.
pfSense runs on a XSK NUC Intel Celeron J3160 with 4 intel gigabit LAN ports (igb)
Running the modem in router mode with my public IP assigned to the router, and with pfsense behind thus double NAT, runs fine. Getting full speed, and grade B bufferbloat on tests.
Running the modem in bridge mode and the pfsense grabbing my public IP, results in slow download speeds, bufferbloat grade F, temporary loss of connectivity and general packet loss, but only when under load. As soon as load stops, connectivity picks up.
I have had a macbook running osx connected directly to the bridged modem and assigned my public IP. Running iperf against a 1gbit fiber connection, I got full speed both ways (934/110 avg) with minimal jitter and packet loss.
When pfsense is behind modem in bridge mode and directly assigned the public IP, iperf3 throughput will be between 100-400mbps, with high packet loss/loss of connectivity for other connected devices. This is both running iperf3 from the pfsense itself, or from LAN clients. This only affects TCP though. An UDP iperf3 test, will yield full speed with minimal packet loss (< 0.5%), using 9 threads at 100mbps each.
I’ve enabled/disabled hardware offloading, tso, ethernet flow control, increased mbufs and whatever else I could find about optimizing performance
Running an iperf3 test locally routed through the pfsense, yielded full speed, zero packet loss, while clients behind pfsense still had connectivity and no packet loss. I’ve tried various traffic shaping configurations and limiters, to no avail.I am at a loss to what is going on here. I am aware of the pump6 chipset issues with some cable modems, including mine, but it does not seem like this is related, as this is only with the pfsense connected - the macbook ran perfectly, and so does router mode.
I read up on some DOCSIS stuff, and it appears that some cable modems will do some internal TCP ACK hackery to compensate for the upstream running in bursts, but again I don’t know how to explain the good test results with the macbook as sole client, and it is a bit beyond the realm of my knowledge.
Next step is a call from tier 2 ISP support, and I'll perhaps try some other routing platform - in the meantime, does anyone have any suggestions for a possible solution?
-
I trust you know that TCP flow control requires some packet loss to work.
-
I am not talking about some minor packetloss, inherent in TCP, but very high icmp packetloss, very slow downloadspeed, or total loss of connectivity during load.
I am sorry if i did not make that clear.Sep 5 11:18:25 rc.gateway_alarm 43569 >>> Gateway alarm: WAN_DHCP (Addr:37.205.xxx.xxx Alarm:1 RTT:6.308ms RTTsd:1.357ms Loss:22%) Sep 8 10:45:43 rc.gateway_alarm 12133 >>> Gateway alarm: WAN_DHCP (Addr:37.205.xxx.xxx Alarm:1 RTT:122.669ms RTTsd:91.533ms Loss:21%) Sep 8 10:48:52 rc.gateway_alarm 86454 >>> Gateway alarm: WAN_DHCP (Addr:37.205.xxx.xxx Alarm:0 RTT:82.858ms RTTsd:92.843ms Loss:15%) Sep 8 16:10:02 rc.gateway_alarm 6178 >>> Gateway alarm: WAN_DHCP (Addr:37.205.xxx.xxx Alarm:1 RTT:39.456ms RTTsd:150.982ms Loss:22%) Sep 8 16:10:45 rc.gateway_alarm 54332 >>> Gateway alarm: WAN_DHCP (Addr:37.205.xxx.xxx Alarm:1 RTT:576.200ms RTTsd:324.412ms Loss:95%) Sep 8 16:10:56 rc.gateway_alarm 97633 >>> Gateway alarm: WAN_DHCP (Addr:37.205.xxx.xxx Alarm:1 RTT:0.000ms RTTsd:0.000ms Loss:100%) -
The WAN connection is DHCP? Not PPPoE?
-
Yeah, no PPPoE, just straight up DHCP.
-
Hmm, what sort of CPU usage do you see when testing this? How does it spread across the cores? The output of
top -aSHwill show that.What you describe is exactly what I'd expect if it's somehow using 1 CPU core and that is pegged at 100%.
Steve
-
@stephenw10 said in Low throughput and packloss with pfsense and cable modem in bridge mode.:
Hmm, what sort of CPU usage do you see when testing this? How does it spread across the cores? The output of
top -aSHwill show that.What you describe is exactly what I'd expect if it's somehow using 1 CPU core and that is pegged at 100%.
Steve
I'll have a look when I have opportunity. The CPU usage on the dashboard does not peak out at 100%, though I realize this might now show the full picture.
However, the router is capable of routing at fullspeed when modem is in router mode, or when I did a local test through the router - Where i had a iperf client on the LAN side of the interface, and an iperf client on the WAN side - it had full throughput.
-
This is from double NAT setup, but should still give a look into cpu usage. I don't know why that whould change, with a public ip on the wan interface, instead of an internal. I can do the other test, I just requires more time and planning, since the network will down.
-
Mmm, no problem there. That's exactly what I expect to see, all 4 cores still show some idle time.
Hard to say what's different when connected direct. It's obviously physically it's connected to a different NIC so that would be where I'd first look. It does 'feel' like a flow control issue though you said you tried that.
Steve
-
@stephenw10 said in Low throughput and packloss with pfsense and cable modem in bridge mode.:
Mmm, no problem there. That's exactly what I expect to see, all 4 cores still show some idle time.
Hard to say what's different when connected direct. It's obviously physically it's connected to a different NIC so that would be where I'd first look. It does 'feel' like a flow control issue though you said you tried that.
Steve
Actually, the physical setup remains the same. Its still just connected directly to one of the switch ports of the Arris modem, no matter if the modem is bridged or not. Maybe the modem does something different with its ethernet ports when bridged.
I've also tried putting on a "mediator" switch between the modem and the pfsense - both a dumb switch and a managed hp one.
-
Hmm, weird.
Is is possible the modem is doing something like using a VLAN upstream which it doesn't do in bridge mode? Or priority tagging maybe?
Though anything like that would apply to the laptop connected to the modem equally.
I guess at a minimum I'd compare the verbose interface status in each situation (ifconfig -vvv) and take some pcaps on WAN to see what is actually missing.
Steve
-
So an update.
I now booted OpenWRT on a usb stick, on the exact same hardware setup, and everything runs flawlessly.
I don't know - I've spend quite some time troubleshooting here, but it definately seems like some wierd issue with pfsense. For now, I'll settle for using something other than pfsense, though it would prefer not to. Thanks the input. @stephenw10
-
Hmm, does the parent NIC link exactly the same way in OpenWRT?
-
I did not note it down, and compare thoroughly, but in general yes - Same IP / subnet, gateway, mtu / linkspeed. I havent had the opportunity to reconnect pfsense and check the difference, as i have a narrow window in the evening, defined by how tired I am that night :)
I tried booting from an opnsense USB, and I had the exact same issue. I then broadened my search to include FreeBSD, and I found a few posts with similar issues for the first time. https://forum.opnsense.org/index.php?topic=11015.0 for instance.
