Issue with throughput with X710-DA2 and PFsense 2.6

[bawoodruff]

My 10Gb SFP+ fiber modules I bought for the ixl interface came in today (weren't supposed to be in till tomorrow). I switched the LAN interface out with the Gbit SFP module that was in there and I'm able to get about 4.5Gbps between the LAN Client and PFsense using multiple streams running:iperf3 -c 192.168.1.1 -P 20 but still only getting 300ish when I try a normal speed test from PFsense to the internet or the LAN Client to the internet.

WAN: igb 1Gbit interface
LAN: ixl 10Gbit interface

Client running iperf3 -> 10Gb LAN->PFsense (iperf3 server) = 4.5ish Gbps
ssh -> PFsense -> Speedtest-cli -> WAN -> Internet = 300ish Mb/s
client running speedtest-> 10Gb LAN -> PFsense -> WAN -> Internet = 300ish Mb/s

The 300Mbps speed was basically the same whether the WAN was on a igb 1Gbit interface or the other ixl interface with a 1Gbit SFP module

I've tried multiple speed test sites from the client machine during testing and they all show basically the same speed.

[stephenw10]

Hmm, how is the WAN actually connected here?

Do you see any collisions or errors on the interfaces?

We have seen ISPs that provide a reduced rate profile to devices other than what is registered, which could still be the UDM. Have you tried spoofing the MAC address on WAN to match the UDM?

Steve

[bawoodruff]

Cable Co. ->Docsis3.1 Modem ->Cat6->igb Interface on PFsense box

ISP Connection is regulated by MAC address filtering of the Modem itself.

I actually already tried cloning the UDMP's MAC on the PFsense box. I saw no change in speed. Just grabbed the IP address last assigned to the UDMP.

[stephenw10]

And can I assume a client connected directly to the modem gets a public IP and can see the full bandwidth?

[bawoodruff]

yup

[bawoodruff]

Wonder if I backup my current config, and get re-install would make a difference.... Seen some posts involving throughout issues being resolved by this. although this is a rather new install to begin with tbh.

[stephenw10]

It's an easy enough test but I wouldn't expect it to make any difference at all.

I would want to test against a WAN side local iperf server to confirm that is capable of passing 1G.
Testing against a remote speedtest server there are a lot of variables.

Steve

[bawoodruff]

Off to try and locate a decent (local to me) iperf server, unless someone knows of any in the Indianapolis area, lol

[bawoodruff]

While SSH'ed into the pfSense box and running several tests using"

iperf3 -c speedtest.serverius.net -p 5002 -P 10 -4 -R

shows a bitrate of 280ish Mbps

With the processor cores showing about 20% utilization

[stephenw10]

When I say local I mean actually in your location. When I do this I just use another pfSense box to server as a gateway for the unit under test and run the iperf server on that. That's not ideal but as long as the gateway device is sufficiently powerful it will expose any restrictions on the test device.

Steve

[bawoodruff]

So, I removed all the past settings I had in place and out these in place and reboot.

\boot\loader.conf.local

legal.intel_iwi.license_ack=1
legal.intel_ipw.license_ack=1
net.link.ifqmaxlen=2048
net.isr.defaultqlimit=2048
net.inet.tcp.soreceive_stream=1
net.inet.tcp.syncache.hashsize=1024
net.inet.tcp.syncache.bucketlimit=100
net.pf.source_nodes_hashsize=1048576

System Tunables

ev.igb.0.eee_control=0
dev.igb.1.eee_control=0
dev.igb.2.eee_control=0
dev.igb.3.eee_control=0
dev.igb.0.fc=0
dev.igb.1.fc=0
dev.igb.2.fc=0
dev.igb.3.fc=0
net.inet.ip.redirect=0
net.inet6.ip6.redirect=0
net.route.netisr_maxqlen=2048
net.inet.ip.intr_queue_maxlen=2048
net.inet.udp.maxdgram=16384
net.inet.tcp.syncookies=0
hw.intr_storm_threshold=0
kern.ipc.soacceptqueue=2048

And with FQ_Codel queues and Suricata disabled, I now seem to be hitting 900+Mbps
When I run :iperf3 -c proof.ovh.net -p 5201 -P 40 -R
Had to increase the number of streams, whereas before anything past 10 made no difference.

And with FQ_Codel and Suricata enabled, I now seem to be hitting 850+Mbps

[stephenw10]

Is that testing from pfSense directly?

You usually don't see any improvement with parallel streams beyond the number of NIC queues. iperf itself is still (deliberately) single threaded so it hits a single core limit there.

Steve

[bawoodruff]

Your right, all I was doing was increasing the number of retries generated beyond a certain number of streams. So I lowered the number of streams till the retry counts were at 0.

This is running on a LAN side Linux client to the internet with Suricata and FQ_Codel queues disabled: iperf3 -c proof.ovh.net -p 5201 -P 6 -R -t 30

The throughput is slightly lower today, but I'm attributing that to slightly higher ISP congestion considering the time of day, relative to when I last tested.

[stephenw10]

Hmm, well that looks pretty good. It at least proves the NIC is good.

I believe speedtest uses 4 connections though it could vary by server. You might test at 2 and 4 parallel streams in iperf to get a comparable result.

If you are able to test against your own server locally that will also be good test.

Steve