Slow upload speed through pfSense to NAS, but download is sometimes faster (according to iperf)
-
Hello everyone!
I am having a strange issue with the upload bitrate to my TrueNAS server from the WAN-side of my pfSense firewall. Note that the WAN-side of my firewall is actually just the private LAN for my home network. This is intentional because I didn't want to disrupt other people on the network while I was labbing. See attached image.
.All cables are cat 7 or cat 8. The cable from pfSense to the switch is an SFP+ Module (4PCS 1000BASE-T Gigabit SFP to RJ45 Copper Ethernet Modular Transceiver for Cisco,Meraki,Ubiquiti,D/TP Link,Supermicro,Netgear,Broadcom, 1.25G SFP-T CAT5E/CAT6 up to 100m Mikrotik S-RJ01
.pfSense Server Specs (CPU averages 11% usage, and Memory averages 2%):
Dell Optiplex 7050 SFF Desktop PC
Intel i7-7700 4-Cores 3.60GHz
32GB DDR4
1TB SSD
Dual-Port PCIe X4 Gigabit Network Card 1000M PCI Express Ethernet Adapter with Intel 82576
10Gb PCI-E Network Card NIC Compatible for Intel X540-T2, Dual RJ45 Copper PortRelevant firewall rules
I understand that the first rule makes the second rule irrelevant.
Running
ifconfig -m ix1(WAN interface) indicates the following:mtu 1500 media: Ethernet autoselect (1000baseT <full-duplex,rxpause,txpause>) supported media: media autoselect media 100baseTX media 1000baseT media 10Gbase-TAnd:
ifconfig -m ix0(interface from pfSense to the Switch) indicates the following:mtu 1500 media: Ethernet autoselect (10Gbase-T <full-duplex>) supported media: media autoselect media 100baseTX media 1000baseT media 10Gbase-TThe problem:
My Macbook Pro (Wi-Fi) and both Windows machines (Hardwired, Wifi - WiFi 6E AX210 PCIe WiFi Card) experience abysmally slow transfer speeds when downloading from the NAS (through pfSense). I understand that iperf3 is garbo for Windows, however the reason I sought it out was because I noticed I was no longer saturating my gigabit nic (hardwired) or my Wi-Fi adapters (about 300-400 mbps) when uploading (or downloading) to/from the server.However, when I am uploading/downloading between VMs on either ESXi host, I am able to fully saturate the 10gbe connection (in my SSD pool, 2x2 SSD mirror). Therefore, this should not be a spec issue on the NAS side.
When I hardwire my MBP to the switch, I can saturate the 1gb network adapter I'm using both ways. I would expect this to be the case if I were to hardwire a Windows machine onto the switch, LAN-side of pfSense. To me, this indicates that the switch is capable of the 1gbe connections (which is known, because I can saturate 10gbe between ESXI guests).
I have gigabit fiber, and can nearly saturate about 85-90% of every nic (Hardwired or Wi-Fi), which to me, indicates that all of cables should be fine. I have also replaced cables between the router, one PC, and pfSense.
It is just when I am on the WAN-side of the firewall (hardwired or on Wi-Fi), my upload speeds to the server are abysmally slow (MacOS and Windows). The downloads speeds are actually about what I'd expect. More details below.
Things I've tried:
My home router is not capable of 9000 MTU (Jumbo Frames). However, all hardware and hosts LAN-side of pfSense were configured with Jumbo Frames originally when I discovered this issue. I've since turned that back to 1500, and nothing has changed. I've verified on the hardwired Windows host that full-duplex is being used instead of auto-negotiation, which changed nothing. Interestingly, when I disable the Windows Firewall, sometimes the upload speeds seem to fully saturate the nic--only for a short period of time, and return to the abysmally slow speed. Adding Firewall Rule to allow any to the NAS does not change things. Plus, this does not explain why the speeds from my MPB are about the same as the Windows hosts (on Wi-Fi or hard-wired WAN-side of pfSense).
Forcing the WAN interface to negotiate
1000baseTdoes not resolve the issue.I also happen to have OpenVPN installed in pfSense. Regardless if I am hard-wired or on wi-fi on WAN-side device and connected to OpenVPN, the bitrates exhibit the same behavior.
I am truly at a loss, and don't know what else to look at. I suspect something in pfSense or the Mikrotik Switch (linked to pfSense) is causing the bottleneck, but I don't know what? All ESXi guests use VLAN IDs, but the ESXi and TrueNAS hosts do not, and are on a LAN interface connected to one of the physical 10gbe RJ45 ports.
Speed stats below:
Download from Windows on Wi-Fi (I expect this to be about the same as the following report, ~300-400 Mbits/sec)
.\iperf3.exe -c 10.0.1.x Connecting to host 10.0.1.x, port 5201 [ 4] local 192.168.1.x port 54823 connected to 10.0.1.x port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.01 sec 15.8 MBytes 131 Mbits/sec [ 4] 1.01-2.00 sec 14.4 MBytes 121 Mbits/sec [ 4] 2.00-3.00 sec 15.9 MBytes 133 Mbits/sec [ 4] 3.00-4.00 sec 15.4 MBytes 129 Mbits/sec [ 4] 4.00-5.01 sec 15.2 MBytes 128 Mbits/sec [ 4] 5.01-6.01 sec 16.2 MBytes 136 Mbits/sec [ 4] 6.01-7.01 sec 14.5 MBytes 121 Mbits/sec [ 4] 7.01-8.01 sec 14.8 MBytes 124 Mbits/sec [ 4] 8.01-9.01 sec 15.1 MBytes 127 Mbits/sec [ 4] 9.01-10.00 sec 16.1 MBytes 136 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 153 MBytes 129 Mbits/sec sender [ 4] 0.00-10.00 sec 153 MBytes 129 Mbits/sec receiver iperf Done.Reverse on Windows on Wi-Fi
.\iperf3.exe -c 10.0.1.x -R Connecting to host 10.0.1.x, port 5201 Reverse mode, remote host 10.0.1.x is sending [ 4] local 192.168.1.x port 54830 connected to 10.0.1.x port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 30.9 MBytes 259 Mbits/sec [ 4] 1.00-2.00 sec 35.4 MBytes 297 Mbits/sec [ 4] 2.00-3.00 sec 33.1 MBytes 278 Mbits/sec [ 4] 3.00-4.00 sec 36.3 MBytes 304 Mbits/sec [ 4] 4.00-5.00 sec 32.8 MBytes 274 Mbits/sec [ 4] 5.00-6.00 sec 32.0 MBytes 269 Mbits/sec [ 4] 6.00-7.00 sec 31.5 MBytes 265 Mbits/sec [ 4] 7.00-8.00 sec 31.1 MBytes 261 Mbits/sec [ 4] 8.00-9.00 sec 32.2 MBytes 270 Mbits/sec [ 4] 9.00-10.00 sec 33.1 MBytes 277 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 329 MBytes 276 Mbits/sec 0 sender [ 4] 0.00-10.00 sec 329 MBytes 276 Mbits/sec receiver iperf Done.Download on MBP on Wi-Fi:
iperf3 -c 10.0.1.x Connecting to host 10.0.1.x, port 5201 [ 5] local 192.168.1.x port 61176 connected to 10.0.1.x port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 3.99 MBytes 33.4 Mbits/sec [ 5] 1.00-2.00 sec 4.22 MBytes 35.4 Mbits/sec [ 5] 2.00-3.00 sec 8.70 MBytes 73.2 Mbits/sec [ 5] 3.00-4.00 sec 13.7 MBytes 115 Mbits/sec [ 5] 4.00-5.00 sec 13.5 MBytes 113 Mbits/sec [ 5] 5.00-6.00 sec 13.3 MBytes 111 Mbits/sec [ 5] 6.00-7.00 sec 13.3 MBytes 111 Mbits/sec [ 5] 7.00-8.00 sec 14.0 MBytes 117 Mbits/sec [ 5] 8.00-9.00 sec 13.5 MBytes 114 Mbits/sec [ 5] 9.00-10.00 sec 13.4 MBytes 112 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 112 MBytes 93.6 Mbits/sec sender [ 5] 0.00-10.02 sec 112 MBytes 93.3 Mbits/sec receiver iperf Done.Reverse on MBP:
iperf3 -c 10.0.1.x -R Connecting to host 10.0.1.x, port 5201 Reverse mode, remote host 10.0.1.x is sending [ 5] local 192.168.1.x port 61179 connected to 10.0.1.x port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 20.6 MBytes 173 Mbits/sec [ 5] 1.00-2.00 sec 34.1 MBytes 286 Mbits/sec [ 5] 2.00-3.00 sec 43.5 MBytes 365 Mbits/sec [ 5] 3.00-4.00 sec 40.4 MBytes 339 Mbits/sec [ 5] 4.00-5.00 sec 36.4 MBytes 305 Mbits/sec [ 5] 5.00-6.00 sec 28.8 MBytes 241 Mbits/sec [ 5] 6.00-7.00 sec 12.0 MBytes 101 Mbits/sec [ 5] 7.00-8.00 sec 13.3 MBytes 112 Mbits/sec [ 5] 8.00-9.00 sec 10.9 MBytes 91.2 Mbits/sec [ 5] 9.00-10.00 sec 8.55 MBytes 71.8 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.01 sec 250 MBytes 209 Mbits/sec 65 sender [ 5] 0.00-10.00 sec 249 MBytes 209 Mbits/sec receiverWhen I hardwire the MBP on the same switch as the NAS, I get:
iperf3 -c 10.0.1.x Connecting to host 10.0.1.x, port 5201 [ 5] local 10.0.1.x port 61210 connected to 10.0.1.x port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 110 MBytes 925 Mbits/sec [ 5] 1.00-2.00 sec 112 MBytes 942 Mbits/sec [ 5] 2.00-3.00 sec 112 MBytes 942 Mbits/sec [ 5] 3.00-4.00 sec 112 MBytes 941 Mbits/sec [ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec [ 5] 5.00-6.00 sec 112 MBytes 942 Mbits/sec [ 5] 6.00-7.00 sec 112 MBytes 941 Mbits/sec [ 5] 7.00-8.00 sec 112 MBytes 941 Mbits/sec [ 5] 8.00-9.00 sec 112 MBytes 941 Mbits/sec [ 5] 9.00-10.00 sec 112 MBytes 941 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 1.09 GBytes 940 Mbits/sec sender [ 5] 0.00-10.00 sec 1.09 GBytes 939 Mbits/sec receiver iperf Done.And in reverse:
iperf3 -c 10.0.1.x -R Connecting to host 10.0.1.x, port 5201 Reverse mode, remote host 10.0.1.x is sending [ 5] local 10.0.1.x port 61214 connected to 10.0.1.x port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 108 MBytes 910 Mbits/sec [ 5] 1.00-2.00 sec 110 MBytes 922 Mbits/sec [ 5] 2.00-3.00 sec 110 MBytes 922 Mbits/sec [ 5] 3.00-4.00 sec 110 MBytes 919 Mbits/sec [ 5] 4.00-5.00 sec 109 MBytes 918 Mbits/sec [ 5] 5.00-6.00 sec 110 MBytes 919 Mbits/sec [ 5] 6.00-7.00 sec 109 MBytes 917 Mbits/sec [ 5] 7.00-8.00 sec 109 MBytes 916 Mbits/sec [ 5] 8.00-9.00 sec 110 MBytes 921 Mbits/sec [ 5] 9.00-10.00 sec 109 MBytes 916 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.07 GBytes 918 Mbits/sec 17872 sender [ 5] 0.00-10.00 sec 1.07 GBytes 918 Mbits/sec receiver iperf Done.I don't have an iperf3 report from the hard-wired Windows machine, but it performs similarly (download), but in reverse it saturates the interface at about 900 Mbits/sec.
Any help would be deeply appreciated. Thank you!
-
I'll add that I ran an iperf test between Windows on Wi-Fi (server) and my MBP (client) on Wi-Fi seems to indicate that the router between them might be the issue...
iperf3 -c 192.168.1.x Connecting to host 192.168.1.x, port 5201 [ 5] local 192.168.1.x port 61572 connected to 192.168.1.x port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 3.64 MBytes 30.4 Mbits/sec [ 5] 1.00-2.00 sec 13.3 MBytes 112 Mbits/sec [ 5] 2.00-3.00 sec 13.1 MBytes 110 Mbits/sec [ 5] 3.00-4.00 sec 15.1 MBytes 127 Mbits/sec [ 5] 4.00-5.00 sec 14.4 MBytes 120 Mbits/sec [ 5] 5.00-6.00 sec 15.5 MBytes 130 Mbits/sec [ 5] 6.00-7.00 sec 16.5 MBytes 138 Mbits/sec [ 5] 7.00-8.00 sec 16.2 MBytes 136 Mbits/sec [ 5] 8.00-9.00 sec 13.3 MBytes 112 Mbits/sec [ 5] 9.00-10.00 sec 13.6 MBytes 114 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 135 MBytes 113 Mbits/sec sender [ 5] 0.00-10.00 sec 135 MBytes 113 Mbits/sec receiver iperf Done. iperf3 -c 192.168.1.x -R Connecting to host 192.168.1.x, port 5201 Reverse mode, remote host 192.168.1.x is sending [ 5] local 192.168.1.x port 61578 connected to 192.168.1.x port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.01 sec 3.00 MBytes 25.0 Mbits/sec [ 5] 1.01-2.00 sec 2.62 MBytes 22.0 Mbits/sec [ 5] 2.00-3.00 sec 3.12 MBytes 26.2 Mbits/sec [ 5] 3.00-4.00 sec 13.6 MBytes 115 Mbits/sec [ 5] 4.00-5.00 sec 17.0 MBytes 142 Mbits/sec [ 5] 5.00-6.00 sec 13.5 MBytes 113 Mbits/sec [ 5] 6.00-7.00 sec 8.50 MBytes 71.3 Mbits/sec [ 5] 7.00-8.00 sec 8.62 MBytes 72.4 Mbits/sec [ 5] 8.00-9.00 sec 8.63 MBytes 72.0 Mbits/sec [ 5] 9.00-10.01 sec 6.59 MBytes 55.3 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.01 sec 85.4 MBytes 71.6 Mbits/sec sender [ 5] 0.00-10.01 sec 85.2 MBytes 71.4 Mbits/sec receiverIf I hardwire the MBP behind the firewall and run the test again, I get the following:
iperf3 -c 192.168.1.x Connecting to host 192.168.1.x, port 5201 [ 5] local 192.168.1.x port 61589 connected to 192.168.1.x port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 12.3 MBytes 103 Mbits/sec [ 5] 1.00-2.00 sec 16.6 MBytes 139 Mbits/sec [ 5] 2.00-3.00 sec 15.2 MBytes 127 Mbits/sec [ 5] 3.00-4.01 sec 16.3 MBytes 137 Mbits/sec [ 5] 4.01-5.00 sec 14.0 MBytes 118 Mbits/sec [ 5] 5.00-6.00 sec 14.0 MBytes 118 Mbits/sec [ 5] 6.00-7.00 sec 5.69 MBytes 47.7 Mbits/sec [ 5] 7.00-8.00 sec 3.85 MBytes 32.3 Mbits/sec [ 5] 8.00-9.00 sec 4.08 MBytes 34.2 Mbits/sec [ 5] 9.00-10.00 sec 8.47 MBytes 71.3 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 111 MBytes 92.7 Mbits/sec sender [ 5] 0.00-10.00 sec 111 MBytes 92.7 Mbits/sec receiver iperf Done. iperf3 -c 192.168.1.x -R Connecting to host 192.168.1.x, port 5201 Reverse mode, remote host 192.168.1.x is sending [ 5] local 192.168.1.x port 61597 connected to 192.168.1.x port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 15.4 MBytes 129 Mbits/sec [ 5] 1.00-2.00 sec 10.4 MBytes 87.3 Mbits/sec [ 5] 2.00-3.00 sec 10.1 MBytes 84.7 Mbits/sec [ 5] 3.00-4.00 sec 8.79 MBytes 73.8 Mbits/sec [ 5] 4.00-5.00 sec 11.5 MBytes 96.0 Mbits/sec [ 5] 5.00-6.01 sec 9.87 MBytes 82.7 Mbits/sec [ 5] 6.01-7.00 sec 3.22 MBytes 27.1 Mbits/sec [ 5] 7.00-8.00 sec 2.68 MBytes 22.5 Mbits/sec [ 5] 8.00-9.00 sec 7.12 MBytes 59.8 Mbits/sec [ 5] 9.00-10.00 sec 20.0 MBytes 167 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 99.2 MBytes 83.3 Mbits/sec sender [ 5] 0.00-10.00 sec 99.1 MBytes 83.1 Mbits/sec receiver iperf Done.If I hardwire the MBP in front of the firewall (same side as the Windows on Wi-Fi), I'm actually getting the download speed I'd expect (on Wi-Fi anyways).
iperf3 -c 192.168.1.x Connecting to host 192.168.1.x, port 5201 [ 5] local 192.168.1.x port 61658 connected to 192.168.1.x port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 30.9 MBytes 259 Mbits/sec [ 5] 1.00-2.00 sec 35.7 MBytes 300 Mbits/sec [ 5] 2.00-3.00 sec 34.8 MBytes 292 Mbits/sec [ 5] 3.00-4.00 sec 33.9 MBytes 284 Mbits/sec [ 5] 4.00-5.00 sec 33.0 MBytes 277 Mbits/sec [ 5] 5.00-6.00 sec 38.7 MBytes 324 Mbits/sec [ 5] 6.00-7.00 sec 38.3 MBytes 321 Mbits/sec [ 5] 7.00-8.00 sec 30.6 MBytes 257 Mbits/sec [ 5] 8.00-9.00 sec 33.7 MBytes 283 Mbits/sec [ 5] 9.00-10.00 sec 31.3 MBytes 263 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 341 MBytes 286 Mbits/sec sender [ 5] 0.00-10.00 sec 341 MBytes 286 Mbits/sec receiver iperf Done. iperf3 -c 192.168.1.x -R Connecting to host 192.168.1.x, port 5201 Reverse mode, remote host 192.168.1.x is sending [ 5] local 192.168.1.x port 61662 connected to 192.168.1.x port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 23.0 MBytes 193 Mbits/sec [ 5] 1.00-2.00 sec 16.0 MBytes 134 Mbits/sec [ 5] 2.00-3.00 sec 19.8 MBytes 166 Mbits/sec [ 5] 3.00-4.00 sec 19.3 MBytes 161 Mbits/sec [ 5] 4.00-5.00 sec 23.4 MBytes 197 Mbits/sec [ 5] 5.00-6.00 sec 23.7 MBytes 199 Mbits/sec [ 5] 6.00-7.00 sec 26.1 MBytes 218 Mbits/sec [ 5] 7.00-8.00 sec 14.0 MBytes 118 Mbits/sec [ 5] 8.00-9.00 sec 20.1 MBytes 169 Mbits/sec [ 5] 9.00-10.00 sec 18.8 MBytes 157 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 204 MBytes 171 Mbits/sec sender [ 5] 0.00-10.00 sec 204 MBytes 171 Mbits/sec receiver iperf Done.I should try these last two tests against the other hardwired windows machine, but it's not my machine so that'll have to be on hold at the moment.
-
It certainly could be the WAN side router if everything is going through it.
Try putting a switch between pfSense and that router and then testing from a hardwired client connected to that switch.
The specs on that pfSense machine should very easily pass that traffic.
That other router probably has a built in switch but depending on how it's how it's configured may or may not be routing/filtering the traffic.
Steve
-
@stephenw10 oh great idea for ruling out the WAN-side router, I do have another Mikrotik switch laying around somewhere. I'll report back, might take me a couple of days.
-
I can confirm that the switches on the router mr8300 (or the router hardware itself) appear to be what is the cause behind the slowness when hardwired.
