iperf3 on pfsense server (slower) different to client (faster) - Why?
-
Hi,
We've just setup a pfsense server 2.6.0 on a dedicated box, its got 4GB ram, a 256GB SSD and an i5 4570 running at 3.2Ghz and a four port NC365T pci-e network card. The network is a 1GB symmetric fibre. I normally get circa 910Mb/sec up and down.
The pfsense installed simply and quickly.
The pfsense box is siting behind an existing fibre modem and one of the ports is conncted to the fibre modem and is getting a DHCP address 192.168.1.120. I know it's not supposed to do double NAT but this was just to check that things are working before we turn the fibre router into a bridge.
We have a Mac laptop on 192.168.2.2 on another of the pfsenses intel ports.
We wanted to check things are working and it looks like it is fine. Just to repeat I know we shouldn't be doing double NAT.
If I do an iperf3 test on the pfsense server I get
Connecting to host iperf.par2.as49434.net, port 9237 [ 5] local 192.168.1.150 port 28800 connected to 193.177.162.41 port 9237 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 41.0 MBytes 344 Mbits/sec 0 7.34 MBytes [ 5] 1.00-2.00 sec 63.0 MBytes 529 Mbits/sec 0 7.34 MBytes [ 5] 2.00-3.00 sec 63.1 MBytes 529 Mbits/sec 0 7.34 MBytes [ 5] 3.00-4.00 sec 63.8 MBytes 535 Mbits/sec 0 7.34 MBytes [ 5] 4.00-5.00 sec 63.2 MBytes 530 Mbits/sec 0 7.34 MBytes [ 5] 5.00-6.00 sec 63.2 MBytes 530 Mbits/sec 0 7.34 MBytes [ 5] 6.00-7.00 sec 63.6 MBytes 534 Mbits/sec 0 7.34 MBytes [ 5] 7.00-8.00 sec 63.4 MBytes 532 Mbits/sec 0 7.34 MBytes [ 5] 8.00-9.00 sec 63.3 MBytes 531 Mbits/sec 0 7.34 MBytes [ 5] 9.00-10.00 sec 63.5 MBytes 533 Mbits/sec 0 7.34 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 611 MBytes 513 Mbits/sec 0 sender [ 5] 0.00-10.10 sec 611 MBytes 508 Mbits/sec receiverWhich is not very good.
If I do the same but on the Macbook that is connected to the pfsense server I get
willett@Robs-MacBook-Pro ~ % iperf3 -c iperf.par2.as49434.net -p 9237 Connecting to host iperf.par2.as49434.net, port 9237 [ 7] local 192.168.2.10 port 51766 connected to 193.177.162.41 port 9237 [ ID] Interval Transfer Bitrate [ 7] 0.00-1.00 sec 84.2 MBytes 706 Mbits/sec [ 7] 1.00-2.00 sec 112 MBytes 936 Mbits/sec [ 7] 2.00-3.00 sec 112 MBytes 936 Mbits/sec [ 7] 3.00-4.00 sec 112 MBytes 936 Mbits/sec [ 7] 4.00-5.00 sec 111 MBytes 935 Mbits/sec [ 7] 5.00-6.00 sec 112 MBytes 935 Mbits/sec [ 7] 6.00-7.00 sec 112 MBytes 935 Mbits/sec [ 7] 7.00-8.00 sec 111 MBytes 932 Mbits/sec [ 7] 8.00-9.00 sec 105 MBytes 884 Mbits/sec [ 7] 9.00-10.00 sec 111 MBytes 935 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 7] 0.00-10.00 sec 1.06 GBytes 907 Mbits/sec sender [ 7] 0.00-10.05 sec 1.06 GBytes 903 Mbits/sec receiver iperf Done.I don't understand why the Macbook which is behind the pfsense and is the only device on the LAN network gets 930Mb/sec. No issue but the pfsense server itself only gets 500Mb/sec.
Any explanation welcomed.
Thanjs
Rob
-
Mostly because pfSense is a router and not a very good TCP server.
Though with that hardware I might have expected to see 1G on a single stream. The latency could be an issue.
Try multiple parallel streams.
iperf to or from pfSense directly will always be lower than through it though.
Steve
-
Thanks for this. Now I know that pfsense and testing via iperf is better from a client, that's useful to know.
I ran 5 threads and got significanrtly better throughput on the pfsense server.
iperf3 -c iperf.par2.as49434.net -p 9237 -P 5 Connecting to host iperf.par2.as49434.net, port 9237 [ 5] local 192.168.1.150 port 24779 connected to 193.177.162.41 port 9237 [ 7] local 192.168.1.150 port 24780 connected to 193.177.162.41 port 9237 [ 9] local 192.168.1.150 port 24781 connected to 193.177.162.41 port 9237 [ 11] local 192.168.1.150 port 24782 connected to 193.177.162.41 port 9237 [ 13] local 192.168.1.150 port 24783 connected to 193.177.162.41 port 9237 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 10.2 MBytes 85.2 Mbits/sec 0 272 KBytes [ 7] 0.00-1.00 sec 10.1 MBytes 84.3 Mbits/sec 0 270 KBytes [ 9] 0.00-1.00 sec 4.84 MBytes 40.6 Mbits/sec 0 241 KBytes [ 11] 0.00-1.00 sec 11.0 MBytes 92.0 Mbits/sec 0 275 KBytes [ 13] 0.00-1.00 sec 6.17 MBytes 51.7 Mbits/sec 0 248 KBytes [SUM] 0.00-1.00 sec 42.2 MBytes 354 Mbits/sec 0 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 1.00-2.00 sec 14.5 MBytes 121 Mbits/sec 0 341 KBytes [ 7] 1.00-2.00 sec 14.4 MBytes 121 Mbits/sec 0 338 KBytes [ 9] 1.00-2.00 sec 7.13 MBytes 59.8 Mbits/sec 0 280 KBytes [ 11] 1.00-2.00 sec 14.8 MBytes 124 Mbits/sec 0 344 KBytes [ 13] 1.00-2.00 sec 7.91 MBytes 66.4 Mbits/sec 0 291 KBytes [SUM] 1.00-2.00 sec 58.7 MBytes 492 Mbits/sec 0 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 2.00-3.00 sec 17.7 MBytes 148 Mbits/sec 0 409 KBytes [ 7] 2.00-3.00 sec 17.8 MBytes 149 Mbits/sec 0 408 KBytes [ 9] 2.00-3.00 sec 7.92 MBytes 66.4 Mbits/sec 0 318 KBytes [ 11] 2.00-3.00 sec 18.1 MBytes 152 Mbits/sec 0 414 KBytes [ 13] 2.00-3.00 sec 9.18 MBytes 77.0 Mbits/sec 0 334 KBytes [SUM] 2.00-3.00 sec 70.7 MBytes 593 Mbits/sec 0 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 3.00-4.00 sec 20.8 MBytes 175 Mbits/sec 0 476 KBytes [ 7] 3.00-4.00 sec 21.0 MBytes 176 Mbits/sec 0 477 KBytes [ 9] 3.00-4.00 sec 9.32 MBytes 78.2 Mbits/sec 0 358 KBytes [ 11] 3.00-4.00 sec 21.3 MBytes 179 Mbits/sec 0 482 KBytes [ 13] 3.00-4.00 sec 10.6 MBytes 89.3 Mbits/sec 0 377 KBytes [SUM] 3.00-4.00 sec 83.1 MBytes 697 Mbits/sec 0 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 4.00-5.00 sec 23.9 MBytes 200 Mbits/sec 0 545 KBytes [ 7] 4.00-5.00 sec 24.1 MBytes 202 Mbits/sec 0 545 KBytes [ 9] 4.00-5.00 sec 10.2 MBytes 85.2 Mbits/sec 0 397 KBytes [ 11] 4.00-5.00 sec 24.6 MBytes 207 Mbits/sec 0 552 KBytes [ 13] 4.00-5.00 sec 11.9 MBytes 99.7 Mbits/sec 0 421 KBytes [SUM] 4.00-5.00 sec 94.6 MBytes 794 Mbits/sec 0 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 5.00-6.00 sec 26.5 MBytes 222 Mbits/sec 0 612 KBytes [ 7] 5.00-6.00 sec 26.6 MBytes 223 Mbits/sec 0 612 KBytes [ 9] 5.00-6.00 sec 11.2 MBytes 94.0 Mbits/sec 0 436 KBytes [ 11] 5.00-6.00 sec 27.8 MBytes 233 Mbits/sec 0 620 KBytes [ 13] 5.00-6.00 sec 13.0 MBytes 109 Mbits/sec 0 463 KBytes [SUM] 5.00-6.00 sec 105 MBytes 882 Mbits/sec 0 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 6.00-7.00 sec 27.3 MBytes 229 Mbits/sec 0 673 KBytes [ 7] 6.00-7.00 sec 27.5 MBytes 231 Mbits/sec 0 675 KBytes [ 9] 6.00-7.00 sec 12.2 MBytes 103 Mbits/sec 0 475 KBytes [ 11] 6.00-7.00 sec 30.5 MBytes 256 Mbits/sec 0 689 KBytes [ 13] 6.00-7.00 sec 14.2 MBytes 119 Mbits/sec 0 505 KBytes [SUM] 6.00-7.00 sec 112 MBytes 938 Mbits/sec 0 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 7.00-8.00 sec 27.3 MBytes 229 Mbits/sec 0 730 KBytes [ 7] 7.00-8.00 sec 27.5 MBytes 231 Mbits/sec 0 730 KBytes [ 9] 7.00-8.00 sec 12.5 MBytes 105 Mbits/sec 0 512 KBytes [ 11] 7.00-8.00 sec 30.7 MBytes 257 Mbits/sec 0 750 KBytes [ 13] 7.00-8.00 sec 14.6 MBytes 122 Mbits/sec 0 546 KBytes [SUM] 7.00-8.00 sec 113 MBytes 944 Mbits/sec 0 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 8.00-9.00 sec 19.5 MBytes 164 Mbits/sec 1 401 KBytes [ 7] 8.00-9.00 sec 20.4 MBytes 171 Mbits/sec 1 405 KBytes [ 9] 8.00-9.00 sec 8.76 MBytes 73.4 Mbits/sec 3 278 KBytes [ 11] 8.00-9.00 sec 21.6 MBytes 181 Mbits/sec 2 416 KBytes [ 13] 8.00-9.00 sec 10.3 MBytes 86.6 Mbits/sec 1 297 KBytes [SUM] 8.00-9.00 sec 80.7 MBytes 677 Mbits/sec 8 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 9.00-10.00 sec 20.4 MBytes 171 Mbits/sec 0 470 KBytes [ 7] 9.00-10.00 sec 20.9 MBytes 175 Mbits/sec 0 475 KBytes [ 9] 9.00-10.00 sec 8.04 MBytes 67.4 Mbits/sec 0 317 KBytes [ 11] 9.00-10.00 sec 21.2 MBytes 178 Mbits/sec 0 485 KBytes [ 13] 9.00-10.00 sec 9.43 MBytes 79.1 Mbits/sec 0 340 KBytes [SUM] 9.00-10.00 sec 80.0 MBytes 671 Mbits/sec 0 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 208 MBytes 174 Mbits/sec 1 sender [ 5] 0.00-10.10 sec 207 MBytes 172 Mbits/sec receiver [ 7] 0.00-10.00 sec 210 MBytes 176 Mbits/sec 1 sender [ 7] 0.00-10.10 sec 209 MBytes 174 Mbits/sec receiver [ 9] 0.00-10.00 sec 92.1 MBytes 77.3 Mbits/sec 3 sender [ 9] 0.00-10.10 sec 91.3 MBytes 75.8 Mbits/sec receiver [ 11] 0.00-10.00 sec 222 MBytes 186 Mbits/sec 2 sender [ 11] 0.00-10.10 sec 221 MBytes 184 Mbits/sec receiver [ 13] 0.00-10.00 sec 107 MBytes 90.0 Mbits/sec 1 sender [ 13] 0.00-10.10 sec 107 MBytes 88.5 Mbits/sec receiver [SUM] 0.00-10.00 sec 839 MBytes 704 Mbits/sec 8 sender [SUM] 0.00-10.10 sec 836 MBytes 694 Mbits/sec receiver iperf Done.Interestingly I didn't get much better throughput on the Macbook client with 5 threads.
Thanks for the update, appreciate the help,
Have a very good evening
Rob
-
@rwillett said in iperf3 on pfsense server (slower) different to client (faster) - Why?:
Interestingly I didn't get much better throughput on the Macbook client with 5 threads.
Well this is pretty maxed out for gig connection already.
7] 4.00-5.00 sec 111 MBytes 935 Mbits/sec [ 7] 5.00-6.00 sec 112 MBytes 935 Mbits/secSo no you prob wouldn't see much better than that ;)
