Upgraded from 1Gb nic to 10gb nic, now can't push more than 500Mbps over TCP
-
Hello,
had a working pfsense 2.7.2 box with a 4-port intel IGB nic. (4x gig). Worked great!
In preparation to upgrade to a 10gig switch, I decided to upgrade the NIC in the pfsense box. Pulled out the intel IGB, replaced with a Chelsio T520-BT.
Pretty quickly noticed I can't push more than about 625Mbps through it on TCP. Whaaaa? Before I could push 900Mbps through pfsense with tcp with the intel card! (172.18.0.1 is pfsense, 172.18.0.103 is me.)
iperf3 -c 172.18.0.1
Connecting to host 172.18.0.1, port 5201
[ 4] local 172.18.0.103 port 51368 connected to 172.18.0.1 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 74.4 MBytes 624 Mbits/sec
[ 4] 1.00-2.00 sec 74.5 MBytes 625 Mbits/sec
[ 4] 2.00-3.00 sec 74.1 MBytes 622 Mbits/sec
[ 4] 3.00-4.00 sec 73.9 MBytes 620 Mbits/sec
[ 4] 4.00-5.00 sec 74.1 MBytes 622 Mbits/sec
[ 4] 5.00-6.00 sec 73.0 MBytes 612 Mbits/sec
[ 4] 6.00-7.00 sec 74.4 MBytes 624 Mbits/sec
[ 4] 7.00-8.00 sec 74.6 MBytes 626 Mbits/sec
[ 4] 8.00-9.00 sec 74.5 MBytes 625 Mbits/sec
[ 4] 9.00-10.00 sec 74.6 MBytes 626 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 742 MBytes 622 Mbits/sec sender
[ 4] 0.00-10.00 sec 742 MBytes 622 Mbits/sec receiver
iperf Done.But if I use UDP, I get 958Mbps, same host same card!
iperf3 -c 172.18.0.1 -b 0 -u
Connecting to host 172.18.0.1, port 5201
[ 4] local 172.18.0.103 port 59716 connected to 172.18.0.1 port 5201
[ ID] Interval Transfer Bandwidth Total Datagrams
[ 4] 0.00-1.00 sec 114 MBytes 959 Mbits/sec 14650
[ 4] 1.00-2.00 sec 114 MBytes 958 Mbits/sec 14620
[ 4] 2.00-3.00 sec 114 MBytes 958 Mbits/sec 14620
[ 4] 3.00-4.00 sec 114 MBytes 958 Mbits/sec 14620
[ 4] 4.00-5.00 sec 114 MBytes 958 Mbits/sec 14630
[ 4] 5.00-6.00 sec 114 MBytes 958 Mbits/sec 14620
[ 4] 6.00-7.00 sec 114 MBytes 958 Mbits/sec 14620
[ 4] 7.00-8.00 sec 114 MBytes 958 Mbits/sec 14620
[ 4] 8.00-9.00 sec 114 MBytes 958 Mbits/sec 14630
[ 4] 9.00-10.00 sec 114 MBytes 958 Mbits/sec 14620
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 4] 0.00-10.00 sec 1.12 GBytes 958 Mbits/sec 0.000 ms 0/0 (0%)
[ 4] Sent 0 datagrams
iperf Done.So clearly the links are up and working fine (still at 1Gbps), but something has gone very awry in pfsense itself.
What am I missing here?
I've tried doing multiple concurrent tcp iperf connections... No change, everything just scales down so you get the same throughput of 650Mbps.
I've also done an iperf to a different system on the same segment (so it doesn't hit pfsense), and it's more than happy to push > 900 Mbits over tcp:
iperf3 -c 172.18.0.50
Connecting to host 172.18.0.50, port 5201
[ 4] local 172.18.0.103 port 51395 connected to 172.18.0.50 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 108 MBytes 908 Mbits/sec
[ 4] 1.00-2.00 sec 109 MBytes 912 Mbits/sec
[ 4] 2.00-3.00 sec 109 MBytes 913 Mbits/sec
[ 4] 3.00-4.00 sec 109 MBytes 912 Mbits/sec
[ 4] 4.00-5.00 sec 109 MBytes 913 Mbits/sec
[ 4] 5.00-6.00 sec 109 MBytes 912 Mbits/sec
[ 4] 6.00-7.00 sec 109 MBytes 912 Mbits/sec
[ 4] 7.00-8.00 sec 109 MBytes 913 Mbits/sec
[ 4] 8.00-9.00 sec 109 MBytes 913 Mbits/sec
[ 4] 9.00-10.00 sec 109 MBytes 913 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 1.06 GBytes 912 Mbits/sec sender
[ 4] 0.00-10.00 sec 1.06 GBytes 912 Mbits/sec receiver
iperf Done.
Tried turning on/off all the flags in system->advanced->networking one at a time and rebooting after each (disable/enable hardware checksum offload, disable/enable hardware tcp segmentation offload, disable/enable large receive offload, disable/enable altq of hn NICs), none of which made any difference whatsoever.
So I'm stumped and throwing up an SOS. Anyone know what I'm missing here? Previously I've run this card in a linux box all day pushing 8Gbps+ so I know the card is fine and shouldn't even be breathing heavy at a sub-1Gbps load. So I'm looking for suggestions on what I'm doing wrong here or missing.
Thanks in advance!
-
have you run through all of these troubleshooting/tuning items yet?
if not, start there. there's a lot more to say about what could be going on and what else you can do or try if all that fails.
-
@thekorn said in Upgraded from 1Gb nic to 10gb nic, now can't push more than 500Mbps over TCP:
Chelsio T520-B
There are various reports regarding bad performance with 10G chelsios especially running at 1G.
Google chelsio freebsd drivers.I would suggest switch to 10g and measure again.
-
@thekorn said in Upgraded from 1Gb nic to 10gb nic, now can't push more than 500Mbps over TCP:
through pfsense with tcp with the intel card! (172.18.0.1 is pfsense, 172.18.0.103 is me.)
That is not a good test of what a firewall/router can do - its not meant to be a server or a client.. Test actually "through" pfsense to to or from it. With that iperf command pfsense is sending traffic.. That it has to create and put on the wire, etc. That is way different than just passing the traffic through it.
-
@johnpoz Testing traffic through it gets dramatically worse results. (i.e. iperf through the pfsense box to another box on a different but directly connected network):
iperf3 -c 172.18.25.25
Connecting to host 172.18.25.25, port 5201
[ 4] local 172.18.0.103 port 52568 connected to 172.18.25.25 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 54.1 MBytes 453 Mbits/sec
[ 4] 1.00-2.00 sec 56.6 MBytes 475 Mbits/sec
[ 4] 2.00-3.00 sec 56.5 MBytes 475 Mbits/sec
[ 4] 3.00-4.00 sec 56.6 MBytes 475 Mbits/sec
[ 4] 4.00-5.00 sec 56.1 MBytes 471 Mbits/sec
[ 4] 5.00-6.00 sec 56.6 MBytes 475 Mbits/sec
[ 4] 6.00-7.00 sec 56.6 MBytes 475 Mbits/sec
[ 4] 7.00-8.00 sec 56.6 MBytes 475 Mbits/sec
[ 4] 8.00-9.00 sec 56.6 MBytes 474 Mbits/sec
[ 4] 9.00-10.00 sec 56.4 MBytes 473 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 563 MBytes 472 Mbits/sec sender
[ 4] 0.00-10.00 sec 563 MBytes 472 Mbits/sec receiver
iperf Done.UDP, however, remains steller:
iperf3 -c 172.18.25.25 -b 0 -u
Connecting to host 172.18.25.25, port 5201
[ 4] local 172.18.0.103 port 64192 connected to 172.18.25.25 port 5201
[ ID] Interval Transfer Bandwidth Total Datagrams
[ 4] 0.00-1.00 sec 114 MBytes 959 Mbits/sec 14640
[ 4] 1.00-2.00 sec 114 MBytes 958 Mbits/sec 14620
[ 4] 2.00-3.00 sec 114 MBytes 958 Mbits/sec 14620
[ 4] 3.00-4.00 sec 114 MBytes 958 Mbits/sec 14630
[ 4] 4.00-5.00 sec 114 MBytes 958 Mbits/sec 14620
[ 4] 5.00-6.00 sec 114 MBytes 958 Mbits/sec 14620
[ 4] 6.00-7.00 sec 114 MBytes 958 Mbits/sec 14630
[ 4] 7.00-8.00 sec 114 MBytes 958 Mbits/sec 14620
[ 4] 8.00-9.00 sec 114 MBytes 958 Mbits/sec 14620
[ 4] 9.00-10.00 sec 114 MBytes 958 Mbits/sec 14630
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 4] 0.00-10.00 sec 1.12 GBytes 958 Mbits/sec 0.146 ms 75222/145635 (52%)
[ 4] Sent 145635 datagrams
iperf Done. -
@cyberconsultants I had not! That's exactly the point in a direction I was looking for, thank you!
Adding these as loader tuneables seems to have fixed me right up:
hw.cxgbe.niccaps_allowed="1"
hw.cxgbe.toecaps_allowed="0"
hw.cxgbe.rdmacaps_allowed="0"
hw.cxgbe.iscsicaps_allowed="0"
hw.cxgbe.fcoecaps_allowed="0"iperf3 -c 172.18.25.25 --parallel 2
Connecting to host 172.18.25.25, port 5201
[ 4] local 172.18.0.103 port 52800 connected to 172.18.25.25 port 5201
[ 6] local 172.18.0.103 port 52801 connected to 172.18.25.25 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 53.9 MBytes 452 Mbits/sec
[ 6] 0.00-1.00 sec 51.2 MBytes 430 Mbits/sec
[SUM] 0.00-1.00 sec 105 MBytes 882 Mbits/sec
[ 4] 1.00-2.00 sec 53.5 MBytes 449 Mbits/sec
[ 6] 1.00-2.00 sec 51.2 MBytes 430 Mbits/sec
[SUM] 1.00-2.00 sec 105 MBytes 878 Mbits/sec(etc.)
(I added the parallel connections to squeak out a bit more; without it it's running around 790-ish Mb, which I'm calling good enough.)
And my speed tests to the internet are back in the comfortable 940-ish Mbit range (1G/1G), so I'm calling it fixed, or at least as fixed as it needs to be until my new switch shows up.
Thanks again!