Poor speeds with Chelsio T420-CR 10gb NIC
-
Hello,
I have a Chelsio T420-CR card in the Intel C2758 (Supermicro A1SRi-2758f ) based pfsense system. The router and my computer are connected to an Ubiquiti ES-16-XG switch. If i do an iperf to the router from the computer, i get speed even less than 1g. I am using multiple vlans on the cxgb interface.
iperf3 -c 192.168.1.28 Connecting to host 192.168.1.28, port 5201 [ 4] local 192.168.1.1 port 18300 connected to 192.168.1.28 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 101 MBytes 845 Mbits/sec 0 249 MBytes [ 4] 1.00-2.00 sec 109 MBytes 921 Mbits/sec 0 249 MBytes [ 4] 2.00-3.41 sec 82.3 MBytes 489 Mbits/sec 0 249 MBytes [ 4] 3.41-4.01 sec 65.8 MBytes 930 Mbits/sec 0 249 MBytes [ 4] 4.01-5.00 sec 111 MBytes 931 Mbits/sec 0 249 MBytes [ 4] 5.00-6.00 sec 110 MBytes 923 Mbits/sec 0 249 MBytes [ 4] 6.00-7.00 sec 111 MBytes 929 Mbits/sec 0 249 MBytes [ 4] 7.00-8.00 sec 111 MBytes 933 Mbits/sec 0 249 MBytes [ 4] 8.00-9.00 sec 110 MBytes 928 Mbits/sec 0 249 MBytes [ 4] 9.00-10.00 sec 109 MBytes 917 Mbits/sec 0 249 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 1021 MBytes 856 Mbits/sec 0 sender [ 4] 0.00-10.00 sec 1021 MBytes 856 Mbits/sec receiver iperf Done.#ifconfig cxgbe0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=ec00bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate,rxcsum_ipv6,txcsum_ipv6>ether 00:07:43:11:16:31 inet6 fe80::207:43ff:fe11:2690%cxgbe0 prefixlen 64 scopeid 0x1 nd6 options=21 <performnud,auto_linklocal>media: Ethernet 10Gbase-SR <full-duplex>status: active ncxgbe0: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500 ether 00:07:43:11:16:32 nd6 options=21 <performnud,auto_linklocal>media: Ethernet 10Gbase-SR <full-duplex>status: active #cat /boot/loader.conf.local kern.cam.boot_delay=10000 ahci_load="YES" kern.ipc.nmbclusters="1000000" hw.cxgbe.allow_mbufs_in_cluster=0</full-duplex></performnud,auto_linklocal></broadcast,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate,rxcsum_ipv6,txcsum_ipv6></up,broadcast,running,simplex,multicast>Any idea what could be wrong?
Thanks
-
I have done further testing. I took the card out of the pfsense router and put in my FreeBSD server.
Test 1
Between Mellanox ConnectX-2 EN (linux desktop) and T420-CR (FreeBSD server)[ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1.09 GBytes 9.39 Gbits/sec 189 543 KBytes [ 4] 1.00-2.00 sec 1.09 GBytes 9.40 Gbits/sec 189 679 KBytes [ 4] 2.00-3.00 sec 1.09 GBytes 9.38 Gbits/sec 189 611 KBytes [ 4] 3.00-4.00 sec 1.09 GBytes 9.37 Gbits/sec 189 553 KBytes [ 4] 4.00-5.00 sec 1.09 GBytes 9.39 Gbits/sec 189 690 KBytes [ 4] 5.00-6.00 sec 1.09 GBytes 9.39 Gbits/sec 252 496 KBytes [ 4] 6.00-7.00 sec 1.09 GBytes 9.39 Gbits/sec 189 574 KBytes [ 4] 7.00-8.00 sec 1.09 GBytes 9.39 Gbits/sec 189 638 KBytes [ 4] 8.00-9.00 sec 1.09 GBytes 9.39 Gbits/sec 189 665 KBytes [ 4] 9.00-10.00 sec 1.09 GBytes 9.39 Gbits/sec 189 494 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 10.9 GBytes 9.39 Gbits/sec 1953 sender [ 4] 0.00-10.00 sec 10.9 GBytes 9.39 Gbits/sec receiverTest2
Between Mellanox ConnectX-2 EN (FreeBSD server) and T420-CR (Linux)[ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1.09 GBytes 9.40 Gbits/sec 564 614 KBytes [ 4] 1.00-2.00 sec 1.09 GBytes 9.38 Gbits/sec 507 624 KBytes [ 4] 2.00-3.00 sec 1.09 GBytes 9.38 Gbits/sec 507 607 KBytes [ 4] 3.00-4.00 sec 1.09 GBytes 9.39 Gbits/sec 507 601 KBytes [ 4] 4.00-5.00 sec 1.09 GBytes 9.38 Gbits/sec 508 594 KBytes [ 4] 5.00-6.00 sec 1.09 GBytes 9.39 Gbits/sec 504 445 KBytes [ 4] 6.00-7.00 sec 1.09 GBytes 9.38 Gbits/sec 505 472 KBytes [ 4] 7.00-8.00 sec 1.09 GBytes 9.38 Gbits/sec 461 590 KBytes [ 4] 8.00-9.00 sec 1.09 GBytes 9.38 Gbits/sec 506 605 KBytes [ 4] 9.00-10.00 sec 1.09 GBytes 9.38 Gbits/sec 507 626 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 10.9 GBytes 9.38 Gbits/sec 5076 sender [ 4] 0.00-10.00 sec 10.9 GBytes 9.38 Gbits/sec receiverTest3
I then used the Mellanox ConnectX-2 EN card in the pfsense router
Between Mellanox ConnectX-2 EN (Pfsense router) and Mellanox ConnectX-2 EN (Linux)[ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 108 MBytes 901 Mbits/sec 0 247 MBytes [ 4] 1.00-2.00 sec 108 MBytes 905 Mbits/sec 0 247 MBytes [ 4] 2.00-3.00 sec 109 MBytes 913 Mbits/sec 0 247 MBytes [ 4] 3.00-4.00 sec 108 MBytes 909 Mbits/sec 0 247 MBytes [ 4] 4.00-5.00 sec 108 MBytes 905 Mbits/sec 0 247 MBytes [ 4] 5.00-6.00 sec 107 MBytes 898 Mbits/sec 0 247 MBytes [ 4] 6.00-7.00 sec 108 MBytes 903 Mbits/sec 0 247 MBytes [ 4] 7.00-8.00 sec 107 MBytes 896 Mbits/sec 0 247 MBytes [ 4] 8.00-9.01 sec 108 MBytes 905 Mbits/sec 0 247 MBytes [ 4] 9.01-10.00 sec 108 MBytes 910 Mbits/sec 0 247 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 1.05 GBytes 905 Mbits/sec 0 sender [ 4] 0.00-10.00 sec 1.05 GBytes 905 Mbits/sec receiverTest4
Booted pfsense router into Arch Linux
Between Mellanox ConnectX-2 EN (linux desktop) and T420-CR (pfsense router running Arch Linux)[ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1.10 GBytes 9.41 Gbits/sec 0 679 KBytes [ 4] 1.00-2.00 sec 1.08 GBytes 9.31 Gbits/sec 0 711 KBytes [ 4] 2.00-3.00 sec 1.09 GBytes 9.40 Gbits/sec 0 790 KBytes [ 4] 3.00-4.00 sec 1.09 GBytes 9.37 Gbits/sec 0 870 KBytes [ 4] 4.00-5.00 sec 1.09 GBytes 9.38 Gbits/sec 0 870 KBytes [ 4] 5.00-6.00 sec 1.09 GBytes 9.40 Gbits/sec 0 870 KBytes [ 4] 6.00-7.00 sec 1.09 GBytes 9.40 Gbits/sec 0 870 KBytes [ 4] 7.00-8.00 sec 1.09 GBytes 9.35 Gbits/sec 0 1.05 MBytes [ 4] 8.00-9.00 sec 1.09 GBytes 9.39 Gbits/sec 0 1.05 MBytes [ 4] 9.00-10.00 sec 1.09 GBytes 9.38 Gbits/sec 0 1.17 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 10.9 GBytes 9.38 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 10.9 GBytes 9.38 Gbits/sec receiverThese tests prove that
a) Chelsio card is ok on Linux and FreeBSD
b) Supermicro C2758 based pfsense router is ok
c) pfsense OS has some issue with 10gb cards (Chelsio and Mellanox) -
Formatted the disk and re-installed with no customization i.e. 1 WAN and 1 LAN.
Test1
With pf running[ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 116 MBytes 970 Mbits/sec 0 249 MBytes [ 4] 1.00-2.00 sec 116 MBytes 970 Mbits/sec 0 249 MBytes [ 4] 2.00-3.00 sec 115 MBytes 969 Mbits/sec 0 249 MBytes [ 4] 3.00-4.00 sec 116 MBytes 969 Mbits/sec 0 249 MBytes [ 4] 4.00-5.00 sec 116 MBytes 970 Mbits/sec 0 249 MBytes [ 4] 5.00-6.00 sec 116 MBytes 970 Mbits/sec 0 249 MBytes [ 4] 6.00-7.01 sec 117 MBytes 977 Mbits/sec 0 249 MBytes [ 4] 7.01-8.00 sec 115 MBytes 976 Mbits/sec 0 249 MBytes [ 4] 8.00-9.01 sec 117 MBytes 977 Mbits/sec 0 249 MBytes [ 4] 9.01-10.00 sec 116 MBytes 976 Mbits/sec 0 249 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 1.13 GBytes 972 Mbits/sec 0 sender [ 4] 0.00-10.00 sec 1.13 GBytes 972 Mbits/sec receiverTest2
pf disabled, pfctl -d[ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 189 MBytes 1.58 Gbits/sec 0 247 MBytes [ 4] 1.00-2.00 sec 188 MBytes 1.58 Gbits/sec 0 247 MBytes [ 4] 2.00-3.00 sec 188 MBytes 1.58 Gbits/sec 0 247 MBytes [ 4] 3.00-4.00 sec 189 MBytes 1.58 Gbits/sec 0 247 MBytes [ 4] 4.00-5.00 sec 189 MBytes 1.58 Gbits/sec 0 247 MBytes [ 4] 5.00-6.00 sec 188 MBytes 1.58 Gbits/sec 0 247 MBytes [ 4] 6.00-7.00 sec 189 MBytes 1.58 Gbits/sec 0 247 MBytes [ 4] 7.00-8.00 sec 188 MBytes 1.58 Gbits/sec 0 247 MBytes [ 4] 8.00-9.00 sec 188 MBytes 1.58 Gbits/sec 0 247 MBytes [ 4] 9.00-10.00 sec 189 MBytes 1.58 Gbits/sec 0 247 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 1.84 GBytes 1.58 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 1.84 GBytes 1.58 Gbits/sec receiver10gb is very much broken in pfsense on Supermicro C2758 platform
-
I'll probably never see one of these cards in my lifetime, but here are some links I've seen others post in relation to 10G issues:
https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards
https://calomel.org/freebsd_network_tuning.html
https://calomel.org/network_performance.html
https://doc.pfsense.org/index.php/Low_Throughput_Troubleshooting
-
Can somebody who has official C2758 based from pfsense post their system tunables and /boot/loader.conf. Clearly out of the box the router performs very poorly with 10gbe hardware.
-
I have two systems, one with Chelsio T520-CR-SO and another with Intel x520, neither has had any perf issues I've observed.
Here's the A1SRM-2758f system (although not an official negate box) with T520-CR-SO tuneable which is the only one I can get to easily.
[2.3.2-RELEASE][admin@pfSense.local.lan]/root: cat /boot/loader.conf autoboot_delay="3" comconsole_speed="115200" hw.usb.no_pf="1"
-
I compared the tunables with my system and there is no difference. Is there anything in /boot/loader.conf.local? And what do you have in System>Advanced>Networking>Network Interfaces?
I am guessing you are getting 10gb speeds.
Here is another test where i booted into vanilla FreeBSD-11 on the pfsense router
Chelsio T420-CR (pfsense router running FreeBSD 11) and Mellanox ConnectX-2 EN (linux desktop)[ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1.09 GBytes 9.38 Gbits/sec 0 1.03 MBytes [ 4] 1.00-2.00 sec 1.09 GBytes 9.40 Gbits/sec 0 1.08 MBytes [ 4] 2.00-3.00 sec 1.09 GBytes 9.40 Gbits/sec 0 1.08 MBytes [ 4] 3.00-4.00 sec 1.09 GBytes 9.40 Gbits/sec 0 1.08 MBytes [ 4] 4.00-5.00 sec 1.09 GBytes 9.40 Gbits/sec 0 1.08 MBytes [ 4] 5.00-6.00 sec 1.09 GBytes 9.40 Gbits/sec 0 1.08 MBytes [ 4] 6.00-7.00 sec 1.09 GBytes 9.40 Gbits/sec 0 1.08 MBytes [ 4] 7.00-8.00 sec 1.09 GBytes 9.40 Gbits/sec 0 1.08 MBytes [ 4] 8.00-9.00 sec 1.09 GBytes 9.40 Gbits/sec 0 1.08 MBytes [ 4] 9.00-10.00 sec 1.09 GBytes 9.40 Gbits/sec 0 1.08 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 10.9 GBytes 9.39 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 10.9 GBytes 9.39 Gbits/sec receiverSo, the card works well in native FreeBSD-11. I then tested in pfSense-2.4 snapshot image which is based on FreeBSD-11. Again i got dismal speeds.
-
Nothing interesting…
[2.3.2-RELEASE][admin@pfSense.local.lan]/root: cat /boot/loader.conf.local kern.cam.boot_delay=10000
-
Ill grab some benchmark points for you over the weekend when I get a free moment.
-
-
I compared the tunables with my system and there is no difference. Is there anything in /boot/loader.conf.local? And what do you have in System>Advanced>Networking>Network Interfaces?
I am guessing you are getting 10gb speeds.
Here is another test where i booted into vanilla FreeBSD-11 on the pfsense router
Chelsio T420-CR (pfsense router running FreeBSD 11) and Mellanox ConnectX-2 EN (linux desktop)[ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1.09 GBytes 9.38 Gbits/sec 0 1.03 MBytes [ 4] 1.00-2.00 sec 1.09 GBytes 9.40 Gbits/sec 0 1.08 MBytes [ 4] 2.00-3.00 sec 1.09 GBytes 9.40 Gbits/sec 0 1.08 MBytes [ 4] 3.00-4.00 sec 1.09 GBytes 9.40 Gbits/sec 0 1.08 MBytes [ 4] 4.00-5.00 sec 1.09 GBytes 9.40 Gbits/sec 0 1.08 MBytes [ 4] 5.00-6.00 sec 1.09 GBytes 9.40 Gbits/sec 0 1.08 MBytes [ 4] 6.00-7.00 sec 1.09 GBytes 9.40 Gbits/sec 0 1.08 MBytes [ 4] 7.00-8.00 sec 1.09 GBytes 9.40 Gbits/sec 0 1.08 MBytes [ 4] 8.00-9.00 sec 1.09 GBytes 9.40 Gbits/sec 0 1.08 MBytes [ 4] 9.00-10.00 sec 1.09 GBytes 9.40 Gbits/sec 0 1.08 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 10.9 GBytes 9.39 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 10.9 GBytes 9.39 Gbits/sec receiverSo, the card works well in native FreeBSD-11. I then tested in pfSense-2.4 snapshot image which is based on FreeBSD-11. Again i got dismal speeds.
sorry for a stream of replies… take a look at your network stack variables and see if theres anything obvious. Check buffer/window sizes/scaling etc. The articles KOM listed above are good starting points to troubleshoot.
-
I could be completely off the mark …. from memory (when working with storage) the Chelsio 10GB cards stuggled under heavy load with Flow Control enabled. I.e. Too many pause frames.
Try to disable flow control on all devices and see how things go.I appolgize in advance if this is leading you down the wrong path....
-
I could be completely off the mark …. from memory (when working with storage) the Chelsio 10GB cards stuggled under heavy load with Flow Control enabled. I.e. Too many pause frames.
Try to disable flow control on all devices and see how things go.I appolgize in advance if this is leading you down the wrong path....
I am clutching at the straws myself so appreciate any idea.
I can't find a flowcontrol sysctl for cxgb card. The only ones listed are for the igb interface
$sysctl -a | grep .fc: dev.igb.3.fc: 3 dev.igb.2.fc: 3 dev.igb.1.fc: 3 dev.igb.0.fc: 3However, i did find the dev.cxgbe.0.pause_settings sysctl. man page for cxgbe interface suggests
hw.cxgbe.pause_settings PAUSE frame settings. Bit 0 is rx_pause, bit 1 is tx_pause. rx_pause = 1 instructs the hardware to heed incoming PAUSE frames, 0 instructs it to ignore them. tx_pause = 1 allows the hardware to emit PAUSE frames when its receive FIFO reaches a high threshold, 0 prohibits the hardware from emitting PAUSE frames. The default is 3 (both rx_pause and tx_pause = 1). This tunable establishes the default PAUSE settings for all ports. Settings can be displayed and controlled on a per-port basis via the dev.cxgbe.X.pause_settings (dev.cxl.X.pause_settings for T5 cards) sysctl.The default value was set to
$sysctl dev.cxgbe.0.pause_settings dev.cxgbe.0.pause_settings: 3 <pause_rx,pause_tx></pause_rx,pause_tx>I changed it to 0
$sysctl dev.cxgbe.0.pause_settings=0 dev.cxgbe.0.pause_settings: 3 <pause_rx,pause_tx>-> 3 <pause_rx,pause_tx>$sysctl dev.cxgbe.0.pause_settings dev.cxgbe.0.pause_settings: 0</pause_rx,pause_tx></pause_rx,pause_tx>iperf3 results were the same as before
[ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 110 MBytes 919 Mbits/sec [ 5] 1.00-2.00 sec 111 MBytes 930 Mbits/sec [ 5] 2.00-3.00 sec 111 MBytes 929 Mbits/sec [ 5] 3.00-4.00 sec 111 MBytes 929 Mbits/sec [ 5] 4.00-5.00 sec 111 MBytes 932 Mbits/sec [ 5] 5.00-6.00 sec 111 MBytes 932 Mbits/sec [ 5] 6.00-7.00 sec 111 MBytes 929 Mbits/sec [ 5] 7.00-8.00 sec 111 MBytes 928 Mbits/sec [ 5] 8.00-9.00 sec 111 MBytes 931 Mbits/sec [ 5] 9.00-10.00 sec 111 MBytes 930 Mbits/sec [ 5] 10.00-10.01 sec 660 KBytes 925 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.01 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-10.01 sec 1.08 GBytes 929 Mbits/sec receiver -
I enabled tso on the cxgbe interface
ifconfig cxgbe0 tsoand the speed improved
[ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 190 MBytes 1.59 Gbits/sec 0 251 MBytes [ 4] 1.00-2.00 sec 185 MBytes 1.55 Gbits/sec 0 251 MBytes [ 4] 2.00-3.00 sec 180 MBytes 1.51 Gbits/sec 0 251 MBytes [ 4] 3.00-4.00 sec 176 MBytes 1.48 Gbits/sec 0 251 MBytes [ 4] 4.00-5.00 sec 183 MBytes 1.54 Gbits/sec 0 251 MBytes [ 4] 5.00-6.00 sec 184 MBytes 1.55 Gbits/sec 0 251 MBytes [ 4] 6.00-7.00 sec 180 MBytes 1.51 Gbits/sec 0 251 MBytes [ 4] 7.00-8.00 sec 190 MBytes 1.59 Gbits/sec 0 251 MBytes [ 4] 8.00-9.00 sec 189 MBytes 1.59 Gbits/sec 0 251 MBytes [ 4] 9.00-10.00 sec 192 MBytes 1.61 Gbits/sec 0 251 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 1.81 GBytes 1.55 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 1.81 GBytes 1.55 Gbits/sec receiverand after disabling pf
#pfctl -d [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1.06 GBytes 9.12 Gbits/sec 0 247 MBytes [ 4] 1.00-2.00 sec 1.09 GBytes 9.38 Gbits/sec 0 247 MBytes [ 4] 2.00-3.00 sec 1.09 GBytes 9.38 Gbits/sec 0 247 MBytes [ 4] 3.00-4.00 sec 1.09 GBytes 9.38 Gbits/sec 0 247 MBytes [ 4] 4.00-5.00 sec 1.09 GBytes 9.38 Gbits/sec 0 247 MBytes [ 4] 5.00-6.00 sec 1.09 GBytes 9.38 Gbits/sec 0 247 MBytes [ 4] 6.00-7.00 sec 733 MBytes 6.15 Gbits/sec 22 -1812469760.00 Bytes [ 4] 7.00-8.00 sec 660 MBytes 5.54 Gbits/sec 0 -1602799360.00 Bytes [ 4] 8.00-9.00 sec 610 MBytes 5.12 Gbits/sec 0 -1405709184.00 Bytes [ 4] 9.00-10.00 sec 908 MBytes 7.61 Gbits/sec 0 -1261036608.00 Bytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 9.36 GBytes 8.04 Gbits/sec 22 sender [ 4] 0.00-10.00 sec 9.36 GBytes 8.04 Gbits/sec receiver -
I changed from an MTU of 1500 to an MTU of 9000
With pf enabled
[ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 808 MBytes 6.78 Gbits/sec 0 -1880960188.00 Bytes [ 4] 1.00-2.00 sec 864 MBytes 7.25 Gbits/sec 0 -1880960188.00 Bytes [ 4] 2.00-3.00 sec 898 MBytes 7.54 Gbits/sec 0 -1880960188.00 Bytes [ 4] 3.00-4.00 sec 873 MBytes 7.32 Gbits/sec 0 -1880960188.00 Bytes [ 4] 4.00-5.00 sec 869 MBytes 7.28 Gbits/sec 0 -1880960188.00 Bytes [ 4] 5.00-6.00 sec 885 MBytes 7.42 Gbits/sec 0 -1880960188.00 Bytes [ 4] 6.00-7.00 sec 843 MBytes 7.08 Gbits/sec 0 -1880960188.00 Bytes [ 4] 7.00-8.00 sec 879 MBytes 7.37 Gbits/sec 0 -1880960188.00 Bytes [ 4] 8.00-9.00 sec 844 MBytes 7.08 Gbits/sec 0 -1880960188.00 Bytes [ 4] 9.00-10.00 sec 871 MBytes 7.31 Gbits/sec 0 -1880960188.00 Bytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 8.43 GBytes 7.24 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 8.43 GBytes 7.24 Gbits/sec receiverWith pf disabled
[ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1.10 GBytes 9.45 Gbits/sec 0 -1880960188.00 Bytes [ 4] 1.00-2.00 sec 1.15 GBytes 9.86 Gbits/sec 0 -1880960188.00 Bytes [ 4] 2.00-3.00 sec 1.15 GBytes 9.86 Gbits/sec 0 -1880960188.00 Bytes [ 4] 3.00-4.00 sec 1.15 GBytes 9.86 Gbits/sec 0 -1880960188.00 Bytes [ 4] 4.00-5.00 sec 1.15 GBytes 9.86 Gbits/sec 0 -1880960188.00 Bytes [ 4] 5.00-6.00 sec 1.15 GBytes 9.86 Gbits/sec 0 -1880960188.00 Bytes [ 4] 6.00-7.00 sec 1.15 GBytes 9.86 Gbits/sec 0 -1880960188.00 Bytes [ 4] 7.00-8.00 sec 1.15 GBytes 9.86 Gbits/sec 0 -1880960188.00 Bytes [ 4] 8.00-9.00 sec 1.15 GBytes 9.86 Gbits/sec 0 -1880960188.00 Bytes [ 4] 9.00-10.00 sec 1.15 GBytes 9.86 Gbits/sec 0 -1880960188.00 Bytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 11.4 GBytes 9.82 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 11.4 GBytes 9.82 Gbits/sec receiver -
Hi trmee,
Can flow control be disabled on the switch and your test host, too
-
Hi,
just read your post again. I can see that with pf disabled you achieve the desired speed.
In that Case it looks like the problem might indeed be with your specific setup.Do you have pfSense support ?
I imagine that pfSense support team could look at diagnostic data and drill down to find the bottleneck.
The fact that you get better speed with higher MTU means pfSense has to handle less data packages per time.The 2758 is an 8 core Atom, but I do not know what the expected throughput ought to be with that CPU. Of course it all depends on what the FW is doing (packages, NAT ?, SNORT? etc).
