Upgrade to 23.01 WAN speed halved
-
Those with Chelsio hardware might want to review this thread:
@user1337 said in Reroot exposes SSH, Telnet, Web UI to WAN:
...I found the answer: https://calomel.org/freebsd_chelsio_toe_firewall.html:
The Chelsio Offload Policy (COP) manages when the TCP Offload Engine (TOE) takes affect allowing the card to only offload TCP connections which you want to offload and leave the other connection to the default FreeBSD TCP stack.
...
SECURITY NOTE: The Chelsio TCP Offload Engine (TOE) will completely bypass the FreeBSD TCP stack as well as any Chelsio filter rules. This means that traffic using TOE will NOT be filtered using our Chelsio Rules of Engagement filter rules or the Pf packet filter, nor will Pf log TOE connections. Netstat will show the connections using "netstat -np tcp" though. -
Ok this appears to be due to ratelimiting in the kernel. Thanks to the Chelsio driver maintainer who dug that out.
If you're seeing this please try setting the following loader value:
hw.cxgbe.niccaps_allowed="1"Add that to /boot/loader.conf.local
Create that file if it doesn't exist.Let us know if that removes the issue.
Steve
-
Looks good after upgrading to 23.01, creating that file and adding the entry. Also went to 23.05 and speed still looks good.
Thanks for getting this figured out.
-
@stephenw10 Thank you for posting this.
I applied the hw.cxgbe.niccaps_allowed="1" value to the /boot/loader.conf.local file as recommended and after a reboot I am also seeing correct speeds. Full gigabit out the WAN interface up and down along with full 10gb speeds to an internal speed test server that I host. This appears to be the fix to this issue. Thank you for all that you do!
-
@stephenw10 I have been using this fix for a while now and it is working great on version 23.05. Thank you to everyone that has put in an effort to solve the problem, it is much appreciated! :)
-
@stephenw10 I have added this line to my bootloader but I have not seen any performance change.
I am running a T422-CR card on 23.05.1-RELEASE of PFsense+
-
What speed are you seeing now? What CPU is that system running?
Did you see the throughput go down after upgrading? From something before 23.01?
-
@stephenw10 I appear to be capped at 470mbs. This issue appeared to happen back in April when I upgraded from the latest CE to Pfsense plus back then. I didnāt realise there was a specific Chelsio fix potentially so I had not attempted to rectify.
CPU is an Intel 4th gen Xeon 1230 v3. It barely sits above a couple of percent when speed testing.
Hereās a copy of my boot log after adding in the tuning variable suggested:
https://pastebin.com/fUSVp71K
Iām not too familiar in with what a normal output in the boot log should look like
-
-
@stephenw10 Looking through that bootlog I can see the following referencing the card:
pci1: <ACPI PCI bus> on pcib1
pci1: <network, ethernet> at device 0.2 (no driver attached)
pci1: <network, ethernet> at device 0.3 (no driver attached)
t4nex0: <Chelsio T422-CR> mem 0xdf080000-0xdf0bffff,0xde800000-0xdeffffff,0xdf344000-0xdf345fff irq 16 at device 0.4 on pci1
cxgbe0: <port 0> on t4nex0
cxgbe0: Ethernet address: 98:be:94:5a:23:80
cxgbe0: 16 txq, 16 rxq (NIC); 8 txq (ETHOFLD)
cxgbe1: <port 1> on t4nex0
cxgbe1: Ethernet address: 98:be:94:5a:23:88
cxgbe1: 16 txq, 16 rxq (NIC); 8 txq (ETHOFLD)
cxgbe2: <port 2> on t4nex0
cxgbe2: Ethernet address: 98:be:94:5a:23:90
cxgbe2: 16 txq, 16 rxq (NIC); 8 txq (ETHOFLD)
cxgbe3: <port 3> on t4nex0
cxgbe3: Ethernet address: 98:be:94:5a:23:98
cxgbe3: 16 txq, 16 rxq (NIC); 8 txq (ETHOFLD)
t4nex0: PCIe gen2 x8, 4 ports, 66 MSI-X interrupts, 164 eq, 65 iq
pci1: <mass storage, SCSI> at device 0.5 (no driver attached)
pci1: <serial bus, Fibre Channel> at device 0.6 (no driver attached)
pci1: <network, ethernet> at device 0.7 (no driver attached) -
Do you see that set in the sysctl output after adding it to the loader file?
-
I can't see them set in System/Advanced/System Tunables - is this where I should be checking?
-
Run at the CLI:
sysctl hw.cxgbe -
@stephenw10 This is my output:
hw.cxgbe.tx_coalesce_gap: 5
hw.cxgbe.tx_coalesce_pkts: 32
hw.cxgbe.tx_coalesce: 1
hw.cxgbe.defrags: 0
hw.cxgbe.pullups: 12
hw.cxgbe.lro_mbufs: 0
hw.cxgbe.lro_entries: 8
hw.cxgbe.tscale: 1
hw.cxgbe.eo_max_backlog: 1048576
hw.cxgbe.tsclk: -1
hw.cxgbe.safest_rx_cluster: 4096
hw.cxgbe.largest_rx_cluster: 16384
hw.cxgbe.fl_pack: -1
hw.cxgbe.buffer_packing: -1
hw.cxgbe.ofld_cong_drop: 0
hw.cxgbe.cong_drop: 0
hw.cxgbe.spg_len: 64
hw.cxgbe.fl_pad: -1
hw.cxgbe.fl_pktshift: 0
hw.cxgbe.nm_txcsum: 0
hw.cxgbe.nm_split_rss: 0
hw.cxgbe.lazy_tx_credit_flush: 1
hw.cxgbe.starve_fl: 0
hw.cxgbe.nm_cong_drop: 1
hw.cxgbe.nm_holdoff_tmr_idx: 2
hw.cxgbe.nm_rx_nframes: 64
hw.cxgbe.nm_rx_ndesc: 256
hw.cxgbe.nm_black_hole: 0
hw.cxgbe.tls.combo_wrs: 0
hw.cxgbe.tls.inline_keys: 0
hw.cxgbe.kern_tls: 0
hw.cxgbe.cop_managed_offloading: 0
hw.cxgbe.drop_pkts_with_l4_errors: 0
hw.cxgbe.drop_pkts_with_l3_errors: 0
hw.cxgbe.drop_pkts_with_l2_errors: 1
hw.cxgbe.drop_ip_fragments: 0
hw.cxgbe.attack_filter: 0
hw.cxgbe.tx_vm_wr: 0
hw.cxgbe.clock_gate_on_suspend: 0
hw.cxgbe.reset_on_fatal_err: 0
hw.cxgbe.panic_on_fatal_err: 0
hw.cxgbe.pcie_relaxed_ordering: 0
hw.cxgbe.num_vis: 1
hw.cxgbe.fcoecaps_allowed: 0
hw.cxgbe.iscsicaps_allowed: 0
hw.cxgbe.cryptocaps_allowed: -1
hw.cxgbe.rdmacaps_allowed: 0
hw.cxgbe.toecaps_allowed: 0
hw.cxgbe.niccaps_allowed: 97
hw.cxgbe.switchcaps_allowed: 3
hw.cxgbe.linkcaps_allowed: 0
hw.cxgbe.nbmcaps_allowed: 0
hw.cxgbe.fw_install: 1
hw.cxgbe.autoneg: -1
hw.cxgbe.force_fec: -1
hw.cxgbe.fec: -1
hw.cxgbe.pause_settings: 7
hw.cxgbe.config_file: default
hw.cxgbe.interrupt_types: 7
hw.cxgbe.qsize_rxq: 1024
hw.cxgbe.qsize_txq: 1024
hw.cxgbe.holdoff_pktc_idx: -1
hw.cxgbe.holdoff_timer_idx: 1
hw.cxgbe.nnmrxq_vi: 2
hw.cxgbe.nnmtxq_vi: 2
hw.cxgbe.nnmrxq: 8
hw.cxgbe.nnmtxq: 8
hw.cxgbe.native_netmap: 2
hw.cxgbe.holdoff_pktc_idx_ofld: -1
hw.cxgbe.holdoff_timer_idx_ofld: 1
hw.cxgbe.nofldrxq_vi: 1
hw.cxgbe.nofldtxq_vi: 1
hw.cxgbe.nofldrxq: 2
hw.cxgbe.nofldtxq: 8
hw.cxgbe.rsrv_noflowq: 0
hw.cxgbe.nrxq_vi: 1
hw.cxgbe.ntxq_vi: 1
hw.cxgbe.nrxq: 16
hw.cxgbe.ntxq: 16
hw.cxgbe.toe.rexmt_backoff.15: -1
hw.cxgbe.toe.rexmt_backoff.14: -1
hw.cxgbe.toe.rexmt_backoff.13: -1
hw.cxgbe.toe.rexmt_backoff.12: -1
hw.cxgbe.toe.rexmt_backoff.11: -1
hw.cxgbe.toe.rexmt_backoff.10: -1
hw.cxgbe.toe.rexmt_backoff.9: -1
hw.cxgbe.toe.rexmt_backoff.8: -1
hw.cxgbe.toe.rexmt_backoff.7: -1
hw.cxgbe.toe.rexmt_backoff.6: -1
hw.cxgbe.toe.rexmt_backoff.5: -1
hw.cxgbe.toe.rexmt_backoff.4: -1
hw.cxgbe.toe.rexmt_backoff.3: -1
hw.cxgbe.toe.rexmt_backoff.2: -1
hw.cxgbe.toe.rexmt_backoff.1: -1
hw.cxgbe.toe.rexmt_backoff.0: -1
hw.cxgbe.toe.rexmt_count: 0
hw.cxgbe.toe.rexmt_max: 0
hw.cxgbe.toe.rexmt_min: 0
hw.cxgbe.toe.keepalive_count: 0
hw.cxgbe.toe.keepalive_interval: 0
hw.cxgbe.toe.keepalive_idle: 0
hw.cxgbe.clip_db_auto: 1 -
@bigtimmyc said in Upgrade to 23.01 WAN speed halved:
hw.cxgbe.niccaps_allowed: 97
97 doesn't seem right
-
Indeed it doesn't! Though I don't have one of those NICs to test it.
If you comment out those loader lines or remove the file does it change after rebooting?
-
@stephenw10 New output:
hw.cxgbe.tx_coalesce_gap: 5
hw.cxgbe.tx_coalesce_pkts: 32
hw.cxgbe.tx_coalesce: 1
hw.cxgbe.defrags: 0
hw.cxgbe.pullups: 9
hw.cxgbe.lro_mbufs: 0
hw.cxgbe.lro_entries: 8
hw.cxgbe.tscale: 1
hw.cxgbe.eo_max_backlog: 1048576
hw.cxgbe.tsclk: -1
hw.cxgbe.safest_rx_cluster: 4096
hw.cxgbe.largest_rx_cluster: 16384
hw.cxgbe.fl_pack: -1
hw.cxgbe.buffer_packing: -1
hw.cxgbe.ofld_cong_drop: 0
hw.cxgbe.cong_drop: 0
hw.cxgbe.spg_len: 64
hw.cxgbe.fl_pad: -1
hw.cxgbe.fl_pktshift: 0
hw.cxgbe.nm_txcsum: 0
hw.cxgbe.nm_split_rss: 0
hw.cxgbe.lazy_tx_credit_flush: 1
hw.cxgbe.starve_fl: 0
hw.cxgbe.nm_cong_drop: 1
hw.cxgbe.nm_holdoff_tmr_idx: 2
hw.cxgbe.nm_rx_nframes: 64
hw.cxgbe.nm_rx_ndesc: 256
hw.cxgbe.nm_black_hole: 0
hw.cxgbe.tls.combo_wrs: 0
hw.cxgbe.tls.inline_keys: 0
hw.cxgbe.kern_tls: 0
hw.cxgbe.cop_managed_offloading: 0
hw.cxgbe.drop_pkts_with_l4_errors: 0
hw.cxgbe.drop_pkts_with_l3_errors: 0
hw.cxgbe.drop_pkts_with_l2_errors: 1
hw.cxgbe.drop_ip_fragments: 0
hw.cxgbe.attack_filter: 0
hw.cxgbe.tx_vm_wr: 0
hw.cxgbe.clock_gate_on_suspend: 0
hw.cxgbe.reset_on_fatal_err: 0
hw.cxgbe.panic_on_fatal_err: 0
hw.cxgbe.pcie_relaxed_ordering: 0
hw.cxgbe.num_vis: 1
hw.cxgbe.fcoecaps_allowed: 0
hw.cxgbe.iscsicaps_allowed: 67
hw.cxgbe.cryptocaps_allowed: -1
hw.cxgbe.rdmacaps_allowed: 3
hw.cxgbe.toecaps_allowed: 1
hw.cxgbe.niccaps_allowed: 97
hw.cxgbe.switchcaps_allowed: 3
hw.cxgbe.linkcaps_allowed: 0
hw.cxgbe.nbmcaps_allowed: 0
hw.cxgbe.fw_install: 1
hw.cxgbe.autoneg: -1
hw.cxgbe.force_fec: -1
hw.cxgbe.fec: -1
hw.cxgbe.pause_settings: 7
hw.cxgbe.config_file: default
hw.cxgbe.interrupt_types: 7
hw.cxgbe.qsize_rxq: 1024
hw.cxgbe.qsize_txq: 1024
hw.cxgbe.holdoff_pktc_idx: -1
hw.cxgbe.holdoff_timer_idx: 1
hw.cxgbe.nnmrxq_vi: 2
hw.cxgbe.nnmtxq_vi: 2
hw.cxgbe.nnmrxq: 8
hw.cxgbe.nnmtxq: 8
hw.cxgbe.native_netmap: 2
hw.cxgbe.holdoff_pktc_idx_ofld: -1
hw.cxgbe.holdoff_timer_idx_ofld: 1
hw.cxgbe.nofldrxq_vi: 1
hw.cxgbe.nofldtxq_vi: 1
hw.cxgbe.nofldrxq: 2
hw.cxgbe.nofldtxq: 8
hw.cxgbe.rsrv_noflowq: 0
hw.cxgbe.nrxq_vi: 1
hw.cxgbe.ntxq_vi: 1
hw.cxgbe.nrxq: 16
hw.cxgbe.ntxq: 16
hw.cxgbe.toe.rexmt_backoff.15: -1
hw.cxgbe.toe.rexmt_backoff.14: -1
hw.cxgbe.toe.rexmt_backoff.13: -1
hw.cxgbe.toe.rexmt_backoff.12: -1
hw.cxgbe.toe.rexmt_backoff.11: -1
hw.cxgbe.toe.rexmt_backoff.10: -1
hw.cxgbe.toe.rexmt_backoff.9: -1
hw.cxgbe.toe.rexmt_backoff.8: -1
hw.cxgbe.toe.rexmt_backoff.7: -1
hw.cxgbe.toe.rexmt_backoff.6: -1
hw.cxgbe.toe.rexmt_backoff.5: -1
hw.cxgbe.toe.rexmt_backoff.4: -1
hw.cxgbe.toe.rexmt_backoff.3: -1
hw.cxgbe.toe.rexmt_backoff.2: -1
hw.cxgbe.toe.rexmt_backoff.1: -1
hw.cxgbe.toe.rexmt_backoff.0: -1
hw.cxgbe.toe.rexmt_count: 0
hw.cxgbe.toe.rexmt_max: 0
hw.cxgbe.toe.rexmt_min: 0
hw.cxgbe.toe.keepalive_count: 0
hw.cxgbe.toe.keepalive_interval: 0
hw.cxgbe.toe.keepalive_idle: 0
hw.cxgbe.clip_db_auto: 1niccaps appears to be unchanged after commented out all lines in the conf file
-
@stephenw10 I have found the problem. I'm too trusting of copy-pasting directly from the Pfsense docs. I've been stitched up by non-standard quotation characters.
Incorrect quotations around "1"
Corrected:
And won't you look at that
:
-
@bigtimmyc said in Upgrade to 23.01 WAN speed halved:
@stephenw10 I have found the problem. I'm too trusting of copy-pasting directly from the Pfsense docs. I've been stitched up by non-standard quotation characters.
Incorrect quotations around "1"
Corrected:
And won't you look at that
:
I don't think it was the quotes. If you look, they are exactly the same. Its the "#" sign at the start of each line. That is the special code for "a comment follows", and so the system ignores everything after the pound sign (#) up until it sees the end of the current line. The
#character is used to "comment out" a line. -
@bmeeks said in Upgrade to 23.01 WAN speed halved:
I don't think it was the quotes. If you look, they are exactly the same...
No they aren't, if you look closely at the quotes around the "1", in the first pane they are sloping quotes and in the second the normal double quote character.
