Upgrade to 23.01 WAN speed halved
-
@stephenw10 I appear to be capped at 470mbs. This issue appeared to happen back in April when I upgraded from the latest CE to Pfsense plus back then. I didn’t realise there was a specific Chelsio fix potentially so I had not attempted to rectify.
CPU is an Intel 4th gen Xeon 1230 v3. It barely sits above a couple of percent when speed testing.
Here’s a copy of my boot log after adding in the tuning variable suggested:
https://pastebin.com/fUSVp71K
I’m not too familiar in with what a normal output in the boot log should look like
-
-
@stephenw10 Looking through that bootlog I can see the following referencing the card:
pci1: <ACPI PCI bus> on pcib1
pci1: <network, ethernet> at device 0.2 (no driver attached)
pci1: <network, ethernet> at device 0.3 (no driver attached)
t4nex0: <Chelsio T422-CR> mem 0xdf080000-0xdf0bffff,0xde800000-0xdeffffff,0xdf344000-0xdf345fff irq 16 at device 0.4 on pci1
cxgbe0: <port 0> on t4nex0
cxgbe0: Ethernet address: 98:be:94:5a:23:80
cxgbe0: 16 txq, 16 rxq (NIC); 8 txq (ETHOFLD)
cxgbe1: <port 1> on t4nex0
cxgbe1: Ethernet address: 98:be:94:5a:23:88
cxgbe1: 16 txq, 16 rxq (NIC); 8 txq (ETHOFLD)
cxgbe2: <port 2> on t4nex0
cxgbe2: Ethernet address: 98:be:94:5a:23:90
cxgbe2: 16 txq, 16 rxq (NIC); 8 txq (ETHOFLD)
cxgbe3: <port 3> on t4nex0
cxgbe3: Ethernet address: 98:be:94:5a:23:98
cxgbe3: 16 txq, 16 rxq (NIC); 8 txq (ETHOFLD)
t4nex0: PCIe gen2 x8, 4 ports, 66 MSI-X interrupts, 164 eq, 65 iq
pci1: <mass storage, SCSI> at device 0.5 (no driver attached)
pci1: <serial bus, Fibre Channel> at device 0.6 (no driver attached)
pci1: <network, ethernet> at device 0.7 (no driver attached) -
Do you see that set in the sysctl output after adding it to the loader file?
-
I can't see them set in System/Advanced/System Tunables - is this where I should be checking?
-
Run at the CLI:
sysctl hw.cxgbe -
@stephenw10 This is my output:
hw.cxgbe.tx_coalesce_gap: 5
hw.cxgbe.tx_coalesce_pkts: 32
hw.cxgbe.tx_coalesce: 1
hw.cxgbe.defrags: 0
hw.cxgbe.pullups: 12
hw.cxgbe.lro_mbufs: 0
hw.cxgbe.lro_entries: 8
hw.cxgbe.tscale: 1
hw.cxgbe.eo_max_backlog: 1048576
hw.cxgbe.tsclk: -1
hw.cxgbe.safest_rx_cluster: 4096
hw.cxgbe.largest_rx_cluster: 16384
hw.cxgbe.fl_pack: -1
hw.cxgbe.buffer_packing: -1
hw.cxgbe.ofld_cong_drop: 0
hw.cxgbe.cong_drop: 0
hw.cxgbe.spg_len: 64
hw.cxgbe.fl_pad: -1
hw.cxgbe.fl_pktshift: 0
hw.cxgbe.nm_txcsum: 0
hw.cxgbe.nm_split_rss: 0
hw.cxgbe.lazy_tx_credit_flush: 1
hw.cxgbe.starve_fl: 0
hw.cxgbe.nm_cong_drop: 1
hw.cxgbe.nm_holdoff_tmr_idx: 2
hw.cxgbe.nm_rx_nframes: 64
hw.cxgbe.nm_rx_ndesc: 256
hw.cxgbe.nm_black_hole: 0
hw.cxgbe.tls.combo_wrs: 0
hw.cxgbe.tls.inline_keys: 0
hw.cxgbe.kern_tls: 0
hw.cxgbe.cop_managed_offloading: 0
hw.cxgbe.drop_pkts_with_l4_errors: 0
hw.cxgbe.drop_pkts_with_l3_errors: 0
hw.cxgbe.drop_pkts_with_l2_errors: 1
hw.cxgbe.drop_ip_fragments: 0
hw.cxgbe.attack_filter: 0
hw.cxgbe.tx_vm_wr: 0
hw.cxgbe.clock_gate_on_suspend: 0
hw.cxgbe.reset_on_fatal_err: 0
hw.cxgbe.panic_on_fatal_err: 0
hw.cxgbe.pcie_relaxed_ordering: 0
hw.cxgbe.num_vis: 1
hw.cxgbe.fcoecaps_allowed: 0
hw.cxgbe.iscsicaps_allowed: 0
hw.cxgbe.cryptocaps_allowed: -1
hw.cxgbe.rdmacaps_allowed: 0
hw.cxgbe.toecaps_allowed: 0
hw.cxgbe.niccaps_allowed: 97
hw.cxgbe.switchcaps_allowed: 3
hw.cxgbe.linkcaps_allowed: 0
hw.cxgbe.nbmcaps_allowed: 0
hw.cxgbe.fw_install: 1
hw.cxgbe.autoneg: -1
hw.cxgbe.force_fec: -1
hw.cxgbe.fec: -1
hw.cxgbe.pause_settings: 7
hw.cxgbe.config_file: default
hw.cxgbe.interrupt_types: 7
hw.cxgbe.qsize_rxq: 1024
hw.cxgbe.qsize_txq: 1024
hw.cxgbe.holdoff_pktc_idx: -1
hw.cxgbe.holdoff_timer_idx: 1
hw.cxgbe.nnmrxq_vi: 2
hw.cxgbe.nnmtxq_vi: 2
hw.cxgbe.nnmrxq: 8
hw.cxgbe.nnmtxq: 8
hw.cxgbe.native_netmap: 2
hw.cxgbe.holdoff_pktc_idx_ofld: -1
hw.cxgbe.holdoff_timer_idx_ofld: 1
hw.cxgbe.nofldrxq_vi: 1
hw.cxgbe.nofldtxq_vi: 1
hw.cxgbe.nofldrxq: 2
hw.cxgbe.nofldtxq: 8
hw.cxgbe.rsrv_noflowq: 0
hw.cxgbe.nrxq_vi: 1
hw.cxgbe.ntxq_vi: 1
hw.cxgbe.nrxq: 16
hw.cxgbe.ntxq: 16
hw.cxgbe.toe.rexmt_backoff.15: -1
hw.cxgbe.toe.rexmt_backoff.14: -1
hw.cxgbe.toe.rexmt_backoff.13: -1
hw.cxgbe.toe.rexmt_backoff.12: -1
hw.cxgbe.toe.rexmt_backoff.11: -1
hw.cxgbe.toe.rexmt_backoff.10: -1
hw.cxgbe.toe.rexmt_backoff.9: -1
hw.cxgbe.toe.rexmt_backoff.8: -1
hw.cxgbe.toe.rexmt_backoff.7: -1
hw.cxgbe.toe.rexmt_backoff.6: -1
hw.cxgbe.toe.rexmt_backoff.5: -1
hw.cxgbe.toe.rexmt_backoff.4: -1
hw.cxgbe.toe.rexmt_backoff.3: -1
hw.cxgbe.toe.rexmt_backoff.2: -1
hw.cxgbe.toe.rexmt_backoff.1: -1
hw.cxgbe.toe.rexmt_backoff.0: -1
hw.cxgbe.toe.rexmt_count: 0
hw.cxgbe.toe.rexmt_max: 0
hw.cxgbe.toe.rexmt_min: 0
hw.cxgbe.toe.keepalive_count: 0
hw.cxgbe.toe.keepalive_interval: 0
hw.cxgbe.toe.keepalive_idle: 0
hw.cxgbe.clip_db_auto: 1 -
@bigtimmyc said in Upgrade to 23.01 WAN speed halved:
hw.cxgbe.niccaps_allowed: 97
97 doesn't seem right
-
Indeed it doesn't! Though I don't have one of those NICs to test it.
If you comment out those loader lines or remove the file does it change after rebooting?
-
@stephenw10 New output:
hw.cxgbe.tx_coalesce_gap: 5
hw.cxgbe.tx_coalesce_pkts: 32
hw.cxgbe.tx_coalesce: 1
hw.cxgbe.defrags: 0
hw.cxgbe.pullups: 9
hw.cxgbe.lro_mbufs: 0
hw.cxgbe.lro_entries: 8
hw.cxgbe.tscale: 1
hw.cxgbe.eo_max_backlog: 1048576
hw.cxgbe.tsclk: -1
hw.cxgbe.safest_rx_cluster: 4096
hw.cxgbe.largest_rx_cluster: 16384
hw.cxgbe.fl_pack: -1
hw.cxgbe.buffer_packing: -1
hw.cxgbe.ofld_cong_drop: 0
hw.cxgbe.cong_drop: 0
hw.cxgbe.spg_len: 64
hw.cxgbe.fl_pad: -1
hw.cxgbe.fl_pktshift: 0
hw.cxgbe.nm_txcsum: 0
hw.cxgbe.nm_split_rss: 0
hw.cxgbe.lazy_tx_credit_flush: 1
hw.cxgbe.starve_fl: 0
hw.cxgbe.nm_cong_drop: 1
hw.cxgbe.nm_holdoff_tmr_idx: 2
hw.cxgbe.nm_rx_nframes: 64
hw.cxgbe.nm_rx_ndesc: 256
hw.cxgbe.nm_black_hole: 0
hw.cxgbe.tls.combo_wrs: 0
hw.cxgbe.tls.inline_keys: 0
hw.cxgbe.kern_tls: 0
hw.cxgbe.cop_managed_offloading: 0
hw.cxgbe.drop_pkts_with_l4_errors: 0
hw.cxgbe.drop_pkts_with_l3_errors: 0
hw.cxgbe.drop_pkts_with_l2_errors: 1
hw.cxgbe.drop_ip_fragments: 0
hw.cxgbe.attack_filter: 0
hw.cxgbe.tx_vm_wr: 0
hw.cxgbe.clock_gate_on_suspend: 0
hw.cxgbe.reset_on_fatal_err: 0
hw.cxgbe.panic_on_fatal_err: 0
hw.cxgbe.pcie_relaxed_ordering: 0
hw.cxgbe.num_vis: 1
hw.cxgbe.fcoecaps_allowed: 0
hw.cxgbe.iscsicaps_allowed: 67
hw.cxgbe.cryptocaps_allowed: -1
hw.cxgbe.rdmacaps_allowed: 3
hw.cxgbe.toecaps_allowed: 1
hw.cxgbe.niccaps_allowed: 97
hw.cxgbe.switchcaps_allowed: 3
hw.cxgbe.linkcaps_allowed: 0
hw.cxgbe.nbmcaps_allowed: 0
hw.cxgbe.fw_install: 1
hw.cxgbe.autoneg: -1
hw.cxgbe.force_fec: -1
hw.cxgbe.fec: -1
hw.cxgbe.pause_settings: 7
hw.cxgbe.config_file: default
hw.cxgbe.interrupt_types: 7
hw.cxgbe.qsize_rxq: 1024
hw.cxgbe.qsize_txq: 1024
hw.cxgbe.holdoff_pktc_idx: -1
hw.cxgbe.holdoff_timer_idx: 1
hw.cxgbe.nnmrxq_vi: 2
hw.cxgbe.nnmtxq_vi: 2
hw.cxgbe.nnmrxq: 8
hw.cxgbe.nnmtxq: 8
hw.cxgbe.native_netmap: 2
hw.cxgbe.holdoff_pktc_idx_ofld: -1
hw.cxgbe.holdoff_timer_idx_ofld: 1
hw.cxgbe.nofldrxq_vi: 1
hw.cxgbe.nofldtxq_vi: 1
hw.cxgbe.nofldrxq: 2
hw.cxgbe.nofldtxq: 8
hw.cxgbe.rsrv_noflowq: 0
hw.cxgbe.nrxq_vi: 1
hw.cxgbe.ntxq_vi: 1
hw.cxgbe.nrxq: 16
hw.cxgbe.ntxq: 16
hw.cxgbe.toe.rexmt_backoff.15: -1
hw.cxgbe.toe.rexmt_backoff.14: -1
hw.cxgbe.toe.rexmt_backoff.13: -1
hw.cxgbe.toe.rexmt_backoff.12: -1
hw.cxgbe.toe.rexmt_backoff.11: -1
hw.cxgbe.toe.rexmt_backoff.10: -1
hw.cxgbe.toe.rexmt_backoff.9: -1
hw.cxgbe.toe.rexmt_backoff.8: -1
hw.cxgbe.toe.rexmt_backoff.7: -1
hw.cxgbe.toe.rexmt_backoff.6: -1
hw.cxgbe.toe.rexmt_backoff.5: -1
hw.cxgbe.toe.rexmt_backoff.4: -1
hw.cxgbe.toe.rexmt_backoff.3: -1
hw.cxgbe.toe.rexmt_backoff.2: -1
hw.cxgbe.toe.rexmt_backoff.1: -1
hw.cxgbe.toe.rexmt_backoff.0: -1
hw.cxgbe.toe.rexmt_count: 0
hw.cxgbe.toe.rexmt_max: 0
hw.cxgbe.toe.rexmt_min: 0
hw.cxgbe.toe.keepalive_count: 0
hw.cxgbe.toe.keepalive_interval: 0
hw.cxgbe.toe.keepalive_idle: 0
hw.cxgbe.clip_db_auto: 1niccaps appears to be unchanged after commented out all lines in the conf file
-
@stephenw10 I have found the problem. I'm too trusting of copy-pasting directly from the Pfsense docs. I've been stitched up by non-standard quotation characters.
Incorrect quotations around "1"
Corrected:
And won't you look at that
:
-
@bigtimmyc said in Upgrade to 23.01 WAN speed halved:
@stephenw10 I have found the problem. I'm too trusting of copy-pasting directly from the Pfsense docs. I've been stitched up by non-standard quotation characters.
Incorrect quotations around "1"
Corrected:
And won't you look at that
:
I don't think it was the quotes. If you look, they are exactly the same. Its the "#" sign at the start of each line. That is the special code for "a comment follows", and so the system ignores everything after the pound sign (#) up until it sees the end of the current line. The
#character is used to "comment out" a line. -
@bmeeks said in Upgrade to 23.01 WAN speed halved:
I don't think it was the quotes. If you look, they are exactly the same...
No they aren't, if you look closely at the quotes around the "1", in the first pane they are sloping quotes and in the second the normal double quote character.
-
@bmeeks he means the quote marks: “
In the first image the “ on the first line ane angled. I’ve been there once or twice in my computing time (not pfSense related), it’s not fun to discover.
-
@bmeeks sorry for the confusion, I just reposted my snip from the comment before yours, hence why the lines were still commented
-
@Misterb said in Upgrade to 23.01 WAN speed halved:
@bmeeks said in Upgrade to 23.01 WAN speed halved:
I don't think it was the quotes. If you look, they are exactly the same...
No they aren't, if you look closely at the quotes around the "1", in the first pane they are sloping quotes and in the second the normal double quote character.
Okay, I do see a difference now, but the removal of the "#" character is also necessary unless that is supposed to be part of the console shell prompt. I thought you were posting the dump of a local file.
-
@bigtimmyc Did the page get fixed already? It looks OK if I copy/paste to Word. If not there's a "Give Feedback" link the upper right of every doc page.
-
Following this thread, I see that the solution is to add (adjust) the following tunable:
hw.cxgbe.niccaps_allowed="1"Is this adjustment recommended for anyone using Chelsio cards regardless of the make and model of the card? I am using a Chelsio T540-SO-CR and have not had any issues with speed. However, I see that that
hw.cxgbe.niccaps_allowedtunable is set to97- is it recommended that I switch the97to1, or only if impacted by this issue?Thanks in advance.
-
@tman222 said in Upgrade to 23.01 WAN speed halved:
Following this thread, I see that the solution is to add (adjust) the following tunable:
hw.cxgbe.niccaps_allowed="1"Is this adjustment recommended for anyone using Chelsio cards regardless of the make and model of the card? I am using a Chelsio T540-SO-CR and have not had any issues with speed. However, I see that that
hw.cxgbe.niccaps_allowedtunable is set to97- is it recommended that I switch the97to1, or only if impacted by this issue?Thanks in advance.
The SO variant didn't appear to be impacted, so no need to add/create loader conf file. It should be set to 1. It was showing 97 as qoute was incorrect in conf file.
-
@debo4479 said in Upgrade to 23.01 WAN speed halved:
@tman222 said in Upgrade to 23.01 WAN speed halved:
Following this thread, I see that the solution is to add (adjust) the following tunable:
hw.cxgbe.niccaps_allowed="1"Is this adjustment recommended for anyone using Chelsio cards regardless of the make and model of the card? I am using a Chelsio T540-SO-CR and have not had any issues with speed. However, I see that that
hw.cxgbe.niccaps_allowedtunable is set to97- is it recommended that I switch the97to1, or only if impacted by this issue?Thanks in advance.
The SO variant didn't appear to be impacted, so no need to add/create loader conf file. It should be set to 1. It was showing 97 as qoute was incorrect in conf file.
For my T540-SO-CR card,
hw.cxgbe.niccaps_allowed="97"as well upon checking its value earlier. I have not made any changes to that tunable setting, this is the default value. Should this be changed to "1" regardless of Chelsio make and model or only on the non SO variants? What does this tunable represent in terms of capabilities on the card? Does setting it to "1" just disable rate limiting? Thanks again.
