Upgrade to 23.01 WAN speed halved

[stephenw10]

Do you see that set in the sysctl output after adding it to the loader file?

[bigtimmyc]

@stephenw10

I can't see them set in System/Advanced/System Tunables - is this where I should be checking?

[stephenw10]

Run at the CLI: sysctl hw.cxgbe

[bigtimmyc]

@stephenw10 This is my output:

hw.cxgbe.tx_coalesce_gap: 5
hw.cxgbe.tx_coalesce_pkts: 32
hw.cxgbe.tx_coalesce: 1
hw.cxgbe.defrags: 0
hw.cxgbe.pullups: 12
hw.cxgbe.lro_mbufs: 0
hw.cxgbe.lro_entries: 8
hw.cxgbe.tscale: 1
hw.cxgbe.eo_max_backlog: 1048576
hw.cxgbe.tsclk: -1
hw.cxgbe.safest_rx_cluster: 4096
hw.cxgbe.largest_rx_cluster: 16384
hw.cxgbe.fl_pack: -1
hw.cxgbe.buffer_packing: -1
hw.cxgbe.ofld_cong_drop: 0
hw.cxgbe.cong_drop: 0
hw.cxgbe.spg_len: 64
hw.cxgbe.fl_pad: -1
hw.cxgbe.fl_pktshift: 0
hw.cxgbe.nm_txcsum: 0
hw.cxgbe.nm_split_rss: 0
hw.cxgbe.lazy_tx_credit_flush: 1
hw.cxgbe.starve_fl: 0
hw.cxgbe.nm_cong_drop: 1
hw.cxgbe.nm_holdoff_tmr_idx: 2
hw.cxgbe.nm_rx_nframes: 64
hw.cxgbe.nm_rx_ndesc: 256
hw.cxgbe.nm_black_hole: 0
hw.cxgbe.tls.combo_wrs: 0
hw.cxgbe.tls.inline_keys: 0
hw.cxgbe.kern_tls: 0
hw.cxgbe.cop_managed_offloading: 0
hw.cxgbe.drop_pkts_with_l4_errors: 0
hw.cxgbe.drop_pkts_with_l3_errors: 0
hw.cxgbe.drop_pkts_with_l2_errors: 1
hw.cxgbe.drop_ip_fragments: 0
hw.cxgbe.attack_filter: 0
hw.cxgbe.tx_vm_wr: 0
hw.cxgbe.clock_gate_on_suspend: 0
hw.cxgbe.reset_on_fatal_err: 0
hw.cxgbe.panic_on_fatal_err: 0
hw.cxgbe.pcie_relaxed_ordering: 0
hw.cxgbe.num_vis: 1
hw.cxgbe.fcoecaps_allowed: 0
hw.cxgbe.iscsicaps_allowed: 0
hw.cxgbe.cryptocaps_allowed: -1
hw.cxgbe.rdmacaps_allowed: 0
hw.cxgbe.toecaps_allowed: 0
hw.cxgbe.niccaps_allowed: 97
hw.cxgbe.switchcaps_allowed: 3
hw.cxgbe.linkcaps_allowed: 0
hw.cxgbe.nbmcaps_allowed: 0
hw.cxgbe.fw_install: 1
hw.cxgbe.autoneg: -1
hw.cxgbe.force_fec: -1
hw.cxgbe.fec: -1
hw.cxgbe.pause_settings: 7
hw.cxgbe.config_file: default
hw.cxgbe.interrupt_types: 7
hw.cxgbe.qsize_rxq: 1024
hw.cxgbe.qsize_txq: 1024
hw.cxgbe.holdoff_pktc_idx: -1
hw.cxgbe.holdoff_timer_idx: 1
hw.cxgbe.nnmrxq_vi: 2
hw.cxgbe.nnmtxq_vi: 2
hw.cxgbe.nnmrxq: 8
hw.cxgbe.nnmtxq: 8
hw.cxgbe.native_netmap: 2
hw.cxgbe.holdoff_pktc_idx_ofld: -1
hw.cxgbe.holdoff_timer_idx_ofld: 1
hw.cxgbe.nofldrxq_vi: 1
hw.cxgbe.nofldtxq_vi: 1
hw.cxgbe.nofldrxq: 2
hw.cxgbe.nofldtxq: 8
hw.cxgbe.rsrv_noflowq: 0
hw.cxgbe.nrxq_vi: 1
hw.cxgbe.ntxq_vi: 1
hw.cxgbe.nrxq: 16
hw.cxgbe.ntxq: 16
hw.cxgbe.toe.rexmt_backoff.15: -1
hw.cxgbe.toe.rexmt_backoff.14: -1
hw.cxgbe.toe.rexmt_backoff.13: -1
hw.cxgbe.toe.rexmt_backoff.12: -1
hw.cxgbe.toe.rexmt_backoff.11: -1
hw.cxgbe.toe.rexmt_backoff.10: -1
hw.cxgbe.toe.rexmt_backoff.9: -1
hw.cxgbe.toe.rexmt_backoff.8: -1
hw.cxgbe.toe.rexmt_backoff.7: -1
hw.cxgbe.toe.rexmt_backoff.6: -1
hw.cxgbe.toe.rexmt_backoff.5: -1
hw.cxgbe.toe.rexmt_backoff.4: -1
hw.cxgbe.toe.rexmt_backoff.3: -1
hw.cxgbe.toe.rexmt_backoff.2: -1
hw.cxgbe.toe.rexmt_backoff.1: -1
hw.cxgbe.toe.rexmt_backoff.0: -1
hw.cxgbe.toe.rexmt_count: 0
hw.cxgbe.toe.rexmt_max: 0
hw.cxgbe.toe.rexmt_min: 0
hw.cxgbe.toe.keepalive_count: 0
hw.cxgbe.toe.keepalive_interval: 0
hw.cxgbe.toe.keepalive_idle: 0
hw.cxgbe.clip_db_auto: 1

[bigtimmyc]

@bigtimmyc said in Upgrade to 23.01 WAN speed halved:

hw.cxgbe.niccaps_allowed: 97

97 doesn't seem right

[stephenw10]

Indeed it doesn't! Though I don't have one of those NICs to test it.

If you comment out those loader lines or remove the file does it change after rebooting?

[bigtimmyc]

@stephenw10 New output:

hw.cxgbe.tx_coalesce_gap: 5
hw.cxgbe.tx_coalesce_pkts: 32
hw.cxgbe.tx_coalesce: 1
hw.cxgbe.defrags: 0
hw.cxgbe.pullups: 9
hw.cxgbe.lro_mbufs: 0
hw.cxgbe.lro_entries: 8
hw.cxgbe.tscale: 1
hw.cxgbe.eo_max_backlog: 1048576
hw.cxgbe.tsclk: -1
hw.cxgbe.safest_rx_cluster: 4096
hw.cxgbe.largest_rx_cluster: 16384
hw.cxgbe.fl_pack: -1
hw.cxgbe.buffer_packing: -1
hw.cxgbe.ofld_cong_drop: 0
hw.cxgbe.cong_drop: 0
hw.cxgbe.spg_len: 64
hw.cxgbe.fl_pad: -1
hw.cxgbe.fl_pktshift: 0
hw.cxgbe.nm_txcsum: 0
hw.cxgbe.nm_split_rss: 0
hw.cxgbe.lazy_tx_credit_flush: 1
hw.cxgbe.starve_fl: 0
hw.cxgbe.nm_cong_drop: 1
hw.cxgbe.nm_holdoff_tmr_idx: 2
hw.cxgbe.nm_rx_nframes: 64
hw.cxgbe.nm_rx_ndesc: 256
hw.cxgbe.nm_black_hole: 0
hw.cxgbe.tls.combo_wrs: 0
hw.cxgbe.tls.inline_keys: 0
hw.cxgbe.kern_tls: 0
hw.cxgbe.cop_managed_offloading: 0
hw.cxgbe.drop_pkts_with_l4_errors: 0
hw.cxgbe.drop_pkts_with_l3_errors: 0
hw.cxgbe.drop_pkts_with_l2_errors: 1
hw.cxgbe.drop_ip_fragments: 0
hw.cxgbe.attack_filter: 0
hw.cxgbe.tx_vm_wr: 0
hw.cxgbe.clock_gate_on_suspend: 0
hw.cxgbe.reset_on_fatal_err: 0
hw.cxgbe.panic_on_fatal_err: 0
hw.cxgbe.pcie_relaxed_ordering: 0
hw.cxgbe.num_vis: 1
hw.cxgbe.fcoecaps_allowed: 0
hw.cxgbe.iscsicaps_allowed: 67
hw.cxgbe.cryptocaps_allowed: -1
hw.cxgbe.rdmacaps_allowed: 3
hw.cxgbe.toecaps_allowed: 1
hw.cxgbe.niccaps_allowed: 97
hw.cxgbe.switchcaps_allowed: 3
hw.cxgbe.linkcaps_allowed: 0
hw.cxgbe.nbmcaps_allowed: 0
hw.cxgbe.fw_install: 1
hw.cxgbe.autoneg: -1
hw.cxgbe.force_fec: -1
hw.cxgbe.fec: -1
hw.cxgbe.pause_settings: 7
hw.cxgbe.config_file: default
hw.cxgbe.interrupt_types: 7
hw.cxgbe.qsize_rxq: 1024
hw.cxgbe.qsize_txq: 1024
hw.cxgbe.holdoff_pktc_idx: -1
hw.cxgbe.holdoff_timer_idx: 1
hw.cxgbe.nnmrxq_vi: 2
hw.cxgbe.nnmtxq_vi: 2
hw.cxgbe.nnmrxq: 8
hw.cxgbe.nnmtxq: 8
hw.cxgbe.native_netmap: 2
hw.cxgbe.holdoff_pktc_idx_ofld: -1
hw.cxgbe.holdoff_timer_idx_ofld: 1
hw.cxgbe.nofldrxq_vi: 1
hw.cxgbe.nofldtxq_vi: 1
hw.cxgbe.nofldrxq: 2
hw.cxgbe.nofldtxq: 8
hw.cxgbe.rsrv_noflowq: 0
hw.cxgbe.nrxq_vi: 1
hw.cxgbe.ntxq_vi: 1
hw.cxgbe.nrxq: 16
hw.cxgbe.ntxq: 16
hw.cxgbe.toe.rexmt_backoff.15: -1
hw.cxgbe.toe.rexmt_backoff.14: -1
hw.cxgbe.toe.rexmt_backoff.13: -1
hw.cxgbe.toe.rexmt_backoff.12: -1
hw.cxgbe.toe.rexmt_backoff.11: -1
hw.cxgbe.toe.rexmt_backoff.10: -1
hw.cxgbe.toe.rexmt_backoff.9: -1
hw.cxgbe.toe.rexmt_backoff.8: -1
hw.cxgbe.toe.rexmt_backoff.7: -1
hw.cxgbe.toe.rexmt_backoff.6: -1
hw.cxgbe.toe.rexmt_backoff.5: -1
hw.cxgbe.toe.rexmt_backoff.4: -1
hw.cxgbe.toe.rexmt_backoff.3: -1
hw.cxgbe.toe.rexmt_backoff.2: -1
hw.cxgbe.toe.rexmt_backoff.1: -1
hw.cxgbe.toe.rexmt_backoff.0: -1
hw.cxgbe.toe.rexmt_count: 0
hw.cxgbe.toe.rexmt_max: 0
hw.cxgbe.toe.rexmt_min: 0
hw.cxgbe.toe.keepalive_count: 0
hw.cxgbe.toe.keepalive_interval: 0
hw.cxgbe.toe.keepalive_idle: 0
hw.cxgbe.clip_db_auto: 1

niccaps appears to be unchanged after commented out all lines in the conf file

[bigtimmyc]

@stephenw10 I have found the problem. I'm too trusting of copy-pasting directly from the Pfsense docs. I've been stitched up by non-standard quotation characters.

Incorrect quotations around "1"

Corrected:

And won't you look at that :

[bmeeks]

@bigtimmyc said in Upgrade to 23.01 WAN speed halved:

@stephenw10 I have found the problem. I'm too trusting of copy-pasting directly from the Pfsense docs. I've been stitched up by non-standard quotation characters.

Incorrect quotations around "1"

Corrected:

And won't you look at that :

I don't think it was the quotes. If you look, they are exactly the same. Its the "#" sign at the start of each line. That is the special code for "a comment follows", and so the system ignores everything after the pound sign (#) up until it sees the end of the current line. The # character is used to "comment out" a line.

[Misterb]

@bmeeks said in Upgrade to 23.01 WAN speed halved:

I don't think it was the quotes. If you look, they are exactly the same...

No they aren't, if you look closely at the quotes around the "1", in the first pane they are sloping quotes and in the second the normal double quote character.

[SteveITS]

@bmeeks he means the quote marks: “

In the first image the “ on the first line ane angled. I’ve been there once or twice in my computing time (not pfSense related), it’s not fun to discover.

[bigtimmyc]

@bmeeks sorry for the confusion, I just reposted my snip from the comment before yours, hence why the lines were still commented

[bmeeks]

@Misterb said in Upgrade to 23.01 WAN speed halved:

@bmeeks said in Upgrade to 23.01 WAN speed halved:

I don't think it was the quotes. If you look, they are exactly the same...

No they aren't, if you look closely at the quotes around the "1", in the first pane they are sloping quotes and in the second the normal double quote character.

Okay, I do see a difference now, but the removal of the "#" character is also necessary unless that is supposed to be part of the console shell prompt. I thought you were posting the dump of a local file.

[SteveITS]

@bigtimmyc Did the page get fixed already? It looks OK if I copy/paste to Word. If not there's a "Give Feedback" link the upper right of every doc page.

[tman222]

Following this thread, I see that the solution is to add (adjust) the following tunable:

hw.cxgbe.niccaps_allowed="1"

Is this adjustment recommended for anyone using Chelsio cards regardless of the make and model of the card? I am using a Chelsio T540-SO-CR and have not had any issues with speed. However, I see that that hw.cxgbe.niccaps_allowed tunable is set to 97 - is it recommended that I switch the 97 to 1, or only if impacted by this issue?

Thanks in advance.

[debo4479]

@tman222 said in Upgrade to 23.01 WAN speed halved:

Following this thread, I see that the solution is to add (adjust) the following tunable:

hw.cxgbe.niccaps_allowed="1"

Is this adjustment recommended for anyone using Chelsio cards regardless of the make and model of the card? I am using a Chelsio T540-SO-CR and have not had any issues with speed. However, I see that that hw.cxgbe.niccaps_allowed tunable is set to 97 - is it recommended that I switch the 97 to 1, or only if impacted by this issue?

Thanks in advance.

The SO variant didn't appear to be impacted, so no need to add/create loader conf file. It should be set to 1. It was showing 97 as qoute was incorrect in conf file.

[tman222]

@debo4479 said in Upgrade to 23.01 WAN speed halved:

@tman222 said in Upgrade to 23.01 WAN speed halved:

Following this thread, I see that the solution is to add (adjust) the following tunable:

hw.cxgbe.niccaps_allowed="1"

Is this adjustment recommended for anyone using Chelsio cards regardless of the make and model of the card? I am using a Chelsio T540-SO-CR and have not had any issues with speed. However, I see that that hw.cxgbe.niccaps_allowed tunable is set to 97 - is it recommended that I switch the 97 to 1, or only if impacted by this issue?

Thanks in advance.

The SO variant didn't appear to be impacted, so no need to add/create loader conf file. It should be set to 1. It was showing 97 as qoute was incorrect in conf file.

For my T540-SO-CR card, hw.cxgbe.niccaps_allowed="97" as well upon checking its value earlier. I have not made any changes to that tunable setting, this is the default value. Should this be changed to "1" regardless of Chelsio make and model or only on the non SO variants? What does this tunable represent in terms of capabilities on the card? Does setting it to "1" just disable rate limiting? Thanks again.

[SteveITS]

@tman222 The doc page has it listed under "Chelsio cxgbe(4) Cards."
"To disable this rate limiting capability, add the following as a Loader Tunable." If you're not affected I guess it doesn't matter for you...?

[tman222]

@SteveITS said in Upgrade to 23.01 WAN speed halved:

@tman222 The doc page has it listed under "Chelsio cxgbe(4) Cards."
"To disable this rate limiting capability, add the following as a Loader Tunable." If you're not affected I guess it doesn't matter for you...?

Right - I was curious though whether this change to the tunable should be applied regardless of Chelsio card make and model or just if a regular (non SO) card is being used. The documentation wasn't clear on that.

Thanks again.

[stephenw10]

Yup AFAIK it only affected that one card.