Upgrade to 23.01 WAN speed halved
-
There are no changes to the cxgbe driver in 23.05 yet. I would not expect any difference.
The maintainer is investigating.Steve
-
@stephenw10
Per comments on redmine bug, no change with 23.05 -
It would very helpful if anyone hitting this can test a FreeBSD 14 snapshot. That would allow us to rule out any changes pfSense might be doing. It will also give the driver maintainer something specific to test against.
-
@stephenw10
I spun up a FreeBSD14 VM on Unraid and it does not find drive for my T520-CR. It says no driver attached. I have attached screenshot of both boot log and pciconf.
-
You might have to load the module in FreeBSD:
kldload if_cxbge -
If I run that command I get "can't load if_cxbge: No such file or directory"
I can see the file in /boot/kernel/ does that mean it is already loaded?
I even tried to load from that directory and same message. Even tried with sudo. I am mostly windows guy so maybe I am just doing something wrong.
-
Try using the full path:
kldload /boot/kernel/if_cxgbe.ko -
-
@stephenw10
Duh, I should have picked that up.OK I was able to load and pull ip address. I tried iperf3 - s and I doesn't look like it is installed by default.
UPDATE: ok, got it installed. below are my results using chelsio t520-cr
-
That's the send rate, the iperf3 client sends by default. If you run it reversed
-Rdo you see the half speed when receiving? -
@stephenw10
I was running iperf3 server on FreeBSD14 so it was receiving. Screenshot is me sending data from unraid host. -
But you see full line rate both ways?
What does
ifconfig -vvvm cxl0show under FreeBSD 14? It could just be different off loading options set. -
@stephenw10
Yes, both ways look good
output of ifconfig
-
Hmm, well it appears to have similar hardware off-loading options enabled but I don't have anything to compare it with directly. Do you have that same output from pfSense 23.01?
pfSense has TSO and LRO disabled by default. Both are enabled there. You might try disabling those and retesting.
-
Those with Chelsio hardware might want to review this thread:
@user1337 said in Reroot exposes SSH, Telnet, Web UI to WAN:
...I found the answer: https://calomel.org/freebsd_chelsio_toe_firewall.html:
The Chelsio Offload Policy (COP) manages when the TCP Offload Engine (TOE) takes affect allowing the card to only offload TCP connections which you want to offload and leave the other connection to the default FreeBSD TCP stack.
...
SECURITY NOTE: The Chelsio TCP Offload Engine (TOE) will completely bypass the FreeBSD TCP stack as well as any Chelsio filter rules. This means that traffic using TOE will NOT be filtered using our Chelsio Rules of Engagement filter rules or the Pf packet filter, nor will Pf log TOE connections. Netstat will show the connections using "netstat -np tcp" though. -
Ok this appears to be due to ratelimiting in the kernel. Thanks to the Chelsio driver maintainer who dug that out.
If you're seeing this please try setting the following loader value:
hw.cxgbe.niccaps_allowed="1"Add that to /boot/loader.conf.local
Create that file if it doesn't exist.Let us know if that removes the issue.
Steve
-
Looks good after upgrading to 23.01, creating that file and adding the entry. Also went to 23.05 and speed still looks good.
Thanks for getting this figured out.
-
@stephenw10 Thank you for posting this.
I applied the hw.cxgbe.niccaps_allowed="1" value to the /boot/loader.conf.local file as recommended and after a reboot I am also seeing correct speeds. Full gigabit out the WAN interface up and down along with full 10gb speeds to an internal speed test server that I host. This appears to be the fix to this issue. Thank you for all that you do!
-
@stephenw10 I have been using this fix for a while now and it is working great on version 23.05. Thank you to everyone that has put in an effort to solve the problem, it is much appreciated! :)
-
@stephenw10 I have added this line to my bootloader but I have not seen any performance change.
I am running a T422-CR card on 23.05.1-RELEASE of PFsense+
