DSL Reports speed test causing crash on upload

tman222

I'm glad I ran across this post. I have been having a very similar problem and it has been bugging me for the last couple months. While my hardware is slightly different from yours, I also experienced system crashes during the upload portion of the DSL Reports speed test. I tried tweaking my traffic shaping settings, but nothing would help. Eventually it would still crash during the upload and require a reboot. I followed the thread you linked above (and a few related ones) and noticed that others have experienced similar stability issues, however with version 2.4 (like you I'm on 2.3.4). For instance, see here:

https://redmine.pfsense.org/issues/7149

Despite being on 2.3.4. I tried setting the igb queues to 1 today and so far things appears stable. Nothing I have tried so far (including running the DSL Reports speed test several times) has resulted in a crash. Here's to hoping this temporarily solves the problem until 2.4 is released (it looks like this issue may have already been fixed for 2.4).

Hope this helps.

tman222

Thought I would follow up on this thread to give an update: A week and half later things are still rock solid (no crashes). From what I can tell, changing the igb queues to 1 did the trick for me. I'm not 100% sure if the instability is related to the igb driver itself and default number of queues, traffic shaping, or a combination of the two. In any case, modifying the igb queues manually is a workaround worth looking into if the system becomes unstable when the network interfaces are under high load (i.e. close to maxed out).

morgejgmail

FWIW, the following solved my problem

https://forum.pfsense.org/index.php?topic=103841.msg684436#msg684436

nycfly

I had this exact issue on an HP box using HP's version of the Intel i340T4 (82580 based) NIC. Adding hw.igb.num_queues="1" fixed it and things have been rock solid since.

Harvy66

@nycfly:

I had this exact issue on an HP box using HP's version of the Intel i340T4 (82580 based) NIC. Adding hw.igb.num_queues="1" fixed it and things have been rock solid since.

In the case of HP rebadged devices, they sometimes mess with the bioses and may disable certain features that can make the device unstable if you treat it like a retail. Like the HP Intel i-350 has MSI-X hardware feature disabled, which can cause issues if you enable thinking it's a normal i350.

tman222

It has been a couple months since I posted in this thread and I thought I would report back with some more findings. I'm starting to become convinced that there isn't actually some type of bug out there, but rather that pfSense (and FreeBSD for that matter) need to be better tuned for higher speed (e.g. gigabit) internet connections. There are many great guides out there for this, here are a couple links (the first being a very nice thread that Harvy66 started on the subject):

https://forum.pfsense.org/index.php?topic=113496.0
https://calomel.org/freebsd_network_tuning.html

After adjusting/changing various tuning parameters and testing for stability, I decided to increase the number of queues and set hw.igb.num_queues="4" from "1" about a month ago. I actually have an Intel i350 4 port NIC and Intel 211 2 port NIC in my pfSense box - the former supports up to 8 queues, the latter up to 4. I figured I'd slowly try to increase the number of queues to see if the system stayed stable. Now after about a month with no issues whatsoever, I decided tonight to remove the num_queues restriction completely (i.e. use the default and let the system decide, which means the i350 interfaces use 8 queues and the i211 interfaces use 4). After performing some initial testing things appear stable so far - hoping it will stay this way. Will report back again in a few weeks with another update.

I hope this helps - would be curious to see if tuning network parameters also helps others overcome the num_queues workaround while maintaining stability.

nycfly

Thank you, tman222, that is very interesting and helpful. Indeed I lifted Harvy66's performance tuned parameters from the other thread, removed the limit on queues and some initial stress tests (the dslreports speedtest and simultaneous p2p download of all 12 current Ubuntu flavors) do not result in any kernel panics.

This is what my /boot/loader.conf.local now looks like:


kern.cam.boot_delay=10000
# hw.igb.num_queues="4"
kern.ipc.nmbclusters="1000000"
hw.igb.rxd=2048
hw.igb.txd=2048
net.pf.states_hashsize=524288
net.pf.source_nodes_hashsize=524288
hw.igb.fc_setting=0
hw.igb.rx_process_limit="-1"
hw.igb.tx_process_limit="-1"
net.inet.tcp.syncache.hashsize="2048"
net.inet.tcp.syncache.bucketlimit="16"
net.inet.tcp.syncache.cachelimit="32768"

What parameters did you end up with?

tman222

That's great to hear! Here are parameters I'm currently using:


kern.cam.boot_delay=10000
hw.igb.rx_process_limit="-1"
hw.igb.tx_process_limit="-1"
hw.igb.txd="4096"
hw.igb.rxd="4096"
net.link.ifqmaxlen="8192"
hw.igb.max_interrupt_rate="32000"
net.pf.states_hashsize="2097152"
net.pf.source_nodes_hashsize="65536"

Note that some of these values are rather large - probably larger than they need to be. However, I have a decent amount of memory (16GB) in my pfSense box so I figured I can afford to be a bit more generous. Now that I'm looking at it, I can probably increase net.pf.source_nodes.hashsize, but I think I need to read up on it a bit first.

tman222

Just thought I would follow up on this post with an update about a week later: I recently upgraded my system to 2.4.0-RC and kept all the tuned network settings. After performing some additional testing things are still stable for me and as mentioned above I no longer limit the interface queues in order to keep stability. It seems to reason then that tuning network parameters is definitely worth trying first if one experiences stability issues (as described in this thread) before limiting the queues on the interfaces.

nycfly

Mine has also been stable with the tuned parameters so definitely agree this is a better solution than limiting queue length.