Zotac Zbox CI323 Bad Throughput

PWn3R

Hi - I've scoured the forums here, as well as reddit. I haven't been able to find anything other than this thread with much talking about throughput. https://forum.netgate.com/topic/92884/zotac-zbox-ci323-nano/148

I just got my hands on one of these devices, and was hoping to use it to replace the UniFi USG-3 that I have which is randomly hanging. I was able to follow the steps to install the official FreeBSD realtek driver and turn it on, and confirm it is loading. The device no longer freezes when I load the connection.

My connection is a symmetrical 1Gbps fiber to the home and I can't get more than ~350Mbps down and ~420Mbps up with this device. I read somewhere that there may be significant performance penalties for the CPU and other hardware with FreeBSD as the core, but that seems bunk after this many years.

Has anyone been able to get closer to full gigabit throughput on these devices or something similar? I've tried enabling and disabling all the network offload options and they seem to make no difference. I can confirm the CPU usage never exceeded 5% when the throughput was capping out at these poor numbers.

I was able to get ~920 down and ~975 up using Windows and this hardware earlier today, so I don't think it has anything to do with the hardware itself, probably bad drivers and support on the FreeBSD side.

I love the feature set offered with PFsense, but am thinking I may have to look at hand building something on Linux to get better performance.

Any thoughts?

stephenw10

Is it a PPPoE connection?

https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#pppoe-with-multi-queue-nics

Steve

PWn3R

@stephenw10 said in Zotac Zbox CI323 Bad Throughput:

https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#pppoe-with-multi-queue-nics

Yes it absolutely is, I'll take a look at what you shared. I forgot to mention above that I installed the kernel module updated realtek official driver 1.96. Without that, the nics would off themselves immediately when I loaded it to more than like 100Mbps.

Edit: Update. I disabled the spectre/meltdown CPU changes in the GUI (not sure if this makes a difference) and turned on the encryption hardware support that the CPU works with. I also changed the settings outlined above; net.isr.dispatch=deferred and am now playing with the TSO LRO settings to see if they work at all with the updated driver version that I have installed.

PWn3R

I've tried a crapton of options in loader.conf.local, basically everything I can find about optimizing networking on FreeBSD and on PFsense. I was able to get it to do about 830/800 and nothing will push it higher than that. For reference, with the newer driver, I am using offloads on in the configuration panel. This is my conf.

net.isr.dispatch=deferred
net.inet.tcp.tso=1
net.inet.tcp.lro=1
hw.pci.enable_msix=1
net.isr.direct=0
net.isr.direct_force=0
net.inet.ip.intr_queue_maxlen=3000

I think this is no longer the realtek devices, but the fact that you can't do multi-threaded PPPoE. I know this has been looked at in detail by both the PFsense team and the FreeBSD team but seems to have stagnated a couple years ago. I'll keep poking at this when I have time, but I don't expect to make any progress as I've put about 7 hours in to research today alone and basically been constantly rebooting and testing, changing and rebooting the whole time.

stephenw10

Setting net.isr.bindthreads="1" has increased throughput for some people.

The other net.isr thread settings in the doc may help.

I would not expect anything in net.inet.tcp to make any difference for traffic through the firewall. The PPPoE session is no TCP and TCP sessions to clients are not terminated on the firewall.

Steve

PWn3R

@stephenw10 Thanks for the information. I tried that and it ended up making the performance worse, down into the low 700s on download. I tried a couple hacky things including trying to add someone elses BSD patch for muti-threaded PPPoE and that blew the whole thing up. I'm going to table this for the time being and I was able to improve the performance to ~890/930 by using OpenWRT with some tweaking. Thanks for all your help!