Zotac Zbox CI323 Bad Throughput
Hi - I've scoured the forums here, as well as reddit. I haven't been able to find anything other than this thread with much talking about throughput. https://forum.netgate.com/topic/92884/zotac-zbox-ci323-nano/148
I just got my hands on one of these devices, and was hoping to use it to replace the UniFi USG-3 that I have which is randomly hanging. I was able to follow the steps to install the official FreeBSD realtek driver and turn it on, and confirm it is loading. The device no longer freezes when I load the connection.
My connection is a symmetrical 1Gbps fiber to the home and I can't get more than ~350Mbps down and ~420Mbps up with this device. I read somewhere that there may be significant performance penalties for the CPU and other hardware with FreeBSD as the core, but that seems bunk after this many years.
Has anyone been able to get closer to full gigabit throughput on these devices or something similar? I've tried enabling and disabling all the network offload options and they seem to make no difference. I can confirm the CPU usage never exceeded 5% when the throughput was capping out at these poor numbers.
I was able to get ~920 down and ~975 up using Windows and this hardware earlier today, so I don't think it has anything to do with the hardware itself, probably bad drivers and support on the FreeBSD side.
I love the feature set offered with PFsense, but am thinking I may have to look at hand building something on Linux to get better performance.
Is it a PPPoE connection?
Yes it absolutely is, I'll take a look at what you shared. I forgot to mention above that I installed the kernel module updated realtek official driver 1.96. Without that, the nics would off themselves immediately when I loaded it to more than like 100Mbps.
Edit: Update. I disabled the spectre/meltdown CPU changes in the GUI (not sure if this makes a difference) and turned on the encryption hardware support that the CPU works with. I also changed the settings outlined above; net.isr.dispatch=deferred and am now playing with the TSO LRO settings to see if they work at all with the updated driver version that I have installed.
I've tried a crapton of options in loader.conf.local, basically everything I can find about optimizing networking on FreeBSD and on PFsense. I was able to get it to do about 830/800 and nothing will push it higher than that. For reference, with the newer driver, I am using offloads on in the configuration panel. This is my conf.
I think this is no longer the realtek devices, but the fact that you can't do multi-threaded PPPoE. I know this has been looked at in detail by both the PFsense team and the FreeBSD team but seems to have stagnated a couple years ago. I'll keep poking at this when I have time, but I don't expect to make any progress as I've put about 7 hours in to research today alone and basically been constantly rebooting and testing, changing and rebooting the whole time.
net.isr.bindthreads="1"has increased throughput for some people.
The other net.isr thread settings in the doc may help.
I would not expect anything in net.inet.tcp to make any difference for traffic through the firewall. The PPPoE session is no TCP and TCP sessions to clients are not terminated on the firewall.