Upgrading Realtek with alternate driver - Is it worth it?
-
Hi All,
I'm on 2.4.5 release-p1. My ISP just started offering 1Gbps speeds, but I haven't been able to take full advantage of this as I realised that the pfSense box I purchased from AliExpress is the N3160 with Realtek RTL8111E network controllers.
Currently I'm only able to get a bandwidth of ~600Mbps on a Fiber to the Premise connection when routing via pfSense. I have tested at the NTD directly and can actually receive the full bandwidth, so I been able to determine for a fact that the Realtek network controllers are the bottleneck. FWIW, I'm not running any major packages at this stage, so the CPU and memory consumption is very low and stable.
I have looked at the ways to improve network throughput (based on https://docs.netgate.com/pfsense/en/latest/interfaces/low-throughput-troubleshooting.html) but haven't been able to see any improvements at all.
The only other option beyond replacing the appliance with better Intel cards seems to be to use an alternate driver (to the one FreeBSD provides - a version available here:https://forum.netgate.com/topic/153469/official-realtek-driver-binary-1-95-for-2-5-0-development-amd64-freebsd-12-1-stable/) for Realtek Network Controller chipset.
My questions are:
- Is there anyone here who is using a Realtek RTL8111 based NIC who is able to take advantage of the full 1Gbps bandwidth by tweaking any other settings?
- Is this upgrade worth it? Will I be able to get better throughput, or should I be looking to upgrade to an appliance with better Intel NIC's?
- Is there a step by step instruction to upgrade the driver that anyone has followed that you could pls point me to? Is there an easy way to rollback if anything goes wrong?
At the moment, I only have the pfSense as my primary firewall/router and if anything goes wrong, working from home will be badly affected. So really keen to avoid that situation.
Any help/guidance is appreciated....
-
Hi,
you recently joined the team - if I see it right
-you won't find many members here in the forum who would recommend any type of Realtek, this is a FACT
-this j1900 based hardware alone is also a very borderline case for a 1Gig ISP
-I see it's a miniPC, ergo you can't really expand it with an intel based (ethernet controler) NICs - there is not enough space in it
-miniPCIe NIC, which is Intel based I only know this: http://www.commell.com.tw/Product/Peripheral/PCI%20Express%20mini%20card/MPX-350.htm-it is not worth "soldering" the Realtek driver, but there are fanatics who deal with similar things, I think just for gaining experience
(https://forum.netgate.com/topic/135850/official-realtek-driver-binary-1-95-for-2-4-4-release) - this is my opinion
(but that doesn't mean it's not impossible to "weld" from a FreeBSD OP kernel system to a newdriver****.ko to transfer to pfSense and stuff like that
)-I think this 600Mbps is pretty good, if you don't have better hardware, so it's risky to move on....
this is important -
@sinbox_pfs "At the moment, I only have the pfSense as my primary firewall/router and if anything goes wrong, working from home will be badly affected. So really keen to avoid that situation."I would first get a factory Netgate hardware or other more serious device and only then experiment...
https://www.pfsense.org/products/or
https://www.pcengines.ch/apu4d4.htmPS:
specifically recommended NICs (eth. controllers):Intel I340, I350, I210, i211
you will get rid of a lot of headaches, if you use theseV
-
I have recently done some benchmarking with j1900 and realtek 8111 and have found that under linux, iperf3 can do approximately 970Mbits on the same Lan.
Changing this to pfsense 2.4.5.p1 even with new driver never made it more than 600Mbit
So switching lan hardware is probably the only way to go. -
@DaddyGo thanks...all valid responses. I was considering the same mini-PCIE card, but at almost $100 it may make more sense for me to buy a new appliance altogether and use the current one for DR.
-
@netblues thats what I was worried about - i.e no appreciable increase in performance. I have seen some watchdog errors every now and then, which is probably the only reason one may want to consider changing the driver
-
As @netblues said, for example, it performs better under Linux, so you can build anything from this little unit (future use).
For NGFWs, on the other hand, the most important basis is NICs, thereafter followed by the other hardware elements...Come back to us when your new hardware ideas come true
-
@DaddyGo An opportunity to buy a used SG-2220 (https://www.netgate.com/solutions/pfsense/sg-2220.html) has just come up. Is that worth looking into as something which can provide the WAN throughput I need?
-
Hi,
My problem with this is as follows:
After the EOL period, we can’t know the amount of additional support, it certainly won't be like an active series device support.
The recommended (offered) SG-3100 though is ARM CPU based and has many unresolved issues, such as:
@bmeeks "The underlying root cause is poor C code programming practices scattered all over the Snort binary code (incorrect use of pointer casting is usually the cause of unaligned access memory bus errors). This bad C code accumulates in a large binary program such as Snort over the years. Because the code runs fine on genuine Intel hardware (due to the auto-fixup logic within Intel processors), there is no driving incentive on the upstream code maintainers/creators of Snort to invest the time and effort required to ferret out all the incorrect C code and fix it. It is not an easy task as a change you make in one place to fix the error can easily introduce a new bug in another part of the code that happens to reference the code area you changed. It turns into a mess of spaghetti code very quickly. And because the code runs fine on genuine Intel hardware, and the vast majority of users have Intel processors, the bad code lives on.
I am so familiar with this because the same issue has bitten pfSense with the ARM hardware in the SG-1000, SG-1100 and SG-3100 Netgate appliances. Bad C coding in a number of binary packages causes similar issues (Telegraph, Snort, Suricata, FRR and others)."
Maybe, if you can afford a SG-5100 the excellent choice will be in the long run.
-
Ok...so over the weekend, I picked up a used SG-2200 pfSense appliance, which comes with 2 Intel I350 chipset based NIC's. Thought it could serve as redundant/failover box, so may as well have one.
After hours of testing this weekend with the SG-2200, I was disappointed I'm still only averaging ~450 Mbps with the Intel NIC's on the WAN side. On the LAN side, I'm also only able to get ~600Mbps, which is actually quite low. Even the SG-1100's are meant to hit close to 900 Mbps on the LAN side as others have previously reported on this forum.
I re-built my entire network this weekend, so all testing was without any packages/overhead on the pfSense side. All testing was done using iPerf3 for LAN and/or dedicated Speed Test apps on the client side for WAN's. Of interesting note was very high usage (~80-90%) on the CPU on the SG-2200, whereas on my AliExpress box, the CPU has never spiked over 7-10%.
Could my Unifi US-24/US-24 POE switches be the bottleneck here? Considering they are all Gigabit (and no port configurations), I find that case to be unlikely. They were also factory reset so no VLAN tagging or any other overhead on them at the moment.
@DaddyGo At this rate, I'd rather build a custom box which is 10G capable rather than go for any other appliance.
-
Is it running at full speed? Check the CPU frequency shown on the dashboard. You should definietely see faster than 450Mbps in a local iperf test there.
Steve
-
There should not be any differences when testing from the wan or the lan side on iperf3.
I doubt its the unifys for sure..
You really need to establish a testing baseline in order to rule out various subtle (but critical) parameters). Use a third machine (pc) as a reference.
Run iperf from alibox to this machine and verfiy you get 900+mbits., either directly or through your switces. Then put sg2200 into testing and see what happens.
The only way to know its not a faulty cable, a bad lan port , a switch etc, or just plain misconfiguration. -
@sinbox_pfs said in Upgrading Realtek with alternate driver - Is it worth it?:
Unifi US-24
Hi,
I'm glad you found a used unit for your system, this is definitely a good starting point
BTW: this is definitely a configuration issue or a network building issue (physically problem cables or miss connection, etc.)
the Unifi US-24 switches are perfectly suitable not these devices cause the issue, the 10Gig in this environment is unnecessary.
-the SG-2200 knows everything what you want to achieve.
you can begin a step-by-step examination - where does the bottleneck live in your system,
do you have any drawings you can publish?have you already gone through this description?
https://docs.netgate.com/pfsense/en/latest/hardware/tuning-and-troubleshooting-network-cards.htmlit can also help, but treat it carefully:
I350 NIC Tunning loader_conf_local.txt -
@stephenw10 The CPU spikes temporarily to ~80% when running iPerf. Goes back to ~12-20% when idle
-
But what is the reported frequency?
Those devices should have powerd enabled or they can end up running at a much reduced speed.
You should see something like:
CPU Type Intel(R) Atom(TM) CPU C2358 @ 1.74GHz Current: 1411 MHz, Max: 1744 MHz 2 CPUs: 1 package(s) x 2 core(s) AES-NI CPU Crypto: Yes (inactive)Steve
-
@netblues said in Upgrading Realtek with alternate driver - Is it worth it?:
There should not be any differences when testing from the wan or the lan side on iperf3.
I doubt its the unifys for sure..
You really need to establish a testing baseline in order to rule out various subtle (but critical) parameters). Use a third machine (pc) as a reference.
Run iperf from alibox to this machine and verfiy you get 900+mbits., either directly or through your switces. Then put sg2200 into testing and see what happens.
The only way to know its not a faulty cable, a bad lan port , a switch etc, or just plain misconfiguration.I'm a bit challenged at the moment as I have no desktop with ethernet ports except a Mac Mini. I have a Dell XPS, Microsoft Surface and Macbook Pro 13" all of which lack physical Ethernet ports and hence I need to use dongles to test them out. I have had various issues with dongles, so I'm trying to avoid them for the time being until I can get hold of a reliable Thunderbolt > Ethernet Dock/dongle.
For now I re-ran all tests on my a Mac Mini 2018 which does have a 1G ethernet port and this is what I have found. Hope this covers all bases. If there is anything else I can test, please let me know.
LAN iPerf3 Tests:
On Aliexpress Box: [FTTP NTD WAN>Patch Panel>Aliexpress pfSense Box>Mac mini]
With pfSense as Client and Mac Mini as server, the network throughput is 586 Mbps (receiver) and 621 Mbps (sender);
With Mac Mini as Client and pfSense as server, the network throughput is 429 Mbps (receiver) and 436 Mbps (sender)
CPU usage during iPerf3 tests is ~ 22% and Idle is ~2%On SG-2220 [FTTP NTD WAN>US-24>Patch Panel>SG2220>Mac mini]
With pfSense as Client and Mac Mini as server, the network throughput is 865 Mbps (receiver) and 942 Mbps (sender);
However, With Mac Mini as Client and pfSense as server, the network throughput is back to 398 Mbps (receiver) and 398 Mbps (sender)!
CPU usage during iPerf3 tests is ~ 89-92% and Idle is ~12%WAN Speedtest.net tests:
On Aliexpress Box: pfSense > WAN [FTTP NTD WAN>Patch Panel>Aliexpress pfSense Box>Mac mini]
Speedtest ~585 Mbps via SpeedTest native App on Mac Mini to a known server location (~6Kilometers from my property)
Speedtest-CLI on pfSense box itself: Only 306.15 Mbps! (same server location selected) CPU was close to 95%On SG-2220 Appliance: pfs[FTTP NTD WAN>US-24>Patch Panel>SG2220>Mac mini]
Speedtest ~582 Mbps via SpeedTest native App on Mac Mini to a known server location (~6Kilometers from my property)
Speedtest-CLI on pfSense box itself: ~465 Mbps! (same server location selected) CPU was close to 95%Here's is all the things that I think I can rule out. Happy to be corrected:
- As mentioned previously, I have basically setup my network from scratch. There are no Firewall rules apart from the OOTB ones.
- cc: @DaddyGo, It is a new home built less than 2 yrs back and can confirm Cat 6a cables, patch panels etc.
- When I connect the Mac Mini directly to the NTD's ethernet port, I can get close to ~970-980 Mbps, so I think I can rule the Mac Mini as bottleneck
- So, If I have to assume, I'm getting the best possible output off the AliExpress box
So, what baffles me is the LAN results on the SG-2220 with Mac Mini as Client and pfSense as server. Is this where the bottleneck is?
Next steps is to try what @DaddyGo suggests above...
-
@stephenw10 On the SG-2220 it is:
CPU Type Intel(R) Atom(TM) CPU C2338 @ 1.74GHz
Current: 1400 MHz, Max: 2100 MHz
2 CPUs: 1 package(s) x 2 core(s)
AES-NI CPU Crypto: Yes (active)On the AliExpress box it is:
CPU Type Intel(R) Celeron(R) CPU N3160 @ 1.60GHz
Current: 1600 MHz, Max: 1601 MHz
4 CPUs: 1 package(s) x 4 core(s)
AES-NI CPU Crypto: Yes (inactive)powerd was Off on both boxes. You suggest I re-run the tests with this flipped back ok?
-
On the SG-2220 definitely. I imagine it never shows anything other than 1400 MHz? That's what happens on those without powerd enabled. You should see a significant performance improvement with it enabled.
Steve
-
@stephenw10 Tested on the SG-2220 with powerd enabled (followed by a reboot). CPU still spikes ~95% during iPerf (i.e when using SG-2200 as iPerf Server). And with similar throughput may I add...
-
Does the dashboard now show it running at full speed?
Any test where the 2220 is actually running iperf is not a good one. pfSense is not optimised as TCP terminator.
Really you need to test through it, with an iperf3 server on one interface and a client on the other. Running iperf3 on the 2220 will itself use a lot of CPU leaving far less for actually moving traffic.
Steve
