The Router Rumble
-
Without any way to replicate their tests or results, it's crap. Even if pfSense came out on top it would still be a crap article.
The fact that they tossed it on something and didn't do any tweaking or troubleshooting is exceptionally dumb. And in the case of the APU, they didn't even use an official pfSense image, they used some other image for what seems to be no discernible valid reason.
For the larger connection sizes they probably needed to increase the state table size and/or set the firewall optimization to aggressive, at a minimum.
I'd like to see their test replicated on hardware we actually sell, rather than something DIY.
-
Without any way to replicate their tests or results, it's crap. Even if pfSense came out on top it would still be a crap article.
The fact that they tossed it on something and didn't do any tweaking or troubleshooting is exceptionally dumb. And in the case of the APU, they didn't even use an official pfSense image, they used some other image for what seems to be no discernible valid reason.
For the larger connection sizes they probably needed to increase the state table size and/or set the firewall optimization to aggressive, at a minimum.
I'd like to see their test replicated on hardware we actually sell, rather than something DIY.
Would love to see you guys doing something more constructive than calling it a crap article and ignoring it. Right now I have to wondering if I have my own RCC-VE 2440 setup wrong and maybe missing some tweaks required, I don't have a stress environment so I do partially rely on the assumption that the broader pfsense community has helped make sure the defaults are reasonable.
They do list their testing methodology in their previous article, so if there are questions about how to reproduce hopefully you have contacted the author:
http://arstechnica.com/gadgets/2016/04/the-ars-guide-to-building-a-linux-router-from-scratch/Excluding if you want to call their actual test broken, loading an system and using it without much tweaking is actually very relevant test for many users. A lot of people looking at building their own system or using open source do not have the HW to build a stress environment to validate and tune their settings and normal usage would only occasionally hit the limits and issues this stress testing shows. What would be dumb is if there was a well documented set of recommendations for gigabit connections for pfsense and they didn't apply them. If that is the case hopefully you are working with the author to resolve this.
Have you guys contacted them to see if they would like to test your HW? Or to improve this article and resolve some of the issues you see?
-
We're not ignoring it. We've all read it. But without being able to replicate the test or the results, it's still crap, and using just one metric like that is also crap.
We have covered testing effectively pretty well, their test only covers a number of rapid connections, which does not relate well at all to real-world behavior. Their test uses nginx and apachebench, apparently, but aside from the config files the details are still light.
The original article also goes into detail about how they tweaked their firewall box in many ways, none of which they bothered to do with pfSense. So forgive me if I still call it crap, but it's crap. Apples and oranges. Meaningless comparisons, made worse by sloppy procedures.
Sure, pf itself is no speed demon, but one measurement alone does not make or break any firewall.
To pursue replicating the test on our own by guessing at details would only legitimize what was clearly a half-hearted attempt on their part. They need to do better.
If we can replicate a problem and find a fix, we'd be all over it.
-
Speaking about tweaks, I read setting net.pf.states_hashsize equal to the smallest power of 2 that is larger than or equal to your state table size and make almost a 3x-4x difference when dealing with lots of states. Having a default of 32k entries, this limit may not have been an issue in their tests, but it is possible.
Also these two settings that can make a difference with MSI-X NICs.
hw.igb.rx_process_limit="-1"
hw.igb.tx_process_limit="-1"https://forum.pfsense.org/index.php?topic=113496.msg631076#msg631076
-
I can't wait for the future FreeBSD SMP changes to the network stack. Should make PFSense freaking awesome with multi-core scaling.
-
ARS is a far cry from what it used to be.
That being said, it's still cool to bring this sort of thing into the public narrative. Pfsense rocked my world and it will rock many others' as well.
-
I can't wait for the future FreeBSD SMP changes to the network stack. Should make PFSense freaking awesome with multi-core scaling.
What changes?
pf has been SMP-friendly since FreeBSD 10: https://wiki.freebsd.org/WhatsNew/FreeBSD10#Networking_improvements
-
ARS is a far cry from what it used to be.
Oh? I've been a reader for 10+ years and I haven't noticed any general decline in quality. One author getting this one thing wrong does not mean their entire output is just as poor.
-
I can't wait for the future FreeBSD SMP changes to the network stack. Should make PFSense freaking awesome with multi-core scaling.
What changes?
pf has been SMP-friendly since FreeBSD 10: https://wiki.freebsd.org/WhatsNew/FreeBSD10#Networking_improvements
I was watching an interview with one of the FreeBSD network guys and he said they're looking at lock-less datastructures for the network stack. Instead of a single systemwide state table, there is a statetable per core, allowing each core to read/write knowing nothing else will touch its datastructures.
Going along with this is current useland network API calls currently don't know what core a network queue is attached to. This means if a user thread is running on Core 1 but the state for the traffic is on Core 0, you now have cross-core state access, which violates the above lock-less feature. So they need to add APIs to allow userland to find out what core a given network flow is attached to and process that flow only on the appropriate thread.
-
@KOM:
ARS is a far cry from what it used to be.
Oh? I've been a reader for 10+ years and I haven't noticed any general decline in quality. One author getting this one thing wrong does not mean their entire output is just as poor.
I'd say in the early 00's I first came across ARS and the information and quality was so far above everything else it seemed like it was run by aliens. My mind was blown. Then as the years went on it seemed to become more and more "accessible" and "watered down".
I bothered to make a forum account in the 10's. Yes, I was so blown away that I didn't even feel like I could communicate with the members for 10 years as well. Long story short we disagreed about a few things and they brow beat the heck out of me for it. 5 years later I was right about the AMD stuff we disagreed on. I nearly got banned for what I had to say about mp3's.
Anyway, I still read the website. I just don't participate in the forums no matter how badly I want to share my perspective. I also don't see it with the same reverence either. It's just another engadget or yahoo tech to me now.
-
I myself prefer meritocratic forums. Not a huge fan of democratic, everyone's opinion matters, everyone gets a trophy forums. But I do frequent support forums where the target audience are the general public and need help.
