10G NAT/Firewall performance problems

farmwald

I have a 10Gbps fiber connection. Currently, I use a UBNT EdgeRouter Infinity router/firewall, which works quite well. I get 4Gbps down, 5.5 Gbps up with Speedtest, which seems about right.
I'm testing other firewalls, specifically OpenWRT, OPNSense, and PFSense on some medium to good performance PCs (i3-8100 and i7-7xxx) with 32GB of memory.
For straight routing (no firewall no NAT), all of the firewalls give me well over 6 Gbps - probably limited by the test rather than the router - on all of the PCs I've tested. This is fine.
When I use Speedtest (i.e., using NAT and a firewall), I get very odd results:
OpenWRT 18.06.2 ~100 Mbps down, > 4 Gbps up
OPNSense 19.1 (and 18.7) - ~100 Mbps down, > 4 Gbps up
PfSense CD 2.4.4r1 - 4 Gbps down, > 4 Gbps up
These are fresh installs, with no tweaks or tuning. The numbers are consistent across different PCs. I'm using an Intel X540-T2 for all tests and I've tested 3 PCs with every firewall, so it's not hardware dependent.
Clearly, there is a problem with my use of OPNSense and OpenWRT, but I don't know what to try to fix this.
Why does PFSense work out-of-the-box, and nothing else?
Any suggestions?

farmwald

Just to be clear, PFSense performance is great, but I'm dependent on Wireguard, and, for now, there seems to be some religious feeling against Wireguard in the PFSense developers.
All of the other router/firewalls I'm using/testing make using Wireguard easy (OPNSense, OpenWRT, and EdgeRouter.)
So, I'm hoping for some help understanding why OPNSense and OpenWRT have 1/30 the firewall/NAT performance of PFSense and EdgeOS on the same hardware.
Also, the EdgeRouter Infinity works in all respects except that it's expensive ($1600) and has a steeper learning curve.
I've installed 10G fiber to 2500 homes (more soon) and am looking for a cheaper solution for the 10G customers, preferably open source.

Rico

You really ask in the pfSense forum why others performance is bad?

-Rico

Grimson

If you that dependent on Wireguard you can always create a package yourself: https://docs.netgate.com/pfsense/en/latest/development/index.html#pfsense-package-development

chrismacmahon

I cannot speak to why it won't work on other OS'es.

We take pride in our work, and strive for it to 'just work' out of the box.

We have taken this to the next level with our new TNSR product. If you are wanting to route at the speeds you are talking about out of the box (over 10gbps), please drop us a line at sales@netgate(dot)com.

Wireguard, we are not against it.
As @jimp said in Installing WireGuard VPN:

It will never be a "high priority feature" until they actually make a proven secure/stable release.

https://www.wireguard.com/#about-the-project

About The Project
Work in Progress

WireGuard is not yet complete. You should not rely on this code. It has not undergone proper degrees of security auditing and the protocol is still subject to change. We're working toward a stable 1.0 release, but that time has not yet come. There are experimental snapshots tagged with "0.0.YYYYMMDD", but these should not be considered real releases and they may contain security vulnerabilities (which would not be eligible for CVEs, since this is pre-release snapshot software). If you are packaging WireGuard, you must keep up to date with the snapshots.

However, if you're interested in helping out, we could really use your help and we readily welcome any form of feedback and review. There's currently quite a bit of work to do on the project todo list, and the more folks testing this out, the better.

It may be fast, but that means nothing with all of those disclaimers around it.

Once the product is a bit more mature, I'm sure we will have it incorporated.

farmwald

Yes, I'd like to understand why PFSense is so much better (30x) than OPNSense, given the same base OS. Someone must have fixed some problem with OpenBSD that OPNSense hasn't dealt with, and I'm hoping that they will see some value in responding.

Failing that, maybe I can add some support for Wireguard, which is why I can't use PFSense.
Of course, given the near-religious responses, I've observed with other PFSense posts (especially on WIreguard), I don't expect much.

Grimson

@farmwald said in 10G NAT/Firewall performance problems:

Yes, I'd like to understand why PFSense is so much better (30x) than OPNSense, given the same base OS.

Nope, not the same base OS. pfSense uses FreeBSD instead of OpenBSD.

chrismacmahon

Nope, not the same base OS. pfSense uses FreeBSD instead of OpenBSD.

They are also using HardendBSD...

farmwald

@grimson
Thanks, I didn't realize that. I suppose it's likely that the problem is OpenBSD vs FreeBSD. 30x is a pretty big number, though so it must be a pretty serious problem with OpenBSD.

By the way, I'd say that Wireguard is pretty mature, probably more secure than alternatives (due to the vastly smaller and well-examined code base), and it is substantially faster (3-10x in my tests across a wide range of processors), and much easier to understand and set up.
I think the disclaimers are overstated at this point but were probably justified a few years ago.
I'd like to think that "customers" had a choice. If I could help, I would, but I have no experience with BSD or PFSense development. If someone is able and willing to port it, I'm willing to contribute.

stephenw10

@chrismacmahon said in 10G NAT/Firewall performance problems:

They are also using HardendBSD...

Which is actually FreeBSD. OPNSense is not using OpenBSD unless things have dramatically changed since I last tested it. Which was admittedly a while ago.

Steve

stephenw10

The pfSense devs are not against Wireguard in any way except that it wasn't really ready at the last review.

https://news.ycombinator.com/item?id=19187694

Steve

Grimson

@stephenw10 said in 10G NAT/Firewall performance problems:

Which as actually FreeBSD. OPNSense is not using OpenBSD unless things have dramatically changed since I last tested it. Which was admittedly a while ago.

Steve

IIRC they were thinking about switching to OpenBSD the last time I looked at their page. Which was a bit over 2 years ago, as I currently see no reason to follow their progress.

farmwald

I'm confused too. I did some web searches and pfsense and opnsense both claim to be based on FreeBSD.
PFSense 2.2.4 - FreeBSD 11.2-RELEASE-p4 (wth backports from HardenedBSD.)
OPNSense 19.1 - HardenedBSD 11.2

So maybe there is an issue with HardenedBSD 11.2 vs FreeBSD 11.2.

Grimson

TBH I doubt anyone here is interested in fixing performance issues with OPNSense, this is something you have to take up with them.

farmwald

Set mss to 1300. 25x faster (2.5 Gbps) download.
So OPNSense fix was easy, going to try the same fix on OpenWRT.

I guess PFSense has better defaults.

I'm quite serious about being willing to make financial contributions to Wireguard port to PFSense. PFSense seems "better" than OPNSense (after using it for a day), but I really need Wireguard.
I had lots of installation problem with OPNSense, but no problems with PFSense. Generally, PFSense seems a bit more serious and professional.

Grimson

@farmwald said in 10G NAT/Firewall performance problems:

I'm quite serious about being willing to make financial contributions to Wireguard port to PFSense.

https://forum.netgate.com/category/30/bounties good luck.