Pfsense hardware for home

vicotnic

Is a SG-4860 enough for 250 Mbit openvpn throughput ?

whosmatt

@whosmatt:

Between those two I'd choose the X11SBA-LN4F. ECC isn't really necessary for an application like pfsense. QuickAssist support is on the radar but won't help you now. I'd choose the N3700 for the higher turbo clock speed and additional cores.

thank you very much! Did you test the openvpn performance?

I don't own the N3700, so no. I'm just going on what I know about OpenVPN. The N3700 is a faster CPU than the C2358 and thus will provide better OpenVPN performance. I can't say in absolute terms how well it will perform, though.

Paint

@vicotnic:

Is a SG-4860 enough for 250 Mbit openvpn throughput ?

take a look here: https://forum.pfsense.org/index.php?topic=115673.0

I dont think it will safely push that much bandwidth. Based on the PassMark benchmark, its about half the capacity of the i7-4510U - I can push about 300mbps OpenVPN theoretically when my CPU is set to CMax (turbo at 3.0ghz)

i7-4510U PassMark: https://www.cpubenchmark.net/cpu.php?cpu=Intel+Core+i7-4510U+%40+2.00GHz
C2558 Atom CPU PassMark: http://www.cpubenchmark.net/cpu.php?cpu=Intel+Atom+C2558+%40+2.40GHz

user09

@user09:

Intel Pentium Processor N3700

X11SBA-LN4F Supermicro

8 GB S0-DDR3

Kingston SV300S37A/60G SSDNow V300 interne SSD-Festplatte 60GB

is it possible to use Snort with this config?

Paint

@user09:

@user09:

Intel Pentium Processor N3700

X11SBA-LN4F Supermicro

8 GB S0-DDR3

Kingston SV300S37A/60G SSDNow V300 interne SSD-Festplatte 60GB

is it possible to use Snort with this config?

Yes, snort and pfblockerng will work and love the 8gb of ram

mauroman33

@user09:

@user09:

Intel Pentium Processor N3700

X11SBA-LN4F Supermicro

8 GB S0-DDR3

Kingston SV300S37A/60G SSDNow V300 interne SSD-Festplatte 60GB

is it possible to use Snort with this config?

I got a miniPC with the Celeron N3150 as home router with a fiber connection 100/100
I've added 8GB RAM and a 120GB SSD (not easy to find less). Total cost was about $220.
It has two Realtek NICs, maybe I'm lucky but I've never seen lost packets in four months.
I'm really satisfied, it's capable to run snort, pfBlocker and the OpenVpn client to PIA smooth as silk.
No problem to reach the full line speed in OpenVPN.
Intel N3700 it's a little more performant than N3150 so I think you should easily reach 130Mbs in OpenVPN.

MikeV7896

Just a note with the N3700… while it doesn't have QuickAssist support, it does still have AES-NI support. I did some digging and see that about 6 months ago, OpenVPN added support for AES-GCM (ticket 301), so if you can set it up to use that, you might find much faster VPN performance. Not sure if it's set to take advantage of the Intel AES-NI or not, but it might help.

If pfSense doesn't have that option for OpenVPN, then going with IPSEC using AES-GCM should also be accelerated. Of course, that's a much larger change to be making.

Paint

@virgiliomi:

Just a note with the N3700… while it doesn't have QuickAssist support, it does still have AES-NI support. I did some digging and see that about 6 months ago, OpenVPN added support for AES-GCM (ticket 301), so if you can set it up to use that, you might find much faster VPN performance. Not sure if it's set to take advantage of the Intel AES-NI or not, but it might help.

If pfSense doesn't have that option for OpenVPN, then going with IPSEC using AES-GCM should also be accelerated. Of course, that's a much larger change to be making.

currently pfSense only supports AES acceleration via IPsec, not through OpenVPN. I believe the developers are looking to add support for AES with OpenVPN on the next release.

MikeV7896

@Paint:

@virgiliomi:

Just a note with the N3700… while it doesn't have QuickAssist support, it does still have AES-NI support. I did some digging and see that about 6 months ago, OpenVPN added support for AES-GCM (ticket 301), so if you can set it up to use that, you might find much faster VPN performance. Not sure if it's set to take advantage of the Intel AES-NI or not, but it might help.

If pfSense doesn't have that option for OpenVPN, then going with IPSEC using AES-GCM should also be accelerated. Of course, that's a much larger change to be making.

currently pfSense only supports AES acceleration via IPsec, not through OpenVPN. I believe the developers are looking to add support for AES with OpenVPN on the next release.

Ok, good to know. But that's still a plus that the N3700 will offer, when the update comes. That will bring even more value to the N3700 system then!

Pippin

currently pfSense only supports AES acceleration via IPsec, not through OpenVPN

You sure?

Do you know that OpenSSL, which is part of OpenVPN, will automatically use AES-NI when available on SOC?
No need to enable anything in pfSense in that case, as in, do not load any module, to take advantage of it.

It does very well support AES through OpenVPN, no doubt about it.
The problem is more the hashing that takes place which will be "kind of history" when GCM comes with OpenVPN 2.4.

Just do

env OPENSSL_ia32cap=0 openssl speed -elapsed -evp aes-256-cbc

for speedtest without AES-NI, and

openssl speed -elapsed -evp aes-256-cbc

with AES-NI.
See the (big) difference?

Paint

@Pippin:

currently pfSense only supports AES acceleration via IPsec, not through OpenVPN

You sure?

Do you know that OpenSSL, which is part of OpenVPN, will automatically use AES-NI when available on SOC?
No need to enable anything in pfSense in that case, as in, do not load any module, to take advantage of it.

It does very well support AES through OpenVPN, no doubt about it.
The problem is more the hashing that takes place which will be "kind of history" when GCM comes with OpenVPN 2.4.

Just do
env OPENSSL_ia32cap=0 openssl speed -elapsed -evp aes-256-cbc
for speedtest without AES-NI, and
openssl speed -elapsed -evp aes-256-cbc
with AES-NI.
See the (big) difference?

You may be right, but I dont see the difference:

[2.3.3-DEVELOPMENT][root@pfSense.pf.lan]/root: env OPENSSL_ia32cap=0 openssl speed -elapsed -evp aes-256-cbc
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-cbc for 3s on 16 size blocks: 1714069 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 64 size blocks: 1577402 aes-256-cbc's in 3.01s
Doing aes-256-cbc for 3s on 256 size blocks: 1283958 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 1024 size blocks: 744736 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 8192 size blocks: 149773 aes-256-cbc's in 3.00s
OpenSSL 1.0.1s-freebsd  1 Mar 2016
built on: date not available
options:bn(64,64) rc4(8x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc       9141.70k    33563.84k   109564.42k   254203.22k   408980.14k


[2.3.3-DEVELOPMENT][root@pfSense.pf.lan]/root: openssl speed -elapsed -evp aes-256-cbc
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-cbc for 3s on 16 size blocks: 1725942 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 64 size blocks: 1580980 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 1281281 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 1024 size blocks: 740019 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 8192 size blocks: 148567 aes-256-cbc's in 3.00s
OpenSSL 1.0.1s-freebsd  1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc       9205.02k    33727.57k   109335.98k   252593.15k   405686.95k

Below is a screenshot of my pfSense GUI showing I have AES functionality support for my CPU:

Capture.JPG_thumb

Pippin

Do you have any Cryptographic Hardware module loaded in pfSense?
System/ Advanced/ Miscellaneous
Set to none and reboot.

Without AES-NI

env OPENSSL_ia32cap=0 openssl speed -elapsed -evp aes-256-cbc
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc      23990.78k    28635.43k    29824.77k    75725.14k    76436.82k

With AES-NI

openssl speed -elapsed -evp aes-256-cbc
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc     132721.93k   211522.30k   244506.28k   254213.80k   256557.76k

With aesni.ko module loaded, meaning Cryptographic Hardware is set to AES-NI

type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc       4643.26k    17799.15k    57819.31k   136405.67k   218895.70k

Paint

@Pippin:

Do you have any Cryptographic Hardware module loaded in pfSense?
System/ Advanced/ Miscellaneous
Set to none and reboot.

Without AES-NI

env OPENSSL_ia32cap=0 openssl speed -elapsed -evp aes-256-cbc
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc      23990.78k    28635.43k    29824.77k    75725.14k    76436.82k

With AES-NI

openssl speed -elapsed -evp aes-256-cbc
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc     132721.93k   211522.30k   244506.28k   254213.80k   256557.76k

Yes, I have Cryptographic Hardware set to AES-NI CPU-based Acceleration
OpenVPN's Hardware Crypto is set to BSD Cryptodev Engine

Are these settings correct?

[2.3.3-DEVELOPMENT][root@pfSense.pf.lan]/root: dmesg | grep -i aes
[15] aesni0: <aes-cbc,aes-xts,aes-gcm,aes-icm> on motherboard
  Features2=0x7fdafbbf<sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,sdbg,fma,cx16,xtpr,pdcm,pcid,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,xsave,osxsave,avx,f16c,rdrand></sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,sdbg,fma,cx16,xtpr,pdcm,pcid,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,xsave,osxsave,avx,f16c,rdrand></aes-cbc,aes-xts,aes-gcm,aes-icm>

[2.3.3-DEVELOPMENT][root@pfSense.pf.lan]/root: cryptostats
10953025 symmetric crypto ops (0 errors, 0 times driver blocked)
0 key ops (0 errors, 0 times driver blocked)
0 crypto dispatch thread activations
0 crypto return thread activations

Pippin

Update my previous post to include loaded module in pfSense.

As you can see from the results, the best is achieved when selecting no crypto in pfSense.
It loads a module, aesni.ko, which is not needed in case one has a CPU with AES-NI support.

In OpenVPN you dont need to set anything either, as said, OpenSSL will use AES-NI whenever its available on SOC.

MikeV7896

AES-GCM is where AES-NI really shines, not so much AES-CBC. Check out the difference here…

Without AES-NI...

: env OPENSSL_ia32cap=0 openssl speed -elapsed -evp aes-256-gcm
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-gcm for 3s on 16 size blocks: 4318531 aes-256-gcm's in 3.01s
Doing aes-256-gcm for 3s on 64 size blocks: 1237576 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 256 size blocks: 324193 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 1024 size blocks: 82041 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 8192 size blocks: 10292 aes-256-gcm's in 3.00s

With AES-NI…

: openssl speed -elapsed -evp aes-256-gcm
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-gcm for 3s on 16 size blocks: 20466923 aes-256-gcm's in 3.01s
Doing aes-256-gcm for 3s on 64 size blocks: 8766278 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 256 size blocks: 2775125 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 1024 size blocks: 748960 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 8192 size blocks: 95348 aes-256-gcm's in 3.00s

The difference is quite a bit more visible… 5 to 9 times faster depending on the block size, and I have AES-NI selected under System > Advanced > Miscellaneous > Cryptographic Hardware.

AES-CBC still needs the hash to be calculated for authentication, so it might be fast to encrypt, but it's lost in computing the hash to go along with it... AES-GCM has that all wrapped in so there's no additional processing needed. That's why it can be so much faster when accelerated.

But AES-GCM is not available as an option in the pfSense OpenVPN settings right now... so hopefully Paint is correct regarding support coming in a future version. I don't see anything in the OpenVPN category in Redmine that asks for AES-GCM support to be added, but maybe it happens with one of the other updates/fixes there?

Pippin

AES-GCM is where AES-NI really shines, not so much AES-CBC.

Still, AES-CBC does benefit compared to not using AES-NI, I would say, use it if have it.

AES-GCM has that all wrapped in so there's no additional processing needed

Yes, no separate authentication.

I have AES-NI selected under System > Advanced > Miscellaneous > Cryptographic Hardware

Since version 1.0+ OpenSSL automatically detects if AES-NI support is available, not any module is needed to activate it.

When loading the aesni.ko module in WebUI, crypto will take place in kernel, not in hardware.

When not loading the aesni.ko module in WebUI, crypto takes place on crypto hardware if there is any, not in kernel.

But AES-GCM is not available as an option in the pfSense OpenVPN settings right now

OpenVPN will add AES-GCM (AEAD) support in version 2.4 so pfSense has to wait until OpenVPN releases version 2.4.

garyd9

@Pippin:

I have AES-NI selected under System > Advanced > Miscellaneous > Cryptographic Hardware

…
When loading the aesni.ko module in WebUI, crypto will take place in kernel, not in hardware.

When not loading the aesni.ko module in WebUI, crypto takes place on crypto hardware if there is any, not in kernel.

I'm confused about this. Are you saying, for openVPN specifically, that turning OFF the "crypto h/w" option in pfsense results in hardware crypto working, but turning ON the h/w option in pfsense results in h/w crypto NOT working?

Thanks
Gary

Pippin

Yes, we talk about OpenVPN (which uses OpenSSL for the crypto part).
And yes, that`s what I'm saying :)

Will try to explain my findings:

System/ Advanced/ Miscellaneous
Cryptographic Hardware: None
^^^
–- When selecting an option a kernel module gets loaded, but we don`t need
–- it because OpenSSL will use AES-NI on SOC automatically.

VPN/ OpenVPN/ Servers--> Edit Server
Hardware Crypto: No Hardware Crypto Acceleration
^^^
--- When selecting an option then it will set the directive "engine xxxxx" in server config file, but we
–- dont need it because OpenSSL will use AES-NI on SOC automatically. --- If it should be selectable? Yes I think so because when theres no AES-NI on SOC available, one could
--- possibly benefit from the kernel module that gets loaded by one of these options.
--- In that case Cryptographic Hardware needs to be set appropriately.

Now the test to confirm and for who is interested to compare to mine:
Make the two settings as written above, selecting None and No Hardware Crypto Acceleration.
For the first setting above to take immediate effect without a restart, do

kldunload aesni.ko

in console.
Close the WebUI as it will influence the test and if can disable installed packages.
The following however I did with packages running because didn`t feel like disabling them :)

Then this command will tell OpenSSL to not use AES-NI support on SOC
1.

env OPENSSL_ia32cap=0 openssl speed -elapsed -evp aes-256-cbc
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc      23756.57k    28643.61k    29828.52k    75748.35k    76480.51k

And then this command will tell OpenSSL to use AES-NI on SOC
2.

openssl speed -elapsed -evp aes-256-cbc
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc     125684.70k   211853.44k   244509.53k   254263.64k   256703.33k

Compare 1 and 2 and see the improvement.

Now to compare against aesni.ko loaded, do

kldload aesni.ko

This command will load the AES-NI kernel module, its the same as selecting it in the WebUI, "Cryptographic Hardware: None" but on console its more convenient as one no need to restart pfSense.
To see if it`s loaded do

kldstat

To unload it do

kldunload aesni.ko

Do the same as in 1 and 2
3.

env OPENSSL_ia32cap=0 openssl speed -elapsed -evp aes-256-cbc
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc       4026.26k    17522.26k    56889.00k   135259.14k   218065.68k

openssl speed -elapsed -evp aes-256-cbc
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc       4689.60k    17543.15k    58354.77k   136928.60k   217352.87k

In 3 and 4 we can see that there is no difference because aesni.ko is loaded and therefore AES-NI on SOC is not used.
We also see that 2, AES-NI on SOC gives best result.

garyd9

I hope you understand that I believe you, but I wanted to make sure that I understood you. Obviously, what you describe sounds like it's working backwards.

Enable h/w AES support and it's NOT used, but if you disable h/w AES support, then it IS used. Backwards, right?

In the linux world, such a kernel module would probably provide canned routines that make use of AES instructions. Afterall, it wouldn't make much sense to have an entire kernel module just to enable the instructions. So, the only way I can make sense of your experience is that, perhaps, the openVPN code has more efficient use of the AES instructions when compared to the canned routines in the kernel module (for the usage that openVPN makes of them) (assuming my guess on what the module does is correct.) Of course, THAT doesn't make sense either, because even poorly written canned routines using AES should perform at a significantly different speed than no use of AES-NI to begin with.

Is the AES-NI kernel module just an emulator and not really h/w support?

I wonder if the same backwards behavior occurs with IPSec.. (I haven't configured VPN yet, so haven't had a chance to play with it.)

Sadly, I'm not familiar with BSD-based kernels whatsoever, so I can't really contribute anything more than startled exclamations and questions based on knowledge of a different platform.

Pippin

@garyd9:

I hope you understand that I believe you, but I wanted to make sure that I understood you. Obviously, what you describe sounds like it's working backwards.
Enable h/w AES support and it's NOT used, but if you disable h/w AES support, then it IS used. Backwards, right?

I think the thing to understand is that there is nothing needed to enable AES-NI support because OpenSSL will detect/support it automatically.
It`s possible that the built-in code in OpenSSL is doing a better job then the kernel module.
An answer I got a while back from a OpenVPN dev:
–Using AES-NI via kernel cryptodev is almost always a bad idea - because
--it is much slower than just using the same AES-NI instructions in openssl
--userland ("same CPU opcodes, less jump-to-kernel-and-back").

--So "just don't do that"..."

the openVPN code has more efficient use of the AES instructions…........
.............
Of course, THAT doesn't make sense either

It`s actually the OpenSSL code, OpenVPN just makes a call and gets a encrypted/decrypted buffer back.
But i think it can make sense, see "same CPU opcodes, less jump-to-kernel-and-back"

Is the AES-NI kernel module just an emulator and not really h/w support?

About deeper/inner workings my knowledge is limited, only just about seeing the big picture and my results.
Most of it is from trying out myself and trying to filter correct from incorrect info from reading.

I wonder if the same backwards behavior occurs with IPSec..

Never used IPsec, it doesn`t see userland i think? That could possibly mean aesni.ko needs to be loaded?
If IPsec requires aesni.ko module and one also uses OpenVPN then i can imagine that OpenVPN performance is, to some degree, degraded, crypto-wise.
I would think so because when aesni.ko is loaded the OpenSSL built-in engine is not used.

One can repeat the above test and see what works best…