[Many Pics] My new silent firewall build
-
If someone want better stuff but don't want to pay too much for something like Supermicro C2x58 Rangeley CPU platform, you can consider the Supermicro X11SBA-LN4F ITX board, this board is using Intel N3700 CPU which supports AES-NI instruction, good for those who wants better encryption speed.
Please note there are some known issues with this board that are actually for a longer time
will be discussed here in the forum. LinkIt is also available as a fully bare bone from Supermicro that is really looking awesome!
Super Micro Supermicro SuperServer E200-9B Server SYS-E200-9BOtherwise thank you and +1 from me for the picture rich thread about that board and the measuring you
have done. 940 Mbit/s + the overhead + the NAT and firewall rules workload and you got really 1 GBit/s
please don´t forget this! And each installed packet and running service is also "eating" some CPU power
and slows down the entire pfSense box a little bit.Edward, how many RAM you were inserting in that box?
Did you enable PowerD (hi adaptive or adaptive)?
If you are using a mSATA you should enable TRIM support for that machine too.
With 8 GB RAM you could try out to high up the mbuf size from that machine to 1000000.Forgot to mention, I did OpenSSL test (by the internal testing command) with AES-256-CBC encryption algorithm, it's showing about 120-140Mbps, for a CPU without AES-NI, I can't complain too much about it.
Thats not so much but also enough to upload from outside photos and other things with ease.
-
@BlueKobold:
If someone want better stuff but don't want to pay too much for something like Supermicro C2x58 Rangeley CPU platform, you can consider the Supermicro X11SBA-LN4F ITX board, this board is using Intel N3700 CPU which supports AES-NI instruction, good for those who wants better encryption speed.
Please note there are some known issues with this board that are actually for a longer time
will be discussed here in the forum. LinkIt is also available as a fully bare bone from Supermicro that is really looking awesome!
Super Micro Supermicro SuperServer E200-9B Server SYS-E200-9BOtherwise thank you and +1 from me for the picture rich thread about that board and the measuring you
have done. 940 Mbit/s + the overhead + the NAT and firewall rules workload and you got really 1 GBit/s
please don´t forget this! And each installed packet and running service is also "eating" some CPU power
and slows down the entire pfSense box a little bit.Edward, how many RAM you were inserting in that box?
Did you enable PowerD (hi adaptive or adaptive)?
If you are using a mSATA you should enable TRIM support for that machine too.
With 8 GB RAM you could try out to high up the mbuf size from that machine to 1000000.Forgot to mention, I did OpenSSL test (by the internal testing command) with AES-256-CBC encryption algorithm, it's showing about 120-140Mbps, for a CPU without AES-NI, I can't complain too much about it.
Thats not so much but also enough to upload from outside photos and other things with ease.
wow…..the X11SBA-LN4F discussion is long, need to have some time to digest :P
For my system, I installed 4GB x 1 DDR3L. A bit stupid at the very beginning, I was trying to use my old laptop memory but the board doesn't even POST! Then I asked and got a notice from seller that this board accepts ONLY DDR3L SODIMM, but not normal 1.5V SODIMM, so I bought a 4GB DDR3L SODIMM, leaving one blank slot for future. This board supports 8GB max, but to be honest I believe 4GB is already more than enough for me.During my test, I disabled PowerD, but in real world application I've enabled "hi adaptive", I tried "adaptive" and it looks not reacting fast enough so I choose the former one. But during the first day of use, to confirm there will be no heat load issue I disabled PowerD and everything looks good.
mbuf already being set to 1M, and from my tests you can see that NAT never eats > 60% so I do have plenty of remaining cpu power for other packages.
-
mbuf already being set to 1M, and from my tests you can see that NAT never eats > 60% so I do have plenty of remaining cpu power for other packages.
If you are using PPPoE only one single CPU core is in usage, if this will be changed at one day, it will
be more smooth and liquid running as I see it right. The PowerD is also there fore that the CPU frequency
is not freezing and will be used only at some MHz instead of the highest available frequency if this is needed. -
@BlueKobold:
mbuf already being set to 1M, and from my tests you can see that NAT never eats > 60% so I do have plenty of remaining cpu power for other packages.
If you are using PPPoE only one single CPU core is in usage, if this will be changed at one day, it will
be more smooth and liquid running as I see it right. The PowerD is also there fore that the CPU frequency
is not freezing and will be used only at some MHz instead of the highest available frequency if this is needed.In Hong Kong, unless people living in very remote area or building with very old infrastructure, most of our broadband using FTTH/FTTB with fiber/CAT5 to home, the last time I used PPPoE was about 6 years ago (FTTH for that building was built after using PPPoE for a few months) 8)
For commercial, other than those for backup purposes, no one is using PPPoE anymore. -
For commercial, other than those for backup purposes, no one is using PPPoE anyone.
Cool! Then you will be getting out of that single core using and all core can be used to work on that
WAN interface. -
@BlueKobold:
For commercial, other than those for backup purposes, no one is using PPPoE anymore.
Cool! Then you will be getting out of that single core using and all core can be used to work on that
WAN interface.Yep, but with the fast development of internet in the country, most people using 300-1000Mbps broadband, and thus we are always chasing faster hardware for firewall/routers :)
-
BTW I'm now still using 4GB USB memory running as nanobsd, thinking about the re-use of 16GB old pci-e ssd from Asus EEEPC for full install.
I actually wondering, why we can't use HAVP + Proxy with nanobsd with more memory as cache? -
Since you mentioned the Supermicro X11SBA-LN4F ITX and I have a 'hardware repaired one' (see thread posted earlier - the long one), I would like to run the OpenSSL test that you did on yours. Can you give the syntax (assuming it's built into pfsense / FreeBSD?
Just curious as to how the two boards stack up with AES-NI and without?
-
Since you mentioned the Supermicro X11SBA-LN4F ITX and I have a 'hardware repaired one' (see thread posted earlier - the long one), I would like to run the OpenSSL test that you did on yours. Can you give the syntax (assuming it's built into pfsense / FreeBSD?
Just curious as to how the two boards stack up with AES-NI and without?
I did the test with reference to this document:
https://doc.pfsense.org/index.php/Are_cryptographic_accelerators_supportedOf course this is not an accurate test, the most accurate one should be doing this with 2 clients, but I don't have time so trying to use this as a simple reference.
-
Since you mentioned the Supermicro X11SBA-LN4F ITX and I have a 'hardware repaired one' (see thread posted earlier - the long one), I would like to run the OpenSSL test that you did on yours. Can you give the syntax (assuming it's built into pfsense / FreeBSD?
Just curious as to how the two boards stack up with AES-NI and without?
I did the test with reference to this document:
https://doc.pfsense.org/index.php/Are_cryptographic_accelerators_supportedOf course this is not an accurate test, the most accurate one should be doing this with 2 clients, but I don't have time so trying to use this as a simple reference.
Here's what I came up with (modified to 256 from the 128 command since you stated 256)….
$ openssl speed -evp aes-256-cbc -engine cryptodev
engine "cryptodev" set.
Doing aes-256-cbc for 3s on 16 size blocks: 947833 aes-256-cbc's in 0.32s
Doing aes-256-cbc for 3s on 64 size blocks: 945487 aes-256-cbc's in 0.36s
Doing aes-256-cbc for 3s on 256 size blocks: 772576 aes-256-cbc's in 0.21s
Doing aes-256-cbc for 3s on 1024 size blocks: 457823 aes-256-cbc's in 0.20s
Doing aes-256-cbc for 3s on 8192 size blocks: 91829 aes-256-cbc's in 0.03s
OpenSSL 1.0.1l-freebsd 15 Jan 2015
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 47345.41k 168378.90k 937621.12k 2307991.39k 24072421.38k -
Yeah….the AES-NI contributes a lot, can you run it again without crypto engine? I would like to compare, from some other online examples, the AES-NI speeds up about 5-10x
-
Yeah….the AES-NI contributes a lot, can you run it again without crypto engine? I would like to compare, from some other online examples, the AES-NI speeds up about 5-10x
Something wrong. Numbers are as good or better. Do I need to turn off AES-NI in the settings menu?
Edit: Turned off AES-NI in the Advanced menu but no difference (I didn't reboot - like my current 98 days uptime). Anyone have thoughts on why no change?
$ openssl speed -evp aes-256-cbc
Doing aes-256-cbc for 3s on 16 size blocks: 949961 aes-256-cbc's in 0.38s
Doing aes-256-cbc for 3s on 64 size blocks: 968692 aes-256-cbc's in 0.25s
Doing aes-256-cbc for 3s on 256 size blocks: 793691 aes-256-cbc's in 0.31s
Doing aes-256-cbc for 3s on 1024 size blocks: 456773 aes-256-cbc's in 0.19s
Doing aes-256-cbc for 3s on 8192 size blocks: 91937 aes-256-cbc's in 0.05s
OpenSSL 1.0.1l-freebsd 15 Jan 2015
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 40531.67k 247985.15k 650191.67k 2494589.61k 16067155.29k -
Try not to use -evp option, it looks like everything with this option will trigger the hardware engine. And your CPU is quad core, you can actually put a "-multi 4" as the option to run 4 encryption together.
But that's quite interesting, because I owned a higher end processor, the Avoton C2550, and the number with 4 threads together is half of your single thread. Not sure if this is the problem with the OpenSSL on my Linux server (yes, this is not a pfSense machine). I suppose my Avoton should be running a lot faster compared with your N3700. -
$ openssl speed aes-256-cbc
Doing aes-256 cbc for 3s on 16 size blocks: 5467107 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 64 size blocks: 1562852 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 256 size blocks: 403469 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 1024 size blocks: 254859 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 8192 size blocks: 32236 aes-256 cbc's in 3.00s
OpenSSL 1.0.1l-freebsd 15 Jan 2015
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256 cbc 29157.90k 33340.84k 34429.35k 86991.87k 88025.77kWith -multi 4 added on hardware (slower than single thread??)….
System
$ openssl speed -multi 4 -evp aes-256-cbc -engine cryptodev
engine "cryptodev" set.
Forked child 0
Forked child 1
+DT:aes-256-cbc:3:16
Forked child 2
+DT:aes-256-cbc:3:16
+DT:aes-256-cbc:3:16
+DT:aes-256-cbc:3:16
+R:836144:aes-256-cbc:3.000000
+R:824538:aes-256-cbc:3.000000
+DT:aes-256-cbc:3:64
+DT:aes-256-cbc:3:64
+R:857528:aes-256-cbc:3.000000
+R:863606:aes-256-cbc:3.000000
+DT:aes-256-cbc:3:64
+DT:aes-256-cbc:3:64
+R:811091:aes-256-cbc:3.000000
+R:787191:aes-256-cbc:3.000000
+DT:aes-256-cbc:3:256
+DT:aes-256-cbc:3:256
+R:838909:aes-256-cbc:3.000000
+R:814793:aes-256-cbc:3.000000
+DT:aes-256-cbc:3:256
+DT:aes-256-cbc:3:256
+R:657543:aes-256-cbc:3.000000
+DT:aes-256-cbc:3:1024
+R:671720:aes-256-cbc:3.000000
+DT:aes-256-cbc:3:1024
+R:682625:aes-256-cbc:3.000000
+R:679516:aes-256-cbc:3.000000
+DT:aes-256-cbc:3:1024
+DT:aes-256-cbc:3:1024
+R:420495:aes-256-cbc:3.000000
+DT:aes-256-cbc:3:8192
+R:418550:aes-256-cbc:3.000000
+DT:aes-256-cbc:3:8192
+R:426774:aes-256-cbc:3.000000
+R:430329:aes-256-cbc:3.000000
+DT:aes-256-cbc:3:8192
+DT:aes-256-cbc:3:8192
+R:91002:aes-256-cbc:3.000000
+R:90558:aes-256-cbc:3.000000
+R:90635:aes-256-cbc:3.000000
+R:90792:aes-256-cbc:3.000000
Forked child 3
Got: +H:16:64:256:1024:8192 from 0
Got: +F:22:aes-256-cbc:4397536.00:16793408.00:56110336.00:142865066.67:248496128.00 from 0
Got: +H:16:64:256:1024:8192 from 1
Got: +F:22:aes-256-cbc:4459434.67:17303274.67:57320106.67:143528960.00:247283712.00 from 1
Got: +H:16:64:256:1024:8192 from 2
Got: +F:22:aes-256-cbc:4573482.67:17382250.67:57985365.33:145672192.00:247493973.33 from 2
Got: +H:16:64:256:1024:8192 from 3
Got: +F:22:aes-256-cbc:4605898.67:17896725.33:58250666.67:146885632.00:247922688.00 from 3
OpenSSL 1.0.1l-freebsd 15 Jan 2015
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
evp 18036.35k 69375.66k 229666.47k 578951.85k 991196.50k -
Try to add "-elapsed" when you use the hardware engine, according to OpenSSL document this will perform better when using hardware crypto method.
But yeah, you see the difference with/without AES-NI already 8)
-
Don't understand why you are testing without, in the "real world" you are never likely to be using the same chip without aes-ni
-
Don't understand why you are testing without, in the "real world" you are never likely to be using the same chip without aes-ni
I would assume just to see how much extra throughput is gained via AES-NI vs without.
-
Don't understand why you are testing without, in the "real world" you are never likely to be using the same chip without aes-ni
I would assume just to see how much extra throughput is gained via AES-NI vs without.
Yes but I don't see when you would be without it if you had a processor that supports it. Seems counterintuitive.
-
Try not to use -evp option, it looks like everything with this option will trigger the hardware engine. And your CPU is quad core, you can actually put a "-multi 4" as the option to run 4 encryption together.
But that's quite interesting, because I owned a higher end processor, the Avoton C2550, and the number with 4 threads together is half of your single thread. Not sure if this is the problem with the OpenSSL on my Linux server (yes, this is not a pfSense machine). I suppose my Avoton should be running a lot faster compared with your N3700.Where I should put "-multi 4" to run 4 encryption together?
Thank youCiao
-
Try not to use -evp option, it looks like everything with this option will trigger the hardware engine. And your CPU is quad core, you can actually put a "-multi 4" as the option to run 4 encryption together.
But that's quite interesting, because I owned a higher end processor, the Avoton C2550, and the number with 4 threads together is half of your single thread. Not sure if this is the problem with the OpenSSL on my Linux server (yes, this is not a pfSense machine). I suppose my Avoton should be running a lot faster compared with your N3700.Where I should put "-multi 4" to run 4 encryption together?
Thank youCiao
Read the posts above, answer is already there.