PC Engines apu2 experiences

doktornotor

@kevindd992002:

But why is the NIC performance hampered with these settings disabled anyway?

You clearly are confused. When you check them, you DISable the HW offloading features.

kevindd992002

I don't think I am. Clearly, unchecking the boxes = ENABLES these features. checking the boxes=DISABLES these features. It's very easy to distinguish between the two.

j4k3 said in his post: "I had to uncheck Disable hardware large receive offload, and Disable hardware TCP segmentation offload". Which means that enabling (very different from "checking") them improves performance.

So then I asked: "But why is the NIC performance hampered with these settings disabled anyway?". Or in other words: "why is the NIC performance hampered with the boxes CHECKED anyway?"

Does that make sense? Again, disable=checked and enabled=unchecked. Please check the terminologies that I used in my posts.

Guest

Does that mean these two should be unchecked to get the full potential of the NIC's of the APU2C4?

Here under this link you will be able to read what is really needed for getting 1 GBit/s at the
WAN interface, there is told something likes, Server grade hardware and ~2,0GHz CPU speed.
And as I see it right the APU1D4 and APU2C4 are only sorted with something around ~1,1GHz
or 1,2GHz CPU power, that's it in short. Please read under under CPU selection

Any disadvantages of keeping them unchecked (enabled)?

Tunings and pimps can be done on each machine for sure to high up the
throughput but in that case, you should be followed to that guidance
from above at first.

cwagz

@cwagz:

I am looking for some opinions on downsizing my current pfSense system with an APU2C4.

Currently I have:
Supermicro A1SRI-2558
8GB Ram
120GB SSD
Akasa Fanless Enclosure

There are 6 people in my house and 30 or so devices. I am the only person that ever uses OpenVPN and it is usually from a mobile device on LTE so OpenVPN performance is probably not a huge deal. I run Squid and Squidguard to proxy the internet for my kids. Our internet connection is FiOS 150/150 Mbps.

It seems like I could build an apu2c4 and sell my current hardware. I would probably have money left over and a smaller, slightly cooler running device for pfSense.

Do you guys see any potential performance issues or reasons why this is a bad idea?

I went ahead and built the apu2c4 and am very happy with the outcome. The performance seems to be the same for our usage. Also, the overall footprint and heat output into my small network cabinet is improved.

HackedComputer

Hey,

I recently took delivery of an APU2C4. It is certainly a decent performer for the size of it!

I am wondering, has anyone got the AES-NI to work with the OpenVPN? The reason I ask is that I don't appear to see any acceleration happening with AES-128-CBC / AES-256-CBC. The rough maximum I have achieved is 30Mbps.

I have tried enabling the AES-NI within Advanced Options, and then enabling the cryptodev within OpenVPN. As well as disabling AES-NI and leaving Cryptodev enabled vice-versa.

However, I see no changes whatsoever.

I am on the latest PFSense 2.3.x release

Kindest Regards
HC

Guest

I am wondering, has anyone got the AES-NI to work with the OpenVPN? The reason I ask is that I don't appear to see any acceleration happening with AES-128-CBC / AES-256-CBC. The rough maximum I have achieved is 30Mbps.

From what total line speed you archived the 30Mbps? And how strong was the other VPN Peer end?

I have tried enabling the AES-NI within Advanced Options, and then enabling the cryptodev within OpenVPN. As well as disabling AES-NI and leaving Cryptodev enabled vice-versa.

At the moment only IPsec is really benefitting from the AES-NI, so you might be having
perhaps more luck if the OpenVPN version 2.4 is out there.

HackedComputer

From what total line speed you archived the 30Mbps? And how strong was the other VPN pear end?

Connecting from a 317Mbps line, the other end is serviced by a 10Gbit (SFP) line @ Rackspace

At the moment only IPsec is really benefitting from the AES-NI, so you might be having
perhaps more luck if the OpenVPN version 2.4 is out there.

I'll hold out, I'm not too fussed - I didn't expect a lot. But I expected a tad better as my old equipment was a dual core 800Mhz MiPS. I had tried the "fix" here:

http://1101entrails.blogspot.co.uk/2016/05/getting-aes-ni-to-work-using-pfsense-on.html

VAMike

@HackedComputer:

At the moment only IPsec is really benefitting from the AES-NI, so you might be having
perhaps more luck if the OpenVPN version 2.4 is out there.

I'll hold out, I'm not too fussed - I didn't expect a lot. But I expected a tad better as my old equipment was a dual core 800Mhz MiPS. I had tried the "fix" here:

http://1101entrails.blogspot.co.uk/2016/05/getting-aes-ni-to-work-using-pfsense-on.html

That page is mostly correct–openvpn does use aes-ni, having pfsense try to load any cryptographic stuff will slow things down, and you should be getting significantly more than 30Mbps. Make sure you're connecting with aes on the client side and turn off all the hardware crypto settings in pfsense.

HackedComputer

Just an update:

So, changing the cryptographic options within pfSense didn't yield any differences. Perhaps, by 5Mbps.

However, I looked more into the OpenVPN configuration and appended the following to the client configuration:

sndbuf 393216;
rcvbuf 393216

and thus, this was achieved:

FreeMinded

Here some "facts" from tests with APU2C4 and the latest pfSense (2.3.2_1)

WAN Speed on a clean Gigabit Link: ~640 MBits/s
OpenVPN Speed: ~50 MBits/s (AES-128-CBC with SHA256)

By enabling "Hardware Checksum Offloading" (by unchecking the setting) and "Hardware TCP Segmentation Offloading" (by unchecking the setting) I was able to get a 20-30 MBits/s improvement. Better, but still way off real Gigabit Speed.

Same for the OpenVPN throughput

Activating the AES-NI support doesn't currently do anything (should hopefully improve with OpenVPN Version 2.4+)
Activating BSD Crypto engine reduces (!) the throughput by ~5MBits/s
setting sndbuf 393216; rcvbuf 393216 in the client config as suggest above didn't do anything neither. But this might be due to the server side still using an older OpenVPN version (2.3.2) where there are low defaults. The OpenVPN log shows Socket Buffers: R=[42080->393216] S=[57344->393216]

Looking forward to other reports and suggestions!

VAMike

@FreeMinded:

Activating the AES-NI support doesn't currently do anything (should hopefully improve with OpenVPN Version 2.4+)
Activating BSD Crypto engine reduces (!) the throughput by ~5MBits/s

AES-NI is always on in current openvpn. Activating cryptodev overrides the built-in AES-NI support and does generally make openvpn slower. This is not specific to the APU2. If AES-NI were actually off (this can be done by setting an environment variable to tell openssl to ignore AES-NI support when running openvpn from the command line) you'd get a substantial reduction in throughput (more like 30% than 5%; the actual difference in crypto rates is much greater but openvpn has other bottlenecks).

Af0x

hi guys, has anyone managed to use the full capacity of your storage. I installed by this guide: http://pcengines.ch/howto.htm#OS_installation

Problem is that it only uses a part of my storage capacity. Can anyone tell me how to install on all available storage capacity?

doktornotor

@Af0x:

hi guys, has anyone managed to use the full capacity of your storage.

Stop using nanobsd.

Af0x

@doktornotor:

Stop using nanobsd.

ok, thank you. I did that and want to know which kernel is suitable for the apu2? I think it is the embedded one, but am not sure.

doktornotor

Cannot recall ever selecting something there. Get the memstick-serial image and leave it at default. :)

Guest

Can anyone tell me how to install on all available storage capacity?

Where you want to install pfSense? In a mSATA, USB drive or a real HDD/SSD?
If you take an usb drive and put on it the MEMSTICK-Serial-AM64 image and the you install from there onto a…....
That is a fresh and full install and might be the best bet for you.

Af0x

hi, thanks you guys. I already downloaded and installed on my mSATA. During Installation you get the choice like on the picture:
https://doc.pfsense.org/images/1/11/Installer_05_select_console.png

I chose embedded but I read that it has reduced possibilities talking about packages.

hda

@Af0x:

…mSATA...

APU2 & mSata ? >>> 64-bit Full Install.

Ofloo

apu1d4: throughput was arround 700mbit/s ~ 900mbit/s, when i first bought it pfsense did 500mbit/s later on that changed
apu3a4: throughput is 250mbit/s ~ 300mbit/s

What i do notice is that apu1d4 uses realtek(re) drivers and the apu3 uses intel(igb)
for the igb driver: TSO LRO .. checksum offload turned on or off makes no difference

strange thing is when I run iperf through vlan but not hosted on the router but on a server on a different subnet which is routed though the router i get

–----------------------------------------------------------
[ 3] local 192.168.1.100 port 44774 connected with 172.18.1.11 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 715 MBytes 600 Mbits/sec

when i turn on tso lro checksum offload, cpu rather then hiadapt to maximum

------------------------------------------------------------
Client connecting to loki, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.100 port 45468 connected with 172.18.1.11 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   757 MBytes   635 Mbits/sec

------------------------------------------------------------
Client connecting to loki, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.100 port 45462 connected with 172.18.1.11 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   742 MBytes   622 Mbits/sec

Guest

apu1d4: throughput was arround 700mbit/s ~ 900mbit/s, when i first bought it pfsense did 500mbit/s later on that changed

Without PowerD (hi adaptive) it was more around ~450 - 550 MBit/s but with "turned on" PowerD it Comes nearly to that numbers
like you are naming here, ~650 MBit/s - 750 MBit/s and yes for sure that can be different from OS to OS! From FreeBSD to Linux
as the base system, such as IPFire or ZeroShell are based on due to the better driver support.

apu3a4: throughput is 250mbit/s ~ 300mbit/s

The APU1D4 and APU2C4 are regular boards that was produced for the whole public, but likes in the past too, there
are options, that you might be calling for a customized board based on the number of boards you must then buy.
Let us say 150 - 1000 as a minimum and in former times this boards where then only for you or that customer and
not available for the rest of us or the whole public customers, this was changing now, and so rests of such a special
version or customized board will now also available to the public by sale. But please accept that this APU3 boards are
optimized for LTE usage, also the LAN ports as I am right informed and this might be then driving such a throughput
test in a total other direction! If you are using 3G and/or LTE modems inside this will be then perhaps once more
again differing from what you see now or get out now!

What i do notice is that apu1d4 uses realtek(re) drivers and the apu3 uses intel(igb)
for the igb driver: TSO LRO .. checksum offload turned on or off makes no difference

That might be shinning as the changing point for sure and first of all, but there are also two other things
that will be interesting now. The PowerD is bursting the APU1 cpu (T40E dual core) much more then the
APU2 cpu (GX-412TC quad core) that is taking not really a benefit well from activating this PowerD option
and then the igb(4) driver is now also multi cpu threading but even not the PPPoE part!

I don´t know if that "optimized option" will be able to change or something else, but perhaps this might be
then pushing that number somehow in the high, if there will be an option to change them. Perhaps in the BIOS!?