PC Engines apu2 experiences

lachendichatz

I switched from A1SRI-2758/8GB to apu2c4 as I need the 2758 for another server.

Running iperf3 between two pcs connected by the apu (pfsense 2.3.2) I get
600Mbit/s in one direction
615Mbit/s in the other one.
CPU runs at 25% load, e.g. one core is maxed out.

I probably see different speeds because I already imported my old firewall rules and have three rules on one nic and around 20 on the other one.

I observed the same speed & load when I installed debian and configured a few iptable rules.

Adding more clients (iperf -P 8 ) gives me:
940Mbit/s direction a
690Mbit/s direction b
880MBit/s duplex

Enabling (disabling unchecked) segmentation offload gives me
940Mbit/s direction a
695Mbit/s direction b
940Mbit/s duplex
CPU runs at 85%

Speedtest (init7) shows me 930down/940up. Initially I got 720/920 & 670/920 but that was because of my slow laptop (sigh). Restarting firefox gave me consistent speeds around 940.

So I can route a single TCP connection at 600Mbit/s per core. One could probably achieve higher speeds by tuning ISR related configs but as I can saturate my gigabit line with multiple connections I won't change settings.

I set igb_numqueue to 4 and mbuf to 1mio. Unknown if it had an effect.
Somebody suggested to set a rx/tx level (or queue? dont remeber) to 8k. That did not have an effect.
powerD disabled/enabled (hiadaptive) did not make a difference

Stugots

@AndrewZ:

@acascianelli:

Does anyone know of a way to enable TRIM support on the SSD without having to boot of a recovery device?

https://forum.pfsense.org/index.php?topic=66622.0
https://forum.pfsense.org/index.php?topic=113803.msg633795#msg633795

Is there no way to set it so that it's enabled on the next reboot without going into single user mode?

Is there no way to set it so that it's enabled on the next reboot without going into single user mode?

Actually is there no way or workaround, as I am informed right.

kevindd992002

@j4k3:

Thanks for the insight, I managed to finally get close to 1Gbps on the lan interface.

I had to uncheck Disable hardware large receive offload, and Disable hardware TCP segmentation offload

Under System > Advanced > Networking

Based on what I've read so far I know this unit won't route more than 500 Mbps or so but I wanted to at least understand why, the nic was so hobbled right off the bat.

Does that mean these two should be unchecked to get the full potential of the NIC's of the APU2C4? Any disadvantages of keeping them unchecked (enabled)?

hda

@kevindd992002:

… Any disadvantages of keeping them unchecked (enabled)?

Possibly, like no or a snappy WAN-PPPoE connection.

hda

@acascianelli:

…Is there no way to set it so that it's enabled on the next reboot without going into single user mode?

https://forum.pfsense.org/index.php?topic=121515.msg673176#msg673176 / pfSense 2.4

kevindd992002

@hda:

@kevindd992002:

… Any disadvantages of keeping them unchecked (enabled)?

Possibly, like no or a snappy WAN-PPPoE connection.

But why is the NIC performance hampered with these settings disabled anyway?

@hda:

@acascianelli:

…Is there no way to set it so that it's enabled on the next reboot without going into single user mode?

https://forum.pfsense.org/index.php?topic=121515.msg673176#msg673176 / pfSense 2.4

So if I understand this correctly, a fresh install of 2.4 will already enabled TRIM automatically with no user intervention? And same goes with older versions of pfsense that upgrade 2.4, TRIM will be enabled?

doktornotor

@kevindd992002:

But why is the NIC performance hampered with these settings disabled anyway?

You clearly are confused. When you check them, you DISable the HW offloading features.

kevindd992002

I don't think I am. Clearly, unchecking the boxes = ENABLES these features. checking the boxes=DISABLES these features. It's very easy to distinguish between the two.

j4k3 said in his post: "I had to uncheck Disable hardware large receive offload, and Disable hardware TCP segmentation offload". Which means that enabling (very different from "checking") them improves performance.

So then I asked: "But why is the NIC performance hampered with these settings disabled anyway?". Or in other words: "why is the NIC performance hampered with the boxes CHECKED anyway?"

Does that make sense? Again, disable=checked and enabled=unchecked. Please check the terminologies that I used in my posts.

Does that mean these two should be unchecked to get the full potential of the NIC's of the APU2C4?

Here under this link you will be able to read what is really needed for getting 1 GBit/s at the
WAN interface, there is told something likes, Server grade hardware and ~2,0GHz CPU speed.
And as I see it right the APU1D4 and APU2C4 are only sorted with something around ~1,1GHz
or 1,2GHz CPU power, that's it in short. Please read under under CPU selection

Any disadvantages of keeping them unchecked (enabled)?

Tunings and pimps can be done on each machine for sure to high up the
throughput but in that case, you should be followed to that guidance
from above at first.

cwagz

@cwagz:

I am looking for some opinions on downsizing my current pfSense system with an APU2C4.

Currently I have:
Supermicro A1SRI-2558
8GB Ram
120GB SSD
Akasa Fanless Enclosure

There are 6 people in my house and 30 or so devices. I am the only person that ever uses OpenVPN and it is usually from a mobile device on LTE so OpenVPN performance is probably not a huge deal. I run Squid and Squidguard to proxy the internet for my kids. Our internet connection is FiOS 150/150 Mbps.

It seems like I could build an apu2c4 and sell my current hardware. I would probably have money left over and a smaller, slightly cooler running device for pfSense.

Do you guys see any potential performance issues or reasons why this is a bad idea?

I went ahead and built the apu2c4 and am very happy with the outcome. The performance seems to be the same for our usage. Also, the overall footprint and heat output into my small network cabinet is improved.

HackedComputer

Hey,

I recently took delivery of an APU2C4. It is certainly a decent performer for the size of it!

I am wondering, has anyone got the AES-NI to work with the OpenVPN? The reason I ask is that I don't appear to see any acceleration happening with AES-128-CBC / AES-256-CBC. The rough maximum I have achieved is 30Mbps.

I have tried enabling the AES-NI within Advanced Options, and then enabling the cryptodev within OpenVPN. As well as disabling AES-NI and leaving Cryptodev enabled vice-versa.

However, I see no changes whatsoever.

I am on the latest PFSense 2.3.x release

Kindest Regards
HC

I am wondering, has anyone got the AES-NI to work with the OpenVPN? The reason I ask is that I don't appear to see any acceleration happening with AES-128-CBC / AES-256-CBC. The rough maximum I have achieved is 30Mbps.

From what total line speed you archived the 30Mbps? And how strong was the other VPN Peer end?

I have tried enabling the AES-NI within Advanced Options, and then enabling the cryptodev within OpenVPN. As well as disabling AES-NI and leaving Cryptodev enabled vice-versa.

At the moment only IPsec is really benefitting from the AES-NI, so you might be having
perhaps more luck if the OpenVPN version 2.4 is out there.

HackedComputer

From what total line speed you archived the 30Mbps? And how strong was the other VPN pear end?

Connecting from a 317Mbps line, the other end is serviced by a 10Gbit (SFP) line @ Rackspace

At the moment only IPsec is really benefitting from the AES-NI, so you might be having
perhaps more luck if the OpenVPN version 2.4 is out there.

I'll hold out, I'm not too fussed - I didn't expect a lot. But I expected a tad better as my old equipment was a dual core 800Mhz MiPS. I had tried the "fix" here:

http://1101entrails.blogspot.co.uk/2016/05/getting-aes-ni-to-work-using-pfsense-on.html

VAMike

@HackedComputer:

At the moment only IPsec is really benefitting from the AES-NI, so you might be having
perhaps more luck if the OpenVPN version 2.4 is out there.

I'll hold out, I'm not too fussed - I didn't expect a lot. But I expected a tad better as my old equipment was a dual core 800Mhz MiPS. I had tried the "fix" here:

http://1101entrails.blogspot.co.uk/2016/05/getting-aes-ni-to-work-using-pfsense-on.html

That page is mostly correct–openvpn does use aes-ni, having pfsense try to load any cryptographic stuff will slow things down, and you should be getting significantly more than 30Mbps. Make sure you're connecting with aes on the client side and turn off all the hardware crypto settings in pfsense.

HackedComputer

Just an update:

So, changing the cryptographic options within pfSense didn't yield any differences. Perhaps, by 5Mbps.

However, I looked more into the OpenVPN configuration and appended the following to the client configuration:

sndbuf 393216;
rcvbuf 393216

and thus, this was achieved:

FreeMinded

Here some "facts" from tests with APU2C4 and the latest pfSense (2.3.2_1)

WAN Speed on a clean Gigabit Link: ~640 MBits/s
OpenVPN Speed: ~50 MBits/s (AES-128-CBC with SHA256)

By enabling "Hardware Checksum Offloading" (by unchecking the setting) and "Hardware TCP Segmentation Offloading" (by unchecking the setting) I was able to get a 20-30 MBits/s improvement. Better, but still way off real Gigabit Speed.

Same for the OpenVPN throughput

Activating the AES-NI support doesn't currently do anything (should hopefully improve with OpenVPN Version 2.4+)
Activating BSD Crypto engine reduces (!) the throughput by ~5MBits/s
setting sndbuf 393216; rcvbuf 393216 in the client config as suggest above didn't do anything neither. But this might be due to the server side still using an older OpenVPN version (2.3.2) where there are low defaults. The OpenVPN log shows Socket Buffers: R=[42080->393216] S=[57344->393216]

Looking forward to other reports and suggestions!

VAMike

@FreeMinded:

Activating the AES-NI support doesn't currently do anything (should hopefully improve with OpenVPN Version 2.4+)
Activating BSD Crypto engine reduces (!) the throughput by ~5MBits/s

AES-NI is always on in current openvpn. Activating cryptodev overrides the built-in AES-NI support and does generally make openvpn slower. This is not specific to the APU2. If AES-NI were actually off (this can be done by setting an environment variable to tell openssl to ignore AES-NI support when running openvpn from the command line) you'd get a substantial reduction in throughput (more like 30% than 5%; the actual difference in crypto rates is much greater but openvpn has other bottlenecks).

Af0x

hi guys, has anyone managed to use the full capacity of your storage. I installed by this guide: http://pcengines.ch/howto.htm#OS_installation

Problem is that it only uses a part of my storage capacity. Can anyone tell me how to install on all available storage capacity?

doktornotor

@Af0x:

hi guys, has anyone managed to use the full capacity of your storage.

Stop using nanobsd.