PC Engines apu2 experiences
-
I have a site-to-site VPN using an OpenVPN tunnel between two APU2C4's with pfSense on them (2.4.3-RELEASE-p1 (amd64)). I've been reading a few hours now on how to really have OpenVPN utilize hardware AES-NI as the CPU supports it. There are several threads about this but not one is clear enough to really explain how pfSense uses this.
So, if I want to use hardware AES-NI, do I need to choose AES-NI CPU-based Acceleration under System > Advanced > Miscellaneous > Cryptographic Hardware? Or should I set it to None (no module loaded) and OpenVPN will use the AES-NI natively from the hardware without any module conflicts from the pfSense BSD OS?
If I choose the former, under the OpenVPN Server/Client settings the only selection I have for Hardware Crypto is No Hardware Crypto Acceleration. The only time I have an extra option in the Client settings is when I choose BSD Crypto Device (cryptodev) under Miscellaneous. So which is which? I'm starting to have a headache because of the confusing pfSense GUI :)
-
I'm also curious as to what the "correct" settings are... I can say that I did a couple of tests wrt speed, and I eventually settled on (apart from a bunch of other tweaks) enabling AES-NI CPU-based Acceleration under System > Advanced > Miscellaneous > Cryptographic Hardware, and then in the OpenVPN Server/Client settings the only selection I have No Hardware Crypto Acceleration.
I know a while back, in the OpenVPN settings you could choose between AES-NI and cryptodev, but after some update the cryptodev disappeard.
Additionally, I think that the best speeds were achieved when both AES-NI CPU-based Acceleration and cryptodev were enabled, which is now the default if you have AES-NI CPU-based Acceleration enabled on the system.
So, I've just assumed that it doesn't matter anymore about the setting in OpenVPN since the system is already using all it can. -
@veldkornet said in PC Engines apu2 experiences:
Additionally, I think that the best speeds were achieved when both AES-NI CPU-based Acceleration and cryptodev were enabled, which is now the default if you have AES-NI CPU-based Acceleration enabled on the system.
So, I've just assumed that it doesn't matter anymore about the setting in OpenVPN since the system is already using all it can.Enabling both where? Let's call both places Miscellaneous settings and Client settings to avoid confusion. Like I said, I don't have any options for a cryptodev Client setting IF I keep the Miscellaneous settings to AES-NI. All I have is the No Hardware Crypto Acceleration option.
-
My mistake, I actually have the Miscellaneous set to AES-NI and BSD Crypto Device (aesni, cryptodev).
This is the update that I referred to in my previous post where both are enabled.OpenVPN Client & Server:
And these are the options available to me within OpenVPN:
-
@veldkornet said in PC Engines apu2 experiences:
My mistake, I actually have the Miscellaneous set to AES-NI and BSD Crypto Device (aesni, cryptodev).
This is the update that I referred to in my previous post where both are enabled.OpenVPN Client & Server:
And these are the options available to me within OpenVPN:
Ok, that makes more sense and we have the same set of available options in both places. Although I'm reading a lot that it generally is not a good thing to have both modules loaded (which is what you have set in Misc.) so I was wondering how you got better speeds with that?
-
@kevindd992002 said in PC Engines apu2 experiences:
Ok, that makes more sense and we have the same set of available options in both places. Although I'm reading a lot that it generally is not a good thing to have both modules loaded (which is what you have set in Misc.) so I was wondering how you got better speeds with that?
With regards to having both loaded, see resolved bug 7810.
In the OpenVPN settings, selecting BSD cryptodev engine made it slower indeed and shouldn't be selected.
Not related to the crypto, but what also made a big difference is setting the following:
Anyway, this is just my opinion of how it should be since this works the best in my situation.
-
@veldkornet said in PC Engines apu2 experiences:
@kevindd992002 said in PC Engines apu2 experiences:
Ok, that makes more sense and we have the same set of available options in both places. Although I'm reading a lot that it generally is not a good thing to have both modules loaded (which is what you have set in Misc.) so I was wondering how you got better speeds with that?
With regards to having both loaded, see resolved bug 7810.
In the OpenVPN settings, selecting BSD cryptodev engine made it slower indeed and shouldn't be selected.
Not related to the crypto, but what also made a big difference is setting the following:
Anyway, this is just my opinion of how it should be since this works the best in my situation.
Ahh, I see what you mean. Thanks for the heads up. I'm setting it that way then.
Yeah, I have both UDP Fast I/O and send/receiver buffer set to those values also as I read they can speed up things.
-
Setting those tweaks will definitely show an improvement. Usually a significant one though I've never tested it on the APU2 myself.
OpenSSL should use the AES-NI instructions on your CPU directly if it supports them. The danger here is that instead of using them directly it tries to use the BSD crypto framework where the AES-NI kernel module has registered itself for the algorithms it supports. That means a load of additional cycles to do the same calculation.
As long as you don't have AES-NI+BSD crypto set in Adv. > MIsc. and BSD crypto set in openvpn you should avoid that.
When I last tested it Fast I/O and send/receiver buffer made a greater difference to throughput.
Steve
-
@stephenw10 said in PC Engines apu2 experiences:
Setting those tweaks will definitely show an improvement. Usually a significant one though I've never tested it on the APU2 myself.
OpenSSL should use the AES-NI instructions on your CPU directly if it supports them. The danger here is that instead of using them directly it tries to use the BSD crypto framework where the AES-NI kernel module has registered itself for the algorithms it supports. That means a load of additional cycles to do the same calculation.
As long as you don't have AES-NI+BSD crypto set in Adv. > MIsc. and BSD crypto set in openvpn you should avoid that.
When I last tested it Fast I/O and send/receiver buffer made a greater difference to throughput.
Steve
I see. So are you saying that setting AES-NI+BSD in Misc. and just no hardware crypto in the OpenVPN client settings would be fine?
-
@kevindd992002 said in PC Engines apu2 experiences:
I see. So are you saying that setting AES-NI+BSD in Misc. and just no hardware crypto in the OpenVPN client settings would be fine?
Im interested in this too! And Im a bit confused after reading 20 posts about it in this thread.
cheers -
@kevindd992002 said in PC Engines apu2 experiences:
I see. So are you saying that setting AES-NI+BSD in Misc. and just no hardware crypto in the OpenVPN client settings would be fine?
That's what I would expect. In 2.4 at least.
The last time I tested this though I achieved greatest throughput with both fields set to none or BSD with AES-NI disabled. Use AES-GCM and enable fastio and larger send/rec buffers.
That was a while back, 2.3.4 vs 2.4.0.More testing is always good.
Steve
-
@stephenw10 said in PC Engines apu2 experiences:
I achieved greatest throughput with both fields set to none or BSD with AES-NI disabled
What is a good throughput for APU2c4? My initial tests with openvpn server running on the apu2 (128-cbc) is around 20Mbit only... (150Mbit line max).
But I havent done your test with all OFF. -
@daemonix said in PC Engines apu2 experiences:
@stephenw10 said in PC Engines apu2 experiences:
I achieved greatest throughput with both fields set to none or BSD with AES-NI disabled
What is a good throughput for APU2c4? My initial tests with openvpn server running on the apu2 (128-cbc) is around 20Mbit only... (150Mbit line max).
But I havent done your test with all OFF.I currently get around 20Mbit as well, up and down, sometimes a bit more.
I have a 400/40 Mbit line.
I’ll need to double check my settings, but I think I have 256bit encryption.
-
@veldkornet said in PC Engines apu2 experiences:
@daemonix said in PC Engines apu2 experiences:
@stephenw10 said in PC Engines apu2 experiences:
I achieved greatest throughput with both fields set to none or BSD with AES-NI disabled
What is a good throughput for APU2c4? My initial tests with openvpn server running on the apu2 (128-cbc) is around 20Mbit only... (150Mbit line max).
But I havent done your test with all OFF.I currently get around 20Mbit as well, up and down, sometimes a bit more.
I have a 400/40 Mbit line.
I’ll need to double check my settings, but I think I have 256bit encryption.
pif Im getting 40Mbit max here. 128-GCM too.
I found this page: https://github.com/ocochard/netbenches/blob/master/AMD_GX-412TC_4Cores_Intel_i210AT/openvpn/results/fbsd11.0/README.md
But I cant see any luck. my test config below with APU2 running the openvpn server.
EDIT: this page too: https://teklager.se/en/knowledge-base/apu2-vpn-performance/
-
Just wanted to find out how everyone is getting on with 2.4.4?
I had Firmware 4.8.0.4 on my APU2 with 2.4.3 and everything was fine, but after upgrading to 2.4.4 I had lots of “stalls”, both via the web interface and via SSH. I often had to restart the SAH session (or refresh the web page).
I downgraded the firmware all the way back down to 4.0.7, and the stalls seem to be gone, although I can’t exactly say that everything is as snappy as I’d expect it to be, changing screens on the GUI still take quite a bit of time.
If I restart unbound, it seems to take a good 5 minutes before it actually starts resolving.
I seem to have to restart PHP-FPM pretty often to get the interface in a working state (never had to do this before).Anyone else seeing this? Or whats your experience been so far with 2.4.4?
-
I'd be surprised if that was anything to do with the Coreboot version really. About the only thing I could imagine doing that would be some component that is initiallised differently and only supported in FreeBSD 11.2. But I'm not aware of that.
I would first backup the config and do a clean 2.4.4 install. If you still see the same issues you did in the upgraded 2.4.4 then did deeper. I would expect to see errors logged though.Steve
-
I need a recommendation on a console cable for the APU1/APU2 units. My laptop will be retired soon and so I'll no longer have a serial port to use. I'm sure there are USB cables that will connect in. Does anyone have a link to one that they use that we know will work? Thanks for the help!
-
@stewart said in PC Engines apu2 experiences:
I need a recommendation on a console cable for the APU1/APU2 units. My laptop will be retired soon and so I'll no longer have a serial port to use. I'm sure there are USB cables that will connect in. Does anyone have a link to one that they use that we know will work? Thanks for the help!
I have this one, works well for me: https://www.startech.com/eu/m/Cards-Adapters/Serial-Cards-Adapters/USB-to-Null-Modem-RS232-DB9-Serial-Adapter-Cable-DCE-FTDI~ICUSB232FTN
-
@ Veldkornet
Think this is the same thing?
https://www.amazon.com/USB-Serial-Adapter-Modem-9-pin/dp/B008634VJY/ref=sr_1_3?ie=UTF8&qid=1539289152&sr=8-3&keywords=startech+usb+null+modemEDIT: Found the model on the box in the image. It is indeed. Thanks for the rec!
-
@stewart said in PC Engines apu2 experiences:
@ Veldkornet
Think this is the same thing?
https://www.amazon.com/USB-Serial-Adapter-Modem-9-pin/dp/B008634VJY/ref=sr_1_3?ie=UTF8&qid=1539289152&sr=8-3&keywords=startech+usb+null+modemEDIT: Found the model on the box in the image. It is indeed. Thanks for the rec!
Yup, looks like the same one indeed! :)
-
hehe just in time! I was going to ask the same thing!
Last time I used a null modem was back in 2003ish for my Sun v120 :)
My pfsense gives me some php error.cheers
-
I've used a few times so far. Works great!
-
@stephenw10 said in PC Engines apu2 experiences:
I'd be surprised if that was anything to do with the Coreboot version really. About the only thing I could imagine doing that would be some component that is initiallised differently and only supported in FreeBSD 11.2. But I'm not aware of that.
I would first backup the config and do a clean 2.4.4 install. If you still see the same issues you did in the upgraded 2.4.4 then did deeper. I would expect to see errors logged though.Steve
I get the same experience too. Clean 2.4.4 install on a apu2c4.
SSH or Serial stalls after printing 10-15 chars.
Web is dead or I get 504.Basic WAN network works ok.
Any ideas?
-
Not without some sort of error to go on.
Are you able to get any logs?
Does it stop at the same point every time?
Does the boot log complete as expected via serial?
Steve
-
@stephenw10 said in PC Engines apu2 experiences:
Not without some sort of error to go on.
Are you able to get any logs?
Does it stop at the same point every time?
Does the boot log complete as expected via serial?
Steve
boot did well.
after random time I couldnt get to console or see logs.
on hard restart (PSU off/on) everything goes back to ok.Ill continue monitoring and let you know.
-
@daemonix
Right now I'm running 3 APU2C4 devices without any problems but they are all upgrades. No clean installs. I don't know the firmware versions, though. Maybe that would be a culprit? Also, can you send the system logs somewhere so they are viewable after the unit locks up? Then you can see the last thing logged. If you stay logged in via serial is there a kernel panic? You would need to stay logged in via serial. Trying to get in after it locks up would be unsuccessful. Might be a driver issue throwing a panic that doesn't get logged. -
@stewart said in PC Engines apu2 experiences:
@daemonix
Right now I'm running 3 APU2C4 devices without any problems but they are all upgrades. No clean installs. I don't know the firmware versions, though. Maybe that would be a culprit? Also, can you send the system logs somewhere so they are viewable after the unit locks up? Then you can see the last thing logged. If you stay logged in via serial is there a kernel panic? You would need to stay logged in via serial. Trying to get in after it locks up would be unsuccessful. Might be a driver issue throwing a panic that doesn't get logged.Ill investigate. At the moment after a hard power reset Its going good.
Im monitoring via the serial.I havent checked the firmware version.
-
Hi, I also have a few issues with my apu2c4 and pfSense 2.4.4 being unresponsive. I have yet to see anything in logs that's acutally helpfull. My monitoring though raises a few alerts per day on web and ssh being unresponsive. Traffic flows normally though.
But I have also found that doing certain operations in the webUI might block all traffic through the firewall and also block any new connections to the firewall itself. So far I have identified that trying to search in the States diagnostics page will mess everyting up. All traffic stops. Last time I fourtnally had an open SSH connection and could reset php-fpm from the console menu, which cleared everthing without the need of a hard boot. Top did not show any execessive CPU load, just a normal idle system, and I did not see anything interesteing in any logs. (Not reported yet.)
It's a clean full install of 2.4.4. Configration was restored from the previous 2.4.3 install, which ran just fine on the hardware.
Coreboot is of version 4.0.7. I'm considering upgrade to 4.8.0.x just to test.
-
@thewhero said in PC Engines apu2 experiences:
Coreboot is of version 4.0.7. I'm considering upgrade to 4.8.0.x just to test.
which tutorial do you follow for updating the coreboot?
cheers -
@daemonix said in PC Engines apu2 experiences:
which tutorial do you follow for updating the coreboot?
cheersI saw this post: https://forum.netgate.com/topic/120380/pc-engines-apu2-bios-options and have decided to use flashrom directly from pfSense instead of booting a USB-stick to run flashrom. I have verified that flashrom can indeed communicate with the flash by dumping current flash image to disk.
[2.4.4-RELEASE][admin@fw]/root: flashrom --programmer internal --read flash.img flashrom v1.0 on FreeBSD 11.2-RELEASE-p3 (amd64) flashrom is free software, get the source code at https://flashrom.org Using clock_gettime for delay loops (clk_id: 4, resolution: 2ns). coreboot table found at 0xdffae000. Found chipset "AMD FCH". Enabling flash write... OK. Found Winbond flash chip "W25Q64.V" (8192 kB, SPI) mapped at physical address 0x00000000ff800000. Reading flash... done.
Now I just need to find a suitable maitenance window so I have time to recover if anything goes wrong.
-
Indeed I currently flash from pfSense directly as well.
Just install flashrom with the following:
pkg install flashrom
Then, because you're coming from an old version, you'll probably need to force it since they changed the naming conventions:
flashrom -w /tmp/apu2_v4.8.0.5.rom -p internal:boardmismatch=force
You can find all of the latest firmware versions here.
Just a note, on the 4.8.X releases, there is some bug where the system will hang on a reboot if it's been up and running for a while.
Also, if you haven't already done so, you will need to add the following to your /boot/loader.conf:
boot_serial="YES" comconsole_speed="115200" console="comconsole" hint.ahci.0.msi="0" loader_conf_files="/boot/device.hints"
Other than that, the new FW's are fine.
-
I noticed in my 2.4.4 /var/log/dmesg.boot the following:
module_register_init: MOD_LOAD (vesa, 0xffffffff81209800, 0) error 19
But the default config is not to load vesa:
vesa_load="NO"
And if I try load it manually, I get the following:
kldload vesa kldload: can't load vesa: No such file or directory
Anyone know what this is and how to fix it? Why is it trying to load?
-
@veldkornet said in PC Engines apu2 experiences:
Indeed I currently flash from pfSense directly as well.
Just install flashrom with the following:
pkg install flashrom
Then, because you're coming from an old version, you'll probably need to force it since they changed the naming conventions:
flashrom -w /tmp/apu2_v4.8.0.5.rom -p internal:boardmismatch=force
You can find all of the latest firmware versions here.
Just a note, on the 4.8.X releases, there is some bug where the system will hang on a reboot if it's been up and running for a while.
Also, if you haven't already done so, you will need to add the following to your /boot/loader.conf:
boot_serial="YES" comconsole_speed="115200" console="comconsole" hint.ahci.0.msi="0" loader_conf_files="/boot/device.hints"
Other than that, the new FW's are fine.
Does this mean that as long as you add those lines to /boot/loader.conf, running the latest FW's is fine?
-
@kevindd992002 said in PC Engines apu2 experiences:
Does this mean that as long as you add those lines to /boot/loader.conf, running the latest FW's is fine?
I'm currently running 4.8.0.5 on pfSense 4.2.2 with a SSD in ZFS and except for the small things I mentioned about the reboot not working if the system had been running for a long time, all seems to be fine. I have those lines in my config as well.
-
I see. So no fix yet for the system hang on a reboot issue yet? Even just a workaround of any sort?
-
@kevindd992002 said in PC Engines apu2 experiences:
I see. So no fix yet for the system hang on a reboot issue yet? Even just a workaround of any sort?
Yes, pull out the power plug
I don't see it as a major issue, the newer versions have more improvements so I'll stay on it. You'll have to have a read through all of the changes if you want to see everything. 4.8.0.5 now supports ECC for example.
I linked to the reboot issue on Github somewhere above if you want to follow it. -
Argh. It will be an issue for me if I manage a pfsense box remotely.
Ok, I'll take a look at that then. Why can't they fix it?
-
@kevindd992002 said in PC Engines apu2 experiences:
Argh. It will be an issue for me if I manage a pfsense box remotely.
Ok, I'll take a look at that then. Why can't they fix it?
I don't know? Read through this.
-
How does the APU2 stack up against the MBT-2220, performance-wise, for running pfsense, IPSec, and OpenVPN?
I needed a new box about a month ago, and since Netgate wasn't offering APU2 units any more I went with an MBT-2220. It works fine, but I miss the 3rd Ethernet port and the internal expansion slots. [After performing installations on both units I also realize I prefer having a serial console, because then I don't need a monitor and keyboard. I just need a cheap USB-Nullmodem cable.] I thought Netgate stopped selling the APU2 because it was obsolete or unavailable, but apparently the APU2 is still widely available at retail, and is a few dollars less expensive than the MBT-2220.
The APU2 has: "AMD Embedded G series GX-412TC, 1 GHz quad Jaguar core with 64 bit and AES-NI support, 32K data + 32K instruction cache per core, shared 2MB L2 cache."
while the MBT-2220 has: "Intel Atom E3826 (2 x 1.46 GHz, 1MB cache, AES-NI)"
I'm not savvy enough to know which hardware is better. I have 3 (and soon to be 4) sites meshed together with IPSec tunnels among them, and I'm adding one site that will be connecting via OpenVPN. Max wire speed at any of these sites is 50 Mbps, and they're typically 20 Mbps or slower.
Thanks!
-
@thewaterbug
We have a lot of the APU2C4 units out there, and they make an APU4 now with 4 ports instead of 3 if that interests you. The largest client we use it at is a hotel with 6 buildings and normally around 150-200 guests at a time on a 500Mbit fiber connection (~30 devices on the Office LAN network and ~150 devices on the Tenant OPT1 network). No issues. Not sure it could go much higher as a max speed but it is running Suricata (On Office and Tenant networks), pfBlocker (On office and Tenant networks), and Squid+ClamAV+SquidGuard (On the Office network only). No issues there. We have other clients using the IPSEC and OpenVPN and they work very well. Don't know top speed but it certainly doesn't feel slow. I can't compare to the MBT-2220 with 2 cores that are likely twice as strong (Jaguar isn't exactly high IPC) but the APU2C4 can do 500Mbps just fine. Lately I've seen Suricata go wonky and cap out the CPUs on several units but that appears to be a log issue. Uninstalling/reinstalling seems to fix it so far. I hope that gives you some understanding.