PC Engines apu2 experiences
-
Hi, I also have a few issues with my apu2c4 and pfSense 2.4.4 being unresponsive. I have yet to see anything in logs that's acutally helpfull. My monitoring though raises a few alerts per day on web and ssh being unresponsive. Traffic flows normally though.
But I have also found that doing certain operations in the webUI might block all traffic through the firewall and also block any new connections to the firewall itself. So far I have identified that trying to search in the States diagnostics page will mess everyting up. All traffic stops. Last time I fourtnally had an open SSH connection and could reset php-fpm from the console menu, which cleared everthing without the need of a hard boot. Top did not show any execessive CPU load, just a normal idle system, and I did not see anything interesteing in any logs. (Not reported yet.)
It's a clean full install of 2.4.4. Configration was restored from the previous 2.4.3 install, which ran just fine on the hardware.
Coreboot is of version 4.0.7. I'm considering upgrade to 4.8.0.x just to test.
-
@thewhero said in PC Engines apu2 experiences:
Coreboot is of version 4.0.7. I'm considering upgrade to 4.8.0.x just to test.
which tutorial do you follow for updating the coreboot?
cheers -
@daemonix said in PC Engines apu2 experiences:
which tutorial do you follow for updating the coreboot?
cheersI saw this post: https://forum.netgate.com/topic/120380/pc-engines-apu2-bios-options and have decided to use flashrom directly from pfSense instead of booting a USB-stick to run flashrom. I have verified that flashrom can indeed communicate with the flash by dumping current flash image to disk.
[2.4.4-RELEASE][admin@fw]/root: flashrom --programmer internal --read flash.img flashrom v1.0 on FreeBSD 11.2-RELEASE-p3 (amd64) flashrom is free software, get the source code at https://flashrom.org Using clock_gettime for delay loops (clk_id: 4, resolution: 2ns). coreboot table found at 0xdffae000. Found chipset "AMD FCH". Enabling flash write... OK. Found Winbond flash chip "W25Q64.V" (8192 kB, SPI) mapped at physical address 0x00000000ff800000. Reading flash... done.
Now I just need to find a suitable maitenance window so I have time to recover if anything goes wrong.
-
Indeed I currently flash from pfSense directly as well.
Just install flashrom with the following:
pkg install flashrom
Then, because you're coming from an old version, you'll probably need to force it since they changed the naming conventions:
flashrom -w /tmp/apu2_v4.8.0.5.rom -p internal:boardmismatch=force
You can find all of the latest firmware versions here.
Just a note, on the 4.8.X releases, there is some bug where the system will hang on a reboot if it's been up and running for a while.
Also, if you haven't already done so, you will need to add the following to your /boot/loader.conf:
boot_serial="YES" comconsole_speed="115200" console="comconsole" hint.ahci.0.msi="0" loader_conf_files="/boot/device.hints"
Other than that, the new FW's are fine.
-
I noticed in my 2.4.4 /var/log/dmesg.boot the following:
module_register_init: MOD_LOAD (vesa, 0xffffffff81209800, 0) error 19
But the default config is not to load vesa:
vesa_load="NO"
And if I try load it manually, I get the following:
kldload vesa kldload: can't load vesa: No such file or directory
Anyone know what this is and how to fix it? Why is it trying to load?
-
@veldkornet said in PC Engines apu2 experiences:
Indeed I currently flash from pfSense directly as well.
Just install flashrom with the following:
pkg install flashrom
Then, because you're coming from an old version, you'll probably need to force it since they changed the naming conventions:
flashrom -w /tmp/apu2_v4.8.0.5.rom -p internal:boardmismatch=force
You can find all of the latest firmware versions here.
Just a note, on the 4.8.X releases, there is some bug where the system will hang on a reboot if it's been up and running for a while.
Also, if you haven't already done so, you will need to add the following to your /boot/loader.conf:
boot_serial="YES" comconsole_speed="115200" console="comconsole" hint.ahci.0.msi="0" loader_conf_files="/boot/device.hints"
Other than that, the new FW's are fine.
Does this mean that as long as you add those lines to /boot/loader.conf, running the latest FW's is fine?
-
@kevindd992002 said in PC Engines apu2 experiences:
Does this mean that as long as you add those lines to /boot/loader.conf, running the latest FW's is fine?
I'm currently running 4.8.0.5 on pfSense 4.2.2 with a SSD in ZFS and except for the small things I mentioned about the reboot not working if the system had been running for a long time, all seems to be fine. I have those lines in my config as well.
-
I see. So no fix yet for the system hang on a reboot issue yet? Even just a workaround of any sort?
-
@kevindd992002 said in PC Engines apu2 experiences:
I see. So no fix yet for the system hang on a reboot issue yet? Even just a workaround of any sort?
Yes, pull out the power plug
I don't see it as a major issue, the newer versions have more improvements so I'll stay on it. You'll have to have a read through all of the changes if you want to see everything. 4.8.0.5 now supports ECC for example.
I linked to the reboot issue on Github somewhere above if you want to follow it. -
Argh. It will be an issue for me if I manage a pfsense box remotely.
Ok, I'll take a look at that then. Why can't they fix it?
-
@kevindd992002 said in PC Engines apu2 experiences:
Argh. It will be an issue for me if I manage a pfsense box remotely.
Ok, I'll take a look at that then. Why can't they fix it?
I don't know? Read through this.
- 20 days later
-
How does the APU2 stack up against the MBT-2220, performance-wise, for running pfsense, IPSec, and OpenVPN?
I needed a new box about a month ago, and since Netgate wasn't offering APU2 units any more I went with an MBT-2220. It works fine, but I miss the 3rd Ethernet port and the internal expansion slots. [After performing installations on both units I also realize I prefer having a serial console, because then I don't need a monitor and keyboard. I just need a cheap USB-Nullmodem cable.] I thought Netgate stopped selling the APU2 because it was obsolete or unavailable, but apparently the APU2 is still widely available at retail, and is a few dollars less expensive than the MBT-2220.
The APU2 has: "AMD Embedded G series GX-412TC, 1 GHz quad Jaguar core with 64 bit and AES-NI support, 32K data + 32K instruction cache per core, shared 2MB L2 cache."
while the MBT-2220 has: "Intel Atom E3826 (2 x 1.46 GHz, 1MB cache, AES-NI)"
I'm not savvy enough to know which hardware is better. I have 3 (and soon to be 4) sites meshed together with IPSec tunnels among them, and I'm adding one site that will be connecting via OpenVPN. Max wire speed at any of these sites is 50 Mbps, and they're typically 20 Mbps or slower.
Thanks!
-
@thewaterbug
We have a lot of the APU2C4 units out there, and they make an APU4 now with 4 ports instead of 3 if that interests you. The largest client we use it at is a hotel with 6 buildings and normally around 150-200 guests at a time on a 500Mbit fiber connection (~30 devices on the Office LAN network and ~150 devices on the Tenant OPT1 network). No issues. Not sure it could go much higher as a max speed but it is running Suricata (On Office and Tenant networks), pfBlocker (On office and Tenant networks), and Squid+ClamAV+SquidGuard (On the Office network only). No issues there. We have other clients using the IPSEC and OpenVPN and they work very well. Don't know top speed but it certainly doesn't feel slow. I can't compare to the MBT-2220 with 2 cores that are likely twice as strong (Jaguar isn't exactly high IPC) but the APU2C4 can do 500Mbps just fine. Lately I've seen Suricata go wonky and cap out the CPUs on several units but that appears to be a log issue. Uninstalling/reinstalling seems to fix it so far. I hope that gives you some understanding. -
Thanks! That's exactly what I wanted to hear.
-
@thewaterbug
Just don't use snort. It is single-threaded and these don't have great single-threaded power. I've never seen Squid or pfBlocker cause hiccups. Suricata runs much better if you disable stream events but you'll want to keep your eye on it for a bit to make sure it runs smoothly. If there are CPU spikes you'll notice quickly enough. -
Ah, apparently I don't have an APU2. I have an APU1. I booted into TinyCore and got the following:
coreboot table found at 0x7efdf000. Found chipset "AMD SB7x0/SB8x0/SB9x0". Enabling flash write... OK. Identifying board "PC Engines apu1"... OK. Found Macronix flash chip "MX25L1605A/MX25L1606E/MX25L1608E" (2048 kB, SPI) mapped at physical address 0xffe00000. Error: Image size (8388608 B) doesn't match the flash chip's size (2097152 B)!
When I boot this board my console says:
PC Engines APU BIOS build date: Apr 5 2014
and my pfsense Dashboard reports:
System Netgate APU Netgate Device ID: 400a2blahblahblah BIOS Vendor: coreboot Version: SageBios_PCEngines_APU-45 Release Date: Sat Apr 5 2014
Is that the latest legacy version for this board?
I suppose I got confused because I purchased this from Netgate in 2014 as a " Netgate APU2".
-
I don’t know if Netgate make their own versions of the BIOS (because it says “SageBIOS”). I know if you check in package manager on pfSense, there is a package for the apu from them.
However, for the generic firmware, there are much newer ones available. See here: https://pcengines.github.io/#top
If you scroll a few posts back, we were “just discussing it” and how to flash the BIOS.
-
^^
but the oldest board in any of those lists [for the legacy releases] is "APU2", and it looks like have an APU with no digit (e.g. an APU1).I've read many places on this forum that, for pfsense, the "mainline" releases are to be avoided.
-
I don’t know why mainline’s are to be avoided, I have 4.8.0.5 running and it works just fine.
I think with the flash utility you can backup your existing firmware if you can no longer find it on the internet. Then you can downgrade if you have problems.
-
@veldkornet said in PC Engines apu2 experiences:
I don’t know why mainline’s are to be avoided, I have 4.8.0.5 running and it works just fine.
I think with the flash utility you can backup your existing firmware if you can no longer find it on the internet. Then you can downgrade if you have problems.
Agree with you, but why update if everything is working well? :-) Is there any benefit?
-
@fireodo said in PC Engines apu2 experiences:
Agree with you, but why update if everything is working well? :-) Is there any benefit?
Well, I'm not saying that you have to do anything. But you could use that same argument for everything software related... Why update Windows if everything works? ?
You can read through all of the release notes yourself, but they fix bugs and release new features every so often. Recently, they've enabled ECC memory in 4.8.0.5
-
@thewaterbug said in PC Engines apu2 experiences:
I suppose I got confused because I purchased this from Netgate in 2014 as a " Netgate APU2".
yes, it's been confusing people for years. all the netgate apu's were based on the older pcengines apu1 design, regardless of their naming convention.
@veldkornet said in PC Engines apu2 experiences:
Recently, they've enabled ECC memory in 4.8.0.5
not for the apu1
-
stephenw10 Netgate Administratorlast edited by stephenw10 Nov 15, 2018, 3:09 PM Nov 15, 2018, 3:01 PM
@veldkornet said in PC Engines apu2 experiences:
I don’t know if Netgate make their own versions of the BIOS (because it says “SageBIOS”).
We didn't.
It's possible to put the newer BIOS versions on APU1 if you want. I did it a while back just to test if it could be done:
BIOS Vendor: coreboot Version: v4.8.0.1 Release Date: Fri Jun 8 2018
That would be what was sold by Netgate as the APU2 or APU4 to indicate 2 or 4GB of RAM at the time. Also as the VK-T40E2/4 from the pfSense store.
I'm not aware of any advantages that BIOS brings but mine has been running solidly with for months just as one data point.
[Edit: actually it does appear to add new devices as bootable]See: https://forum.netgate.com/post/777287
Steve -
@vamike said in PC Engines apu2 experiences:
@thewaterbug said in PC Engines apu2 experiences:
I suppose I got confused because I purchased this from Netgate in 2014 as a " Netgate APU2".
yes, it's been confusing people for years. all the netgate apu's were based on the older pcengines apu1 design, regardless of their naming convention.
Aha! Thank you. I am very slightly less confused, now.
So I apparently have only a dual-core box with no AES-NI support. And the PCEngines "APU2xxx" was never sold by Netgate, correct?
Does my Netgate APU unit then belong in the "Official Netgate Hardware" forum?
-
@thewaterbug said in PC Engines apu2 experiences:
How does the APU2 stack up against the MBT-2220, performance-wise, for running pfsense, IPSec, and OpenVPN? ...
The APU2 has: "AMD Embedded G series GX-412TC, 1 GHz quad Jaguar core with 64 bit and AES-NI support, 32K data + 32K instruction cache per core, shared 2MB L2 cache."
while the MBT-2220 has: "Intel Atom E3826 (2 x 1.46 GHz, 1MB cache, AES-NI)"
That comparison was inaccurate, since I have an APU1, not an APU2. The correct comparison is now:
The APU1 has: "AMD G series T40E APU, 1 GHz dual core (Bobcat core) with 64 bit support, 32K data + 32K instruction + 512KB L2 cache per core."
while the MBT-2220 has: "Intel Atom E3826 (2 x 1.46 GHz, 1MB cache, AES-NI)"
So my spiffy new MBT-2220 units are clearly more performant than my old APU units, especially for anything that can use AES-NI acceleration.
-
@thewaterbug said in PC Engines apu2 experiences:
Does my Netgate APU unit then belong in the "Official Netgate Hardware" forum?
You can open a thread there but there are a lot of APU users who did not purchase through our store. You might well get more views here.
Steve
-
^^
Then I'll keep it here, where I get more views!By this weekend I should have one of my APU units re-installed with 2.4.4 and an IPSec tunnel to an MBT-2220 running 2.4.4. Can I use iperf between them to measure tunnel performance? Max line rate is only 20 Mbps.
If I can saturate that with AES turn on (software only) then there's no urgency to upgrade the hardware.
-
We used to use the APU1C2 before changing to the APU2C4 with the advent of 2.5 needing aes-ni. We tested them at 300Mbps and, although I don't recall actual numbers for AES in software we were able to get decent speeds and nobody complained. I think you'll be fine with 20Mbps.
-
Thanks! I guess I missed the very loud debate about 2.5 requiring AES-NI. I'll probably limp along with my ancient hardware until support for 2.4 goes away.
-
It should be no problem at 20Mbps.
2.5 is still a way out.Steve
-
and who knows, maybe by the time 2.5 comes along, they will have backed off this pointless aes-ni requirement and won't force retirement of working hardware.
-
While I wish they didn't implement it to be mandatory until the next major revision (and maybe 2.5 is a major revision, idk), I really don't see much of a problem. They gave us something like 2 years notice and 2.4 will be supported for at least a year after 2.5 is released which is likely still some months away as FreeBSD12 isn't even out yet. By that time most of us will have more old equipment that supports aes-ni laying around. For my company it means having to spend about another $1,500 in hardware to replace 6 more devices with APU2s but that's a small price compared to purchasing the alternatives.
I do feel bad for people who paid money for devices specifically for pfSense, like the APU1 series, only to find it will be retired 3 years later but the 2.4 line will still work in the them without issue forever. You still get 5 years of supported service life out of the equipment. The other big name firewalls we use are SonicWall and they don't offer anything beyond the 5 year mark, either.
I don't know what precipitated the requirement but I appreciate the big heads up. I hope it's for more than VPN traffic and it is somehow used foundationally to further enhance security. I also hope it clears up the confusion as to what the aes-ni settings should be to get the best performance out of our boxes. I guess we'll see once it launches.
-
@stewart said in PC Engines apu2 experiences:
While I wish they didn't implement it to be mandatory until the next major revision (and maybe 2.5 is a major revision, idk), I really don't see much of a problem. They gave us something like 2 years notice and 2.4 will be supported for at least a year after 2.5 is released which is likely still some months away as FreeBSD12 isn't even out yet. By that time most of us will have more old equipment that supports aes-ni laying around. For my company it means having to spend about another $1,500 in hardware to replace 6 more devices with APU2s but that's a small price compared to purchasing the alternatives.
That would all be reasonable--if there were a compelling reason to force the obsolescence. Since there isn't, it's just obnoxious.
I don't know what precipitated the requirement but I appreciate the big heads up.
As far as I can tell, poking the china box vendors in the eye was what precipitated the requirement.
I hope it's for more than VPN traffic and it is somehow used foundationally to further enhance security. I also hope it clears up the confusion as to what the aes-ni settings should be to get the best performance out of our boxes. I guess we'll see once it launches.
You leave the settings alone; the confusion is mostly people who don't know what they're doing repeating things they've read on reddit that were written by other people who don't know what they're doing. I would not be at all surprised if there are people pushing "tricks" to "speed up" crypto by doing idiotic things to override the defaults long after 2.5 is released.
-
@vamike said in PC Engines apu2 experiences:
You leave the settings alone; the confusion is mostly people who don't know what they're doing repeating things they've read on reddit that were written by other people who don't know what they're doing. I would not be at all surprised if there are people pushing "tricks" to "speed up" crypto by doing idiotic things to override the defaults long after 2.5 is released.
Maybe that's the way it is now but it's hasn't always been clear. At one time there was a lot of discussion as to what to set where as the results were all over the place depending on the hardware you had. Right now, do you select hardware or software decryption? Or none at all? It all depends on your hardware and which encrypting you are doing.
-
@stewart said in PC Engines apu2 experiences:
@vamike said in PC Engines apu2 experiences:
You leave the settings alone; the confusion is mostly people who don't know what they're doing repeating things they've read on reddit that were written by other people who don't know what they're doing. I would not be at all surprised if there are people pushing "tricks" to "speed up" crypto by doing idiotic things to override the defaults long after 2.5 is released.
Maybe that's the way it is now but it's hasn't always been clear. At one time there was a lot of discussion as to what to set where as the results were all over the place depending on the hardware you had. Right now, do you select hardware or software decryption? Or none at all? It all depends on your hardware and which encrypting you are doing.
It's been pretty clear except for people posting misunderstood and misleading openssl benchmark results and shooting themselves in the foot. The discussions mostly revolved around fanciful numbers in which screwing up the config would magically make a system do crypto even faster than it could access memory. Just leave it alone has been the correct action for a long time.
-
Mmm, that sslspeed thread was.... um...wild!
The default settings should work well for most. Some tuning can help. The asynchronous-crypto setting in 2.4.4 can dramatically increase ipsec throughput in some situations but can also break it in some edge cases so it not enabled by default in CE.
Steve
- 3 months later
-
PC Engines / 3mdeb have released a new legacy coreboot BIOS v4.0.23.
Interestingly the release notes (https://pcengines.github.io/#lr-15) note that ECC is enabled with this release.
So on my test APU2 apu2c4 I've updated the BIOS from v4.0.7 to v4.0.23. All working well although only storage in use is a 32GB mSATA card.
Unfortunately command
dmidecode -t 17
(this should dump memory config) does not work with coreboot - so it is not easy to verify to determine if ECC is actually working. Just have to wait for some cosmic rays or something ... -
@dugeem Thanks for sharing the result, I just upgraded from v4.0.18 to v4.0.23, all went well (for now ;) )
-
PC Engines / 3mdeb have released another new legacy coreboot BIOS v4.0.24.
The release notes (https://pcengines.github.io/#lr-16) note that CPB is now enabled with this release. CPB = Core Performance Boost ... the AMD equivalent of Intel Turbo Boost.
On my test pfSense system (2.4.4-RELEASE-p2) I'm seeing a useful 5-10% improvement in single core task performance.
Example using openssl:
openssl speed -elapsed -evp aes-128-gcm ... type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes aes-128-gcm 52651.25k 134318.72k 296380.07k 408489.98k 482383.19k
PowerD is enabled in adaptive mode. No significant change in CPU temperature was observed.
/boot/loader.conf.local
#hw.acpi.cpu.cx_lowest=C2
hw.igb.rx_process_limit=-1Kudos to PC Engines & 3mdeb for these continuing BIOS improvements.
Edit: the hints hint.p4tcc.0.disabled & hint.acpi_throttle.0.disabled in loader.conf.local are no longer required as they are now defaults in /boot/device.hints
Edit2: hw.acpi.cpu.cx_lowest=C2 does not apply from loader.conf.local -
I am on Bios v4.0.24 also.
edit: noticed that CPU temperature dropped. Normally it was around 54c now it's around 47c.