PC Engines apu2 experiences

daemonix

@stephenw10 said in PC Engines apu2 experiences:

I'd be surprised if that was anything to do with the Coreboot version really. About the only thing I could imagine doing that would be some component that is initiallised differently and only supported in FreeBSD 11.2. But I'm not aware of that.
I would first backup the config and do a clean 2.4.4 install. If you still see the same issues you did in the upgraded 2.4.4 then did deeper. I would expect to see errors logged though.

Steve

I get the same experience too. Clean 2.4.4 install on a apu2c4.
SSH or Serial stalls after printing 10-15 chars.
Web is dead or I get 504.

Basic WAN network works ok.

Any ideas?

stephenw10

Not without some sort of error to go on.

Are you able to get any logs?

Does it stop at the same point every time?

Does the boot log complete as expected via serial?

Steve

daemonix

@stephenw10 said in PC Engines apu2 experiences:

Not without some sort of error to go on.

Are you able to get any logs?

Does it stop at the same point every time?

Does the boot log complete as expected via serial?

Steve

boot did well.
after random time I couldnt get to console or see logs.
on hard restart (PSU off/on) everything goes back to ok.

Ill continue monitoring and let you know.

Stewart

@daemonix
Right now I'm running 3 APU2C4 devices without any problems but they are all upgrades. No clean installs. I don't know the firmware versions, though. Maybe that would be a culprit? Also, can you send the system logs somewhere so they are viewable after the unit locks up? Then you can see the last thing logged. If you stay logged in via serial is there a kernel panic? You would need to stay logged in via serial. Trying to get in after it locks up would be unsuccessful. Might be a driver issue throwing a panic that doesn't get logged.

daemonix

@stewart said in PC Engines apu2 experiences:

@daemonix
Right now I'm running 3 APU2C4 devices without any problems but they are all upgrades. No clean installs. I don't know the firmware versions, though. Maybe that would be a culprit? Also, can you send the system logs somewhere so they are viewable after the unit locks up? Then you can see the last thing logged. If you stay logged in via serial is there a kernel panic? You would need to stay logged in via serial. Trying to get in after it locks up would be unsuccessful. Might be a driver issue throwing a panic that doesn't get logged.

Ill investigate. At the moment after a hard power reset Its going good.
Im monitoring via the serial.

I havent checked the firmware version.

thewhero

Hi, I also have a few issues with my apu2c4 and pfSense 2.4.4 being unresponsive. I have yet to see anything in logs that's acutally helpfull. My monitoring though raises a few alerts per day on web and ssh being unresponsive. Traffic flows normally though.

But I have also found that doing certain operations in the webUI might block all traffic through the firewall and also block any new connections to the firewall itself. So far I have identified that trying to search in the States diagnostics page will mess everyting up. All traffic stops. Last time I fourtnally had an open SSH connection and could reset php-fpm from the console menu, which cleared everthing without the need of a hard boot. Top did not show any execessive CPU load, just a normal idle system, and I did not see anything interesteing in any logs. (Not reported yet.)

It's a clean full install of 2.4.4. Configration was restored from the previous 2.4.3 install, which ran just fine on the hardware.

Coreboot is of version 4.0.7. I'm considering upgrade to 4.8.0.x just to test.

daemonix

@thewhero said in PC Engines apu2 experiences:

Coreboot is of version 4.0.7. I'm considering upgrade to 4.8.0.x just to test.

which tutorial do you follow for updating the coreboot?
cheers

thewhero

@daemonix said in PC Engines apu2 experiences:

which tutorial do you follow for updating the coreboot?
cheers

I saw this post: https://forum.netgate.com/topic/120380/pc-engines-apu2-bios-options and have decided to use flashrom directly from pfSense instead of booting a USB-stick to run flashrom. I have verified that flashrom can indeed communicate with the flash by dumping current flash image to disk.

[2.4.4-RELEASE][admin@fw]/root: flashrom --programmer internal --read flash.img
flashrom v1.0 on FreeBSD 11.2-RELEASE-p3 (amd64)
flashrom is free software, get the source code at https://flashrom.org

Using clock_gettime for delay loops (clk_id: 4, resolution: 2ns).
coreboot table found at 0xdffae000.
Found chipset "AMD FCH".
Enabling flash write... OK.
Found Winbond flash chip "W25Q64.V" (8192 kB, SPI) mapped at physical address 0x00000000ff800000.
Reading flash... done.

Now I just need to find a suitable maitenance window so I have time to recover if anything goes wrong.

Veldkornet

Indeed I currently flash from pfSense directly as well.

Just install flashrom with the following:

pkg install flashrom

Then, because you're coming from an old version, you'll probably need to force it since they changed the naming conventions:

flashrom -w /tmp/apu2_v4.8.0.5.rom -p internal:boardmismatch=force

You can find all of the latest firmware versions here.

Just a note, on the 4.8.X releases, there is some bug where the system will hang on a reboot if it's been up and running for a while.

Also, if you haven't already done so, you will need to add the following to your /boot/loader.conf:

boot_serial="YES"
comconsole_speed="115200"
console="comconsole"
hint.ahci.0.msi="0"
loader_conf_files="/boot/device.hints"

Other than that, the new FW's are fine.

Veldkornet

I noticed in my 2.4.4 /var/log/dmesg.boot the following:

module_register_init: MOD_LOAD (vesa, 0xffffffff81209800, 0) error 19

But the default config is not to load vesa:

vesa_load="NO"

And if I try load it manually, I get the following:

kldload vesa
kldload: can't load vesa: No such file or directory

Anyone know what this is and how to fix it? Why is it trying to load?

kevindd992002

@veldkornet said in PC Engines apu2 experiences:

Indeed I currently flash from pfSense directly as well.

Just install flashrom with the following:
pkg install flashrom
Then, because you're coming from an old version, you'll probably need to force it since they changed the naming conventions:
flashrom -w /tmp/apu2_v4.8.0.5.rom -p internal:boardmismatch=force
You can find all of the latest firmware versions here.

Just a note, on the 4.8.X releases, there is some bug where the system will hang on a reboot if it's been up and running for a while.

Also, if you haven't already done so, you will need to add the following to your /boot/loader.conf:
boot_serial="YES"
comconsole_speed="115200"
console="comconsole"
hint.ahci.0.msi="0"
loader_conf_files="/boot/device.hints"
Other than that, the new FW's are fine.

Does this mean that as long as you add those lines to /boot/loader.conf, running the latest FW's is fine?

Veldkornet

@kevindd992002 said in PC Engines apu2 experiences:

Does this mean that as long as you add those lines to /boot/loader.conf, running the latest FW's is fine?

I'm currently running 4.8.0.5 on pfSense 4.2.2 with a SSD in ZFS and except for the small things I mentioned about the reboot not working if the system had been running for a long time, all seems to be fine. I have those lines in my config as well.

kevindd992002

@veldkornet

I see. So no fix yet for the system hang on a reboot issue yet? Even just a workaround of any sort?

Veldkornet

@kevindd992002 said in PC Engines apu2 experiences:

@veldkornet

I see. So no fix yet for the system hang on a reboot issue yet? Even just a workaround of any sort?

Yes, pull out the power plug

I don't see it as a major issue, the newer versions have more improvements so I'll stay on it. You'll have to have a read through all of the changes if you want to see everything. 4.8.0.5 now supports ECC for example.
I linked to the reboot issue on Github somewhere above if you want to follow it.

kevindd992002

@veldkornet

Argh. It will be an issue for me if I manage a pfsense box remotely.

Ok, I'll take a look at that then. Why can't they fix it?

Veldkornet

@kevindd992002 said in PC Engines apu2 experiences:

@veldkornet

Argh. It will be an issue for me if I manage a pfsense box remotely.

Ok, I'll take a look at that then. Why can't they fix it?

I don't know? Read through this.

TheWaterbug

How does the APU2 stack up against the MBT-2220, performance-wise, for running pfsense, IPSec, and OpenVPN?

I needed a new box about a month ago, and since Netgate wasn't offering APU2 units any more I went with an MBT-2220. It works fine, but I miss the 3rd Ethernet port and the internal expansion slots. [After performing installations on both units I also realize I prefer having a serial console, because then I don't need a monitor and keyboard. I just need a cheap USB-Nullmodem cable.] I thought Netgate stopped selling the APU2 because it was obsolete or unavailable, but apparently the APU2 is still widely available at retail, and is a few dollars less expensive than the MBT-2220.

The APU2 has: "AMD Embedded G series GX-412TC, 1 GHz quad Jaguar core with 64 bit and AES-NI support, 32K data + 32K instruction cache per core, shared 2MB L2 cache."

while the MBT-2220 has: "Intel Atom E3826 (2 x 1.46 GHz, 1MB cache, AES-NI)"

I'm not savvy enough to know which hardware is better. I have 3 (and soon to be 4) sites meshed together with IPSec tunnels among them, and I'm adding one site that will be connecting via OpenVPN. Max wire speed at any of these sites is 50 Mbps, and they're typically 20 Mbps or slower.

Thanks!

Stewart

@thewaterbug
We have a lot of the APU2C4 units out there, and they make an APU4 now with 4 ports instead of 3 if that interests you. The largest client we use it at is a hotel with 6 buildings and normally around 150-200 guests at a time on a 500Mbit fiber connection (~30 devices on the Office LAN network and ~150 devices on the Tenant OPT1 network). No issues. Not sure it could go much higher as a max speed but it is running Suricata (On Office and Tenant networks), pfBlocker (On office and Tenant networks), and Squid+ClamAV+SquidGuard (On the Office network only). No issues there. We have other clients using the IPSEC and OpenVPN and they work very well. Don't know top speed but it certainly doesn't feel slow. I can't compare to the MBT-2220 with 2 cores that are likely twice as strong (Jaguar isn't exactly high IPC) but the APU2C4 can do 500Mbps just fine. Lately I've seen Suricata go wonky and cap out the CPUs on several units but that appears to be a log issue. Uninstalling/reinstalling seems to fix it so far. I hope that gives you some understanding.

TheWaterbug

Thanks! That's exactly what I wanted to hear.

Stewart

@thewaterbug
Just don't use snort. It is single-threaded and these don't have great single-threaded power. I've never seen Squid or pfBlocker cause hiccups. Suricata runs much better if you disable stream events but you'll want to keep your eye on it for a bit to make sure it runs smoothly. If there are CPU spikes you'll notice quickly enough.