PC Engines apu2 experiences

Qinn · Jun 14, 2020, 2:24 PM

@fireodo Just a guess as default Ntopng keeps settings, maybe, uncheck this and then remove the package, then reinstall, maybe vanilla works fine

fireodo · Jun 14, 2020, 2:29 PM

@Qinn said in PC Engines apu2 experiences:

@fireodo Just a guess as default Ntopng keeps settings, maybe, uncheck this and then remove the package, then reinstall, maybe vanilla works fine

It was a vanilla install when that core dump issue occure - i gave it up ...
(And btw: its really overkill for my network)

daemonix · Jun 17, 2020, 11:52 AM

I got some core dumps here again... I think its quite heavy for 24/7 use.

Qinn · Jun 17, 2020, 11:54 AM

@daemonix It's running 24/7 since 6 months never had a problem with it.

daemonix · Jun 17, 2020, 12:02 PM

@Qinn Ill give it another go. all updated but got a core dump.

fireodo · Jun 17, 2020, 12:06 PM

@Qinn said in PC Engines apu2 experiences:

@daemonix It's running 24/7 since 6 months never had a problem with it.

A little question: Have you CPU Boost activated?
https://github.com/pcengines/apu2-documentation/blob/master/docs/apu_CPU_boost.md

apu2 · Jun 17, 2020, 1:00 PM

I recently bought a new APU2 board and installed pfsense 2.4.5 release. BIOS is coreboot version v4.11.0.6.
I had mSATA 64G drive. Installation was easy and straightforward.
This setup was runinig fine for about 1 month after which I started to get the following issues.

Disk started to show some timeouts and disconnects.
I can login to the system but cannot do anything that require disk read.

I am trying to understand if this is a hardware of software issue. Is there anything I can do to troubleshoot the issue further?
PC engines mention bootloader parameter hint.ahci.0.msi="0" but it is applicable only for bios version older than 4.6.7

igb1: link state changed to UP
ahcich0: Timeout on slot 1 port 0
ahcich0: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd d0 serr 00000000 cmd 0040e117
ahcich0: Timeout on slot 18 port 0
ahcich0: is 00000000 cs 00040000 ss 00000000 rs 00040000 tfd c0 serr 00000000 cmd 0040f217
ahcich0: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich0: Timeout on slot 19 port 0
ahcich0: is 00000000 cs 00080000 ss 00000000 rs 00080000 tfd 80 serr 00000000 cmd 0040f317
(aprobe0:ahcich0:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich0:0:0:0): CAM status: Command timeout
(aprobe0:ahcich0:0:0:0): Retrying command
ahcich0: Timeout on slot 20 port 0
ahcich0: is 00000000 cs 00100000 ss 00000000 rs 00100000 tfd 1d0 serr 00000000 cmd 0040f417
(aprobe0:ahcich0:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich0:0:0:0): CAM status: Command timeout
(aprobe0:ahcich0:0:0:0): Error 5, Retries exhausted
ahcich0: AHCI reset: device not ready after 31000ms (tfd = 00000180)
ahcich0: Timeout on slot 21 port 0
ahcich0: is 00000000 cs 00200000 ss 00000000 rs 00200000 tfd 180 serr 00000000 cmd 0040f517
(aprobe0:ahcich0:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich0:0:0:0): CAM status: Command timeout
(aprobe0:ahcich0:0:0:0): Error 5, Retry was blocked
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <MT-64 H180516> s/n 978081794579 detached
g_vfs_done():ufsid/58bf2d4f649f8ad1[READ(offset=30607212544, length=65536)]error = 6
vnode_pager_generic_getpages_done: I/O read error 5
vnode_pager_generic_getpages_done: I/O read error 5
ahcich0: AHCI reset: device not ready after 31000ms (tfd = 00000180)
ahcich0: Timeout on slot 22 port 0
ahcich0: is 00000000 cs 00400000 ss 00000000 rs 00400000 tfd 180 serr 00000000 cmd 0040f617
(aprobe0:ahcich0:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich0:0:0:0): CAM status: Command timeout
(aprobe0:ahcich0:0:0:0): Retrying command
ahcich0: Timeout on slot 23 port 0
ahcich0: is 00000000 cs 00800000 ss 00000000 rs 00800000 tfd 1d0 serr 00000000 cmd 0040f717

thanks!

fireodo · Jun 17, 2020, 1:00 PM

@apu2 said in PC Engines apu2 experiences:

I recently bought a new APU2 board and installed pfsense 2.4.5 release. BIOS is coreboot version v4.11.0.6.
I had mSATA 64G drive. Installation was easy and straightforward.
This setup was runinig fine for about 1 month after which I started to get the following issues.

Disk started to show some timeouts and disconnects.
I can login to the system but cannot do anything that require disk read.

I am trying to understand if this is a hardware of software issue. Is there anything I can do to troubleshoot the issue further?

You can try to boot in single mode and do a "/sbin/fsck -y / " (repeat until no error show)

PC engines mention bootloader parameter hint.ahci.0.msi="0" but it is applicable only for bios version older than 4.6.7
igb1: link state changed to UP
ahcich0: Timeout on slot 1 port 0

On the other hand it is possible that your SSD is dying ... make a backup of your config for the case that you have to change the drive.

thanks!

You're welcome,

fireodo

Qinn · Jun 17, 2020, 3:19 PM

I would check the advice from @fireodo; "A little question: Have you CPU Boost activated?" as this makes a lot off difference.

Then I may add, always install pfSense using the ZFS filesystem not the UFS (default). For a few reasons;

TRIM is default enabled (I am using mSATA), but this can also be enabled manually on a UFS filesystem, but the most important reason I advice to use ZFS is that it can 100% safegard you against bit errors, because the APU2 has ECC memory

apu2 · Jun 17, 2020, 5:47 PM

@fireodo @Qinn Thanks both!
Good advise for my next ssd :)

Drive actually died and no longer detected by the system. Tried reseating it a few times but no lock.
I am surprised SMART didn't detected anything.
Is there any meaning to run smartd for SSDs?

stephenw10 · Jun 17, 2020, 5:53 PM

Yes. It will show the drive wear level as spare blocks are used for example. It can't show if the controller is about to die.

Steve

sikita · Jun 19, 2020, 6:34 AM

Hi guys, are you running pfSense with /var & /tmp in RAM DISKS enabled on APU2?

fireodo · Jun 19, 2020, 9:26 AM

@sikita said in PC Engines apu2 experiences:

Hi guys, are you running pfSense with /var & /tmp in RAM DISKS enabled on APU2?

Hi, no RamDisk here - as stated in my signature its all on a 16GB SSD.
/var/run 4% von 3.4MiB - ufs im RAM

Qinn · Jun 19, 2020, 9:08 AM

@sikita The whole filesystem zfs is running on 16GB mSATA SSD with ZFS only /var/run
4% of 3.4MiB - ufs in RAM

kinch · Jun 22, 2020, 7:55 AM

i retested. same result.
iperf client pc(lan) -> pfsense APU2E4 -> pc2 (opt) iperf server

With 2.4.4 p3 I have 750-850Mbit for 1 connection
After the update to 2.4.5 P1 I have 400-500Mbit.
Nothing changed, only the update was installed.
For me it means, there has changed something in the performance between 2.4.4 P3 and 2.4.5 (P1). But i dont know what.

BR

dugeem · Jun 22, 2020, 7:55 AM

@kinch
Try disabling ICMP Redirects to enable the faster tryforward routing path (under System / Advanced / System Tunables set net.inet.ip.redirect & net.inet6.ip6.redirect to 0)

sikita · Jun 23, 2020, 10:38 AM

@Qinn said in PC Engines apu2 experiences:

@sikita The whole filesystem zfs is running on 16GB mSATA SSD with ZFS only /var/run
4% of 3.4MiB - ufs in RAM

Ok, thanks for a hint

Stella_161 · Jul 6, 2020, 6:33 PM

Hi,

First id like to say thanks for everyone advice.

Id like to ask for a little help from you.

Im having issues with slow LAN interface compared to the WAN.

I have a 100Mbit PPPoe HFC connection. When running a speed test from the apu4c4 or directly plugged in to the HFC box, i get 98Mbit but if i run it from my PC that's behind the apu4c4 its only at 71Mbit no matter what i do.

Ive tried:

Swapping Ethernet cables
Plugging directly in to LAN Port 2 and 3
https://teklager.se/en/knowledge-base/apu2-1-gigabit-throughput-pfsense/
disabling ICMP Redirects to enable the faster tryforward routing path (under System / Advanced / System Tunables set net.inet.ip.redirect & net.inet6.ip6.redirect to 0)
Upgraded to 2.4.5-RELEASE-p1 (amd64) from 2.4.4
Upgraded the BIOS to: Version: v4.11.0.4
Factory reset with no packages enabled

All with no change to my LAN speed. I'm all out of ideas here, can anyone offer any more advice on what i can try next.

Thanks

sToRmInG · Jul 6, 2020, 5:57 PM

How is your CPU utiluzation when you run your test?
Can you try and update the BIOS to 4.12.0.2 and see if this helps?

Stewart · Jul 6, 2020, 6:34 PM

@Stella_161 This is specific to your box, not a general issue with the devices themselves so you would get a much better response opening a general support thread for this. When you do, you'll need to give more detail, specifically what packages are installed and how you're performing the test. Try a mix of speedtest.net and fast.com and see what you get. Make sure to do them both with an ad-blocker running on your PC. I've seen many instances where advertisements can eat up a CPU on a computer and give artificially slow speed results.

I have many of these boxes that are doing up to 500Mbps so the hardware is more than capable. That being said, @sToRmInG is right in that the first place to look would be the CPU %. Something may be chewing up CPU cycles on either the APU2 or the laptop you are using. Obvious suspects could be Snort / Suricata, Squid / SquidGuard, and pfBlockerNG.