Frequent crashes with APU2

cmb

Kernel hasn't changed in any significant area in between 2.3 and 2.3.1, going back doesn't seem like it'll help. Likely the same root cause between them if I had to guess. What's your IP(v6) IP the crash report came from?

j.koopmann

Constant WAN-IP changes on every login and after the last crash I upgraded and received a new one… It was a crash uploaded in the past 60 minutes I would say.

cmb

Found them. What do you have set for kern.ipc.nmbclusters? Run "sysctl kern.ipc.nmbclusters" if you're not sure.

j.koopmann

1000000

cmb

That's fine. There's mbuf in the backtrace, which at times can be indicative of mbuf exhaustion. Wouldn't be the case there though.

I noticed one potentially relevant change. Let's try going back to 2.3 and see what happens. First backup your config and be ready to reinstall just in case as this isn't widely tested, but it does seem to work fine.

Change your /usr/local/etc/pkg/repos/pfSense.conf to contain the following:

FreeBSD: { enabled: no }

pfSense-core: {
  url: "pkg+http://pkg.pfsense.org/pfSense_v2_3_0_amd64-core",
  mirror_type: "srv",
  signature_type: "fingerprints",
  fingerprints: "/usr/local/share/pfSense/keys/pkg"
  enabled: yes
}

pfSense: {
  url: "pkg+http://pkg.pfsense.org/pfSense_v2_3_0_amd64-pfSense_v2_3_0",
  mirror_type: "srv",
  signature_type: "fingerprints",
  fingerprints: "/usr/local/share/pfSense/keys/pkg"
  enabled: yes
}

Then run:

pkg update -f
pkg upgrade -fy

and reboot when it's done.

j.koopmann

Ok.

Will download 2.2.6 and 2.3 images later, backup and try to step back. Will keep you posted. Might take a few hours.

Thanks for the great support!!!!

j.koopmann

Downgrade worked however the kernel is still

pfSense-kernel-pfSense-2.3.1.a.20160506.1958 pfSense kernel (pfSense)

I got a "is locked" message for this package during upgrade (or rather downgrade). So the one thing that needed downgrading is not… :-(

j.koopmann

ok. pkg unlock and a few more tries resulted in a 2.3 system with kernel

pfSense-kernel-pfSense-2.3 pfSense kernel (pfSense)

Let's wait and see.

j.koopmann

Next crash this time under 2.3. However after reboot no crashlogs/dumps… Will keep looking but probably step down to 2.2.6 tomorrow... Need a stable system next week.

JorgeOliveira

Just out of curiously what are the specs on which you are running pfSense?

This issue may boil down to an incompatibility with any hardware in the system, so details would be appreciated.

Regards,
Jorge M. Oliveira

j.koopmann

Sure,

Standard apu 1c with an msata hdd.

DESIGN-COMPANY

Have the same problem with version 2.3 and 2.3.1 with APU 1d4 Board AMD G-T40E Processor. Interrupts 50%. No traffic on LAN and WAN.

Only power off is possible and then power on.

cmb

The third crash was also a completely different backtrace from the first two. I'm guessing that's some kind of issue after disabling additional cores maybe in combination with https://redmine.pfsense.org/issues/6296

It's definitely not after upgrading to 2.3.1. I suspect once the root IPsec issue is fixed and you're not disabling the additional cores, the crashes will be gone.

@DESIGN-COMPANY:

Have the same problem with version 2.3 and 2.3.1 with APU 1d4 Board AMD G-T40E Processor. Interrupts 50%. No traffic on LAN and WAN.

https://redmine.pfsense.org/issues/6296

j.koopmann

Hi cmb,

just experienced a completely new crash with 2.3, four cores enabled. Looking different as it also gave a backtrace. I uploaded the crash and will give you the details via PM.

Regards,
JP

cmb

From further review and private conversation, this can be summarized as "frequent crashes with APU2", it has no relation to 2.3 or 2.3.1. Some of the crashes match things people were getting on APU2 with 2.2.6 as well.

j.koopmann

Agreed.

However: How to proceed (if not in this threat)? PM? A new thread in another forum (and if so which one would be the correct one)?

cmb

You can continue here. I'll move the thread since it's not 2.3.1-related.

The default for AES-NI is off (with the exception of hardware we sell), which is why that one sticks out at me. In the other thread, it wouldn't have been anything config-related as far as certain features in use or not, so anything hardware-related is what came to mind. AES-NI seems most potentially suspect given the remainder of your config.

j.koopmann

Brilliant support (can't really say that enough)!

I turned it off, will wait for the next reboot and then see how this behaves.

I just checked a config-backup from end of march and to the best of my knowledge aesni was turned on there as well:

<crypto_hardware>aesni</crypto_hardware>

And as mentioned in the PM the crashes seem to have started in May (or end of April) after several changes to the config but this part was left untouched. Still I will see what happens and let you know!

Regards,
JP

j.koopmann

No change. Still crashing.

rhyslewis

@j.koopmann:

No change. Still crashing.

Curious that temps are not getting high ?.. I had one freeze up as I had incorrectly mounted the heatsink to the APU2C4 case:)

Do you have one of these boards –> http://www.pcengines.ch/apu2c4.htm