Crash when switching interface OFF and ON again

w0w

I don't think it's related to IPv6, but to RADIX_MPATH option.
https://forum.netgate.com/topic/150986/pf_test-kif-null-if_xname-on-multi-wan-and-reset-all-states-if-wan-ip-address-changes/33
I've tested different variations and also disabled IPv6, nothing.
My current crashes are:

Filename: /var/crash/info.0
Dump header from device: /dev/ada1p3
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 74752
  Blocksize: 512
  Compression: none
  Dumptime: Fri Jul 10 22:17:47 2020
  Hostname: -
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 12.1-STABLE df4360fdf61(devel-12) pfSense
  Panic String: sleeping thread
  Dump Parity: 4138356775
  Bounds: 0
  Dump Status: good

Filename: /var/crash/info.1
Dump header from device: /dev/ada1p3
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 157696
  Blocksize: 512
  Compression: none
  Dumptime: Fri Jul 10 22:28:12 2020
  Hostname: -
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 12.1-STABLE df4360fdf61(devel-12) pfSense
  Panic String: page fault
  Dump Parity: 30683997
  Bounds: 1
  Dump Status: good

Filename: /var/crash/info.2
Dump header from device: /dev/ada1p3
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 157696
  Blocksize: 512
  Compression: none
  Dumptime: Tue Jul 21 13:41:55 2020
  Hostname: -
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 12.1-STABLE fcbc0f88231(devel-12) pfSense
  Panic String: general protection fault
  Dump Parity: 1753744959
  Bounds: 2
  Dump Status: good

Filename: /var/crash/info.3
Dump header from device: /dev/ada1p3
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 157696
  Blocksize: 512
  Compression: none
  Dumptime: Tue Jul 21 13:58:07 2020
  Hostname: -
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 12.1-STABLE cf48cd75cf5(devel-12) pfSense
  Panic String: page fault
  Dump Parity: 680143114
  Bounds: 3
  Dump Status: good

Filename: /var/crash/info.4
Dump header from device: /dev/ada1p3
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 75776
  Blocksize: 512
  Compression: none
  Dumptime: Wed Jul 22 12:01:14 2020
  Hostname: -
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 12.1-STABLE cf48cd75cf5(devel-12) pfSense
  Panic String: general protection fault
  Dump Parity: 387144248
  Bounds: 4
  Dump Status: good

I have ix card also. Played with drivers some time ago. Nothing. Since I've backup firewall I've moved backup to 2.5 and primary to 2.4. The fatal errors are still coming only on 2.5 version even on absolutely different hardware.

JeGr

Then it may be another problem for you @w0w as my appliance uses igb NICs and it's crash-dumping immediatly if something is triggering the NIC like an "internet down" or "interface down" on the WAN. The last few weeks if my ISP has problems and reboots my front router (AVM box) I can wait for the "beep" of my pfSense box shortly after as it crashdumps and reboots.

w0w

@JeGr
No, my backup firewall uses igb and moreover, I have tested em card, same crashes. I do not think it is hardware related, I am pretty sure it is not.

louis2

@w0w said in Crash when switching interface OFF and ON again:

@JeGr
No, my backup firewall uses igb and moreover, I have tested em card, same crashes. I do not think it is hardware related, I am pretty sure it is not.

We simply do not know what is causing the crashes. Can be one bug very deep down or multiple bugs.

However be aware that FreeBSD is in FreeBSD12 redesigning its driver structure.

Where each interface type used to have its own driver, they are transforming that in the direction of "a mini driver per device type" and doing the upperlayer driver things in a new library called "iflib"

I know that on my intel64 system all three IO-ports em, igb and ix are using this principle / the "iflib".

Louis

w0w

@louis2 Yep. But those or similar crashes you have now, I have had when 2.5 was based on 12.0-RELEASE using standard driver. The situation was exactly the same, any movements like up/down, unplugging-replugging interface caused the crash, not always and sometimes it was immediately, sometimes not. After some trial and error, googling, digging and so on I think now that it is radix_mpath option and maybe this feature is not buggy by itself but it shows some system errors that never otherwise appear. The only way I see to test it is to compile the kernel without radix_mpath option enabled.

louis2

@w0w said in Crash when switching interface OFF and ON again:

radix_mpath

radix_mpath is related to multipath rooting. I did not dig into that, but it sounds like route redundancy, something I am not using (what not necessary implies that I am not affected).

I also noticed that there is already a pfSense feature request, noting that there are known issues inside radix_mpath (Feature #9544).
However my personal feeling is that in my case it is more likely to expect the problem in some multicast module or in some generic interface handler.

If …. I just had an "in circuit emulator" here attached to my system / your system and all sources, we would know within hours, ..... since most issues are reproduceable ....... but of course we do not have that kind of very advanced tooling.

If some one would compile a pfSence version having a lot of extra debug statements .... that would probably help as well ....

Louis

w0w

May be @rschell can help, at least with pfSense version compiled without radix_mpath.
Unfortunately, my knowledge is not enough to build the kernel myself.

louis2

@w0w, @rschell

I did open bug at FreeBSD Bugzilla – Bug 248243
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248243

It would help if you support me there by entering info about the issues you noticed (and perhaps your verdict). Of cause supported by facts as far as possible.

Louis

@louis2 I have a 2.5 development version without the RADIX_MPATH option enabled (no additional debug statements) that I could share with you. You have a preference ISO or memstick?

louis2

@rschell, @w0w,

I am not sure RADIX_MPATH is casing the trouble for me, at least not if I do interpret the reason RADIX_MPATH correctly (I do not use multiple paths)

However to be sure I could try a memstick version, if it is based on a very recent generic 2.5 build.

What I really would love to have !!! , is a pfSense debug version tracking all communication between pimd and the freebsd kernel and saving that with timestamps in a file which is not lost in case of a crash.

Louis

w0w

@rschell
Memstick would be great. Thanks!