Wan periodic reset causes system reboot.

[AlexanderK]

@stephenw10 yes i use the same hardware.
i have set up the reset at 3am and it is not possible to log the console. i will try to change that time and log it

[AlexanderK]

@stephenw10 tried to reset manually the wan and i had a reset again. the console was like crazy. couldn't check what it was written.

this is my disk space
🔒 Log in to view

does this mean that my swap is disabled?

[AlexanderK]

disabling ipv6 seems to fix the issue!

[RobbieTT]

@AlexanderK said in Wan periodic reset causes system reboot.:

disabling ipv6 seems to fix the issue!

Ok, seems like the issue I ran into at the redline link above.

I need IPv6 so waiting on a fix is my only option.

️

[stephenw10]

No that shows you do have SWAP enabled so I'd expect to see a crash report if it suffered a kernel panic. And your description sounds like a kernel panic. It spews the full process backtrace and message buffer onto the console before rebooting.

We need to see the initial backtrace and panic string when that happens really. Since you can trigger it on demand, can you hook up a serial console to log it?
If not you should be able to stop the auto reboot and scroll back through the buffer with the scroll-lock key at a vga/keyboard console.

[stephenw10]

Are you running tailscale?

[AlexanderK]

@stephenw10 yes i am running tailscale

[stephenw10]

I have IPv6 (from BT) over PPPoE which seems like it should be nearly identical to connections that are hitting this but I cannot trigger it.
There any special trick I'm missing?

[stephenw10]

@AlexanderK said in Wan periodic reset causes system reboot.:

@stephenw10 yes i am running tailscale

Ooo, try disabling that and see if you can still trigger it.

[RobbieTT]

@stephenw10

But I am not running Tailscale.

️

[AlexanderK]

@stephenw10 how i can stop the auto reboot?

[AlexanderK]

@stephenw10 my tailscale is not enabled

[AlexanderK]

after enabling again ipv6 the problem seems to be solved. really strange situation

[RobbieTT]

@AlexanderK said in Wan periodic reset causes system reboot.:

after enabling again ipv6 the problem seems to be solved. really strange situation

For me it is intermittent, with it rebooting 50-60% of the time.

️

[stephenw10]

Edit /etc/pfSense-ddb.conf and remove or comment 'reset' from the script kdb.enter.default line.

Then after a panic the console will remain at the db prompt:

db:0:kdb.enter.default>  capture off
db:0:kdb.enter.default>  textdump dump
Textdump complete.
db> 

You can then scroll back to see the backtrace shown after db:1:pfs> bt.

You will want to restore the reset command to the file after doing that though because otherwise it will not reboot after a panic at 3am which is probably not what you want!

Steve

[AlexanderK]

@stephenw10
thanks steve made that change but now it has been resolved. let's see what will happen

#update reboot happened
the console didn't remain at the db prompt.
I suppose it is something different

🔒 Log in to view

[AlexanderK]

It happened....i will try to upload everything

crash log was huge.
i took some photos but couldn't upload here because the size is big
https://drive.google.com/drive/folders/17wDoVLqgRgBkQdTGhOVvMfkekIQMCg2Q?usp=sharing

[stephenw10]

Yes it's big! The interesting part is right at the beginning of the report. For example:

debug.kdb.panic:panic: kdb_sysctl_panic
cpuid = 3
time = 1691688092
KDB: enter: panic
[ thread pid 30195 tid 100206 ]
Stopped at      kdb_enter+0x32: movq    $0,0x2342e13(%rip)
db:0:kdb.enter.default> textdump set
textdump set
db:0:kdb.enter.default>  capture on
db:0:kdb.enter.default>  run pfs
db:1:pfs> bt
Tracing pid 30195 tid 100206 td 0xfffffe00c73a4900
kdb_enter() at kdb_enter+0x32/frame 0xfffffe00c713baf0
vpanic() at vpanic+0x183/frame 0xfffffe00c713bb40
panic() at panic+0x43/frame 0xfffffe00c713bba0
kdb_sysctl_panic() at kdb_sysctl_panic+0x61/frame 0xfffffe00c713bbd0
sysctl_root_handler_locked() at sysctl_root_handler_locked+0x90/frame 0xfffffe00c713bc20
sysctl_root() at sysctl_root+0x216/frame 0xfffffe00c713bca0
userland_sysctl() at userland_sysctl+0x177/frame 0xfffffe00c713bd50
sys___sysctl() at sys___sysctl+0x5c/frame 0xfffffe00c713be00
amd64_syscall() at amd64_syscall+0x109/frame 0xfffffe00c713bf30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00c713bf30
--- syscall (202, FreeBSD ELF64, __sysctl), rip = 0x3dada5b3892a, rsp = 0x3dada42e1158, rbp = 0x3dada42e1190 ---

That was a crash I triggered manually. Yours will probably be similar to that shown in the bug report linked above.
Unfortunately those pictures are all from after that part.

[AlexanderK]

@stephenw10
https://forums.freebsd.org/threads/tip-log-console-messages.10090/

Tried to enable console logs with no result.
What i can do to help ?

[stephenw10]

Are you not able to scroll back far enough to see the initial backtrace?

pfSense doesn't use the standard log conf file it uses: /var/etc/syslog.d/pfSense.conf
So you'd have to try that there instead. I have never tried that.

Steve