Wan periodic reset causes system reboot.
-
@stephenw10 found another way to trigger the issue.
Status interface disconnect wan
System reboots -
@AlexanderK said in Wan periodic reset causes system reboot.:
Status interface disconnect wan
System rebootsThat applies to me too but around 50% of the time. My WAN is delivered by PPPoE so that difference may impact the rate of crashes / reboots.
It is unfortunate that the issue impacts v23.09d.
️ -
i will try to create a lab and reproduce the error. i will post progress
-
anything new?
tried to replicate issue at a lab but nothing happened.
I am using my production network without ipv6 -
There have been some backend changes to our build system preventing new snaps for a few days. Let me check.....
-
We are still digging into this. It looks like there may be several related issues here. The NDP issue being one of them.
-
@stephenw10 said in Wan periodic reset causes system reboot.:
We are still digging into this. It looks like there may be several related issues here. The NDP issue being one of them.
If the coding is not too complicated the understanding of this will wipe at least 3 bugs away. A 4th could be the unexplained DNS Resolver cache wipe following a pfBlocker cron-job. Seems you have more in mind though!
️ -
Well I hope there's not more!
-
@stephenw10 said in Wan periodic reset causes system reboot.:
Well I hope there's not more!
Surely one fix that fixes many is better than chasing down all these individual bugs? Well, unless you are the one unpicking the code...
Will this work fold into v23.09 or is it too late for that?
️ -
I hope that it will be 23.09. The ndp fix certainly will be.
-
Ok, sounds hopeful but I appreciate this discovery came very late in the .09 workflow.
️ -
anything new on this issue?
-
Not yet. At least not as far as I know since we still have yet to replicate it locally. There are fixes for other things that could be interacting to cause this on some systems. If you're able to test a 23.09 snapshot and can repeatedly trigger this issue please do so.
-
Still covered by this on redmine:
No improvement yet on 23.09 dev and the issue is (probably) being pushed to 24.03, so another 6 months+ away.
It's not ideal, I know. I'm looking for a non-pfSense option in the interim to cover the periods when I may not be around to resolve these crashes & reboots.
In the meantime I've been pushing data at the Netgate team and running stuff whenever needed and trying every development load.
️ -
anything on beta 23.09?
-
No, nothing substantive. Netgate did ask me to produce results with a modified kernel but the moving set of instructions left me in a bit of a hole with a router that should have been in production.
I am building-up a new server with pfSense+ so I can test with more freedom but my testing hours with a live WAN are limited. I could do with more help really.
It would really help if Netgate provided a complete version to test with, rather than being left to modify stuff myself in order to provide data for them. They expect quite a bit from a paying customer (although I acknowledge this would be different if they could replicate the issue on their own dev systems).
Anyway, I remain committed and will still invest the time needed where I can.
️ -
Yes, I'm sorry about that. Not being able to replicate it ourselves makes everything much more difficult. It's especially annoying here because I have essentially an identical setup to you. The only significant difference is the connection speed.
We are working to add a coredump implementation in the gui to make this process much easier.
In the mean time I'd be happy to work with you here if you're able to.
If you're able to I would simply reinstall with a much larger SWAP size to avoid the need for an external drive with SWAP.
Also one thing I didn't realise until I tested it was that changes to the pfSnse-ddb.conf file are read in a boot so the system needs to be rebooted normally to apply them before the panic is triggered.
Steve
-
Clearly I am happy to work with you Steve.
I just looked at the swap in the GUI dashboard, only to find there isn't one showing on my Supermicro system. Did I miss a step or something?
[23.05.1-RELEASE][admin@Router-7.redacted.me]/root: swapinfo -h Device Size Used Avail Capacity/root: top last pid: 76190; load averages: 0.14, 0.12, 0.09 up 0+02:00:50 18:24:55 67 processes: 1 running, 66 sleeping CPU: 0.2% user, 0.0% nice, 0.1% system, 0.1% interrupt, 99.6% idle Mem: 139M Active, 391M Inact, 870M Wired, 56K Buf, 29G Free ARC: 223M Total, 39M MFU, 177M MRU, 622K Anon, 1091K Header, 5669K Other 164M Compressed, 419M Uncompressed, 2.56:1 Ratio PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 34097 unbound 16 20 0 425M 316M kqread 3 0:51 3.03% unbound 36766 root 1 20 0 14M 3856K CPU6 6 0:00 0.21% topI don't recall setting a swap on my Netgate 6100 or on this machine. The 6100 shows a swap of 1024 MiB - never seen it used though.
️ -
Mmm, yes by default the Plus installer sets 1G for swap. The CE installer at one time used half the RAM size by default if there was sufficient drive space. I'm not sure why yours you have none.
A 6100 here dumps ~750MB from the kernel when configured to do so but that's after running only a short time. If you can trigger this quickly on the 6100 it would be worth trying since the worst case is that it just fails to hold the dump and reboots. If you're not using the 6100 you can reinstall it and add more SWAP, I would expect 2GB to be more than enough.
Steve
-
@stephenw10
UndergpartI can see 1.0G offreebsd-swap- is the value used by pfSense?[23.05.1-RELEASE][admin@Router-7.redacted.me]/root: gpart show => 40 231270320 nvd0 GPT (110G) 40 532480 1 efi (260M) 532520 1024 2 freebsd-boot (512K) 533544 984 - free - (492K) 534528 2097152 3 freebsd-swap (1.0G) 2631680 228636672 4 freebsd-zfs (109G) 231268352 2008 - free - (1.0M) [23.05.1-RELEASE][admin@Router-7.redacted.me]/root:It's still a bit odd for it to be missing from the GUI though.
I'm happy to use either the Netgate 6100 or the Supermicro (SYS-510D-8C-FN6P) for testing; so your choice with repeatability vs flexibility.
️
