SG-5100 - Hardware watchdog? (watchdogd)

luckman212

Been monkeying around with my SG-5100 trying to get it all figured out. Finally received the extra parts I was waiting for (16G RAM + Intel SSD) and got those installed.

Question: in the BIOS, I see there's a real HW Watchdog, but it's disabled:
0_1538757477576_e7963d7a-5e5a-4ff2-be0e-67f43b829e86-image.png
I tried enabling it, but it didn't show up in the pfSense GUI. I guess there's no driver for it(?) because the box reboots itself in a loop when the HW dog is enabled. So I had to shut that back off pretty quickly

Is this going to change in the future? I liked the idea that the watchdog was supported on the ADI units. Going to miss that if it's not going to be a thing anymore...

luckman212

Anybody?

jimp

We were experimenting with this a bit internally yet. It might be feasible to bring in the wbwd driver since it appears to be supported in FreeBSD, just not built in our release yet. Needs more testing though.

luckman212

If you want another field tester ... happy to oblige

jimp

If you don't want to wait for us, you could copy /boot/kernel/wbwd.ko from a FreeBSD 11.2 install to /boot/kernel/ on that firewall and then add wbwd_load="YES" to /boot/loader.conf.local and see if it works for you.

luckman212

Oh, great! I'll give that a shot tonight (can't afford a reboot right now)

luckman212

Update: some good news! I copied the wbwd.ko driver from my FreeBSD 11.2-p3 system onto my 5100. Added the wbwd_load="YES" to /boot/loader.conf.local as you described, and rebooted. The driver loaded (confirmed with kldstat -v -n /boot/kernel/wbwd.ko) and I saw some messages in the dmesg scroll to indicate it was detecting the hardware:

Starting watchdog daemon...wbwd0: <Nuvoton NCT6776 (0xc3/0x33) Watchdog Timer> at port 0x2e-0x2f on isa0
wbwd0: Before watchdog attach: Watchdog enabled. Scaling by 60s, timer at 3 (<=180s left). CRF5 0x0a CRF7 0x00

I set the watchdog timer in the BIOS to 3 minutes. Once the firewall booted up, I noticed it rebooted itself within a minute. I noticed watchdogd wasn't automatically starting. I poked through /etc/inc/system.inc and found the relevant sections where the platform is compared to known-compatible models (RCC-VE, ROGUE-1 etc).

I modified a couple of these functions- adding in case switches for SG-5100 and then removed the /boot/loader.conf.local since I noticed the driver gets manually started anyway by watchdogd_configure(). After doing that, the GUI option appeared, and I enabled it. Another reboot, and this time the firewall stayed up on its own! I checked again and found that wbwd.ko was indeed loaded automatically, and watchdogd was running.

To test it, I issued a pkill -9 watchdogd and waited about 2 min. Sure enough, the firewall rebooted. I think there might be a better way to test using the watchdogd command itself—I was reading watchdogd(8) - EXAMPLES but I couldn't figure it out.

I tried to make a Pull Request for these changes, but it looks like the code on GitHub doesn't include those functions. I guess they're private and only part of the factory image? Hopefully you guys can add that in soon

But overall, looking pretty good. Anything else I missed?

P.S.— In case anyone wants to have a go but doesn't have easy access to a FreeBSD system, here's a copy of wbwd.ko you can use:
wbwd.ko.zip

jimp

That's great!

The code for all of that is specific to our hardware so just a part of factory and not in CE. We'll work on getting the device driver integrated and the related code as well.

Thanks for testing!

luckman212

Excellent. Thanks for your help.

In case it helps, in the end it was just 2 very small patches to system.inc to get everything working:

function has_hw_watchdog() {
[..snip..]
        case 'SG-2220':
   >>>  case 'SG-5100':
        case 'uFW':
            return (1);

and

function watchdogd_configure() {
    [..snip..]
    switch ($platform['name']) {
        case 'RCC-VE':
        case 'SG-2220':
            mwexec("/sbin/kldload ichwd");
            break;
   >>>  case 'SG-5100':
   >>>      mwexec("/sbin/kldload wbwd");
   >>>      break;
    }

Do you want me to open a redmine for this, or no?

jimp

No, we have our own internal tracker for issues that only affect our hardware. Thanks, though!

luckman212

Just an update: 1 month in and no issues to report. Everything working well. Hope this gets baked into 2.4.5

luckman212

@jimp Have you guys done any more testing with this? Things were going well for me but I just hit a rough patch and it seems like it might be related to the watchdog.

jimp

The driver is in factory images for 2.4.4-p1 and it will load if you enabled the watchdog option in the advanced options (Misc tab).

I have had it enabled on my 5100 for over a week and it's been rock solid for me. I set the BIOS timeout to 180 seconds and the GUI left at the default 128.

If you aren't on 2.4.4-p1 yet, remove your custom copied wbwd.ko and then upgrade, and you should get the copy from the image.

It's possible your hiccup is from wbwd.ko not being built from the same kernel source as the kernel.

luckman212

Ok, I'll try a fresh install of 2.4.4-p1 today and report back. Much appreciated