Need driver for hardware watchdog

stephenw10

Your links seem slightly broken there Cino.
Shouldn't we be using 8.1 anyway? My 2.0 box isn't on I can't check.

Steve

Edit: And my link was also broken! Doesn't like ftp.

MrKoen

Steve, uname -a shows the following here:

FreeBSD <fqdn of="" my="" pfsense="" box="">8.1-RELEASE-p3 FreeBSD 8.1-RELEASE-p3 #1: Fri Apr 29 21:17:50 EDT 2011 root@FreeBSD_8.0_pfSense_2.0-AMD64.snaps.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8 amd64

What does this actually tell me? Is it based on FreeBSD 8.1 or 8.0 since I see both being mentioned.

I tried the PC-BSD ISO, but that didn't work within my VirtualBox image. I'll give it another try with FreeBSD 8.1. Thanks for the link!

Cino, I was running pfSense 2.0 RC1 i386 within VMWare ESXi on my SuperMicro X7SPA-HF-D525 board. This was very unstable with crashes and hangs at least 5 times a day. So I tried an install of pfSense 2.0 RC1 AMD64 directly on the hardware. This is less unstable but still creates hangs at least twice a day. This is with it being updated up to the latest beta version. I tried returning to the stable pfSense 2.0 RC1 i386 release, but I couldn't get it gitsynced with the smos git to include the IPv6 support. I remember you stating in a different forum topic that your installation on similar hardware was stable. I would recommend keeping it like that. The only reason I'm trying to get this watchdog to work is because it is so unstable.</fqdn>

stephenw10

It's based on 8.1.

The only thing I can say I've done personally is installed FreeBSD 8.1 x86 and copied files from that to pfSense 2.0Beta5.

Steve

MrKoen

It worked! Thanks guys for helping me out! I indeed needed the files from the FreeBSD 8.1 release and not the FreeBSD 8.2 release. To show my appreciation for you guys helping me out and deliver something back for it to the community, I took the time to put all steps required in one tutorial for both the i386 as well as the AMD64 versions of pfSense 2.0 RC1 together with the downloads for both of them. Find the tutorial and downloads here:

http://www.zomers.eu/knowledge/pfSense/Pages/Configure-pfSense-2.0-RC1-to-use-Watchdog-functionality.aspx

wallabybob

@Koen:

To show my appreciation for you guys helping me out and deliver something back for it to the community, I took the time to put all steps required in one tutorial for both the i386 as well as the AMD64 versions of pfSense 2.0 RC1 together with the downloads for both of them.

Great way to show your appreciation. Thanks.

stephenw10

Great write up. :)

Steve

MrKoen

Thanks. I'm glad to be able to help others with this great product. And you guys know what the funny thing about all of this is? Since I have the watchdog up and running, my pfSense installation hasn't crashed or freezed anymore for a whole day already! It used to crash or freeze at least 5 times a day before I got this up and running. Can't explain it, but I'm very happy with it ;)

Cino

Really nice write up!! I made have to try 64bit one of days now :-) Just need one more driver and i'll be good to go

Good to hear that your box is stable now

MrKoen

I want to express once more that following the steps to enable the watchdog really has made my pfSense system rock solid and stable now. It has been running for a week already without any problems, a single crash or single hang. I did a reinstall on a new disk last weekend and the crashes and hangs started right again. Once applying the watchdog driver again, the system is solid and stable again. Strange, but true. So if you're using the SuperMicro X7SPA-HF-D525 or equivalent board and experience an unstable pfSense installation, apply these easy steps.

stephenw10

Weird? :-
I sounds like the perhaps the ICH is not being correctly setup by either the bios or the standard ICH driver.
Loading the ICHWD driver sets the registers to known values preventing some unstable condition.
Speculation. ;)

Steve

dzeanah

Nice write-up.

Does anyone know if this same procedure will work with a Netgate hamakua?

stephenw10

It should work on anything that has an Intel ICH (I/O Controller Hub). Except that it can be disabled by the manufacturer.
See the man page here.

Steve

_igor_

stephenw10: thanks for your tutorial, but seems as if it doesn't work when your harddisks are configured as AHCI: ichwd.ko does not work/load as expected. So i searched and found a reasonable thing to solve that problem:

First of all i installed freeipmi:

pkg_add -r freeipmi

That installs a bunch of nice tools to control your hardware like sensors, voltages and so on. But here only one program is interesting: bmc-watchdog.

Next i created a shell-script to start/enable the bmc-watchdog, i called it "watchdog.sh":

#!/bin/sh
# /usr/local/etc/rc.d/watchdog.sh
# First stop the running bios-watchdog:
/usr/local/sbin/bmc-watchdog -y
# Next start the watchdog with new settings
# -d start bmc-watchdog as daemon
# -i xx is the initial timer, which generates the reset
# -e xx is our timer which resets to initial timer
/usr/local/sbin/bmc-watchdog -d -i 16 -e 10

# alternative which should run too,
# but not in conjunction with bmc-watchdog
# not tested in my case :)
# /etc/rc.d/watchdogd forcestart

after creating the watchdog.sh, please change permissions:

chmod a+x /usr/local/etc/rc.d/watchdog.sh

Now we configure freeipmi, freeipmi.conf is located in /usr/local/etc/freeipmi/
The freeipmi.conf is rather big, but easy to understand. All options are disabled, so we have to enable some things:

The only relevant part is this one:

#####################################################################################################
#
# BMC-WATCHDOG OPTIONS
#
# The following options are specific to bmc-watchdog(8).  They will be
# ignored by other tools.
#
# bmc-watchdog-workaround-flags workaround1,workaround2,workaround3

bmc-watchdog-logfile /var/log/watchdog.log
#
# bmc-watchdog-no-logging DISABLE
#
#####################################################################################################

You can test your script by running it:

/usr/local/etc/rc.d/watchdog.sh

Now get the info via "bmc-watchdog -g". It will output something like this:

Timer Use: SMS/OS
Timer: Stopped
Logging: Enabled
Timeout Action: None
Pre-Timeout Interrupt: None
Pre-Timeout Interval: 0 seconds
Timer Use BIOS FRB2 Flag: Clear
Timer Use BIOS POST Flag: Clear
Timer Use BIOS OS Load Flag: Clear
Timer Use BIOS SMS/OS Flag: Set
Timer Use BIOS OEM Flag: Clear
Initial Countdown: 16 seconds
Current Countdown: 16 seconds

Next of it reboot your pfsense and enter the BIOS-settings.

Go to "Advanced", "IPMI Configuration" and enable the watchdog here, but ONLY here!
Set it to power-recycle or hard-reset, and timeout to 5 min.

That setting gives your pfsense enough time to get up and running after restart. Even when a file-system-check happens, it will be enough time!
(tested with a geom_mirror after a hard reset - disks have 250GB.)

After rebooting you can control the thing:

ssh into your pfsense and see what "bmc-watchdog -g" outputs:

Timer Use: BIOS POST
Timer: Running
Logging: Enabled
Timeout Action: Power Cycle
Pre-Timeout Interrupt: None
Pre-Timeout Interval: 0 seconds
Timer Use BIOS FRB2 Flag: Clear
Timer Use BIOS POST Flag: Clear
Timer Use BIOS OS Load Flag: Clear
Timer Use BIOS SMS/OS Flag: Set
Timer Use BIOS OEM Flag: Clear
Initial Countdown: 300 seconds
Current Countdown: 93 seconds

When your pfsense is started completely (after the little start-sound), control again:

bmc-watchdog -g
Timer Use: BIOS POST
Timer: Running
Logging: Enabled
Timeout Action: Power Cycle
Pre-Timeout Interrupt: None
Pre-Timeout Interval: 0 seconds
Timer Use BIOS FRB2 Flag: Clear
Timer Use BIOS POST Flag: Clear
Timer Use BIOS OS Load Flag: Clear
Timer Use BIOS SMS/OS Flag: Set
Timer Use BIOS OEM Flag: Clear
Initial Countdown: 16 seconds
Current Countdown: 15 seconds

e voila, all is running as expected!
Test it with a "killall -9 bmc-watchdog"
Your countdown goes to 0 and your pfsense reboots.

Looking at your watchdog.log you see the timer-resets:

[May 11 19:47:25]: BMC-Watchdog Timer Reset
[May 11 19:47:35]: BMC-Watchdog Timer Reset
[May 11 19:47:45]: BMC-Watchdog Timer Reset
[May 11 19:47:55]: BMC-Watchdog Timer Reset
[May 11 19:48:05]: BMC-Watchdog Timer Reset
[May 11 19:48:15]: BMC-Watchdog Timer Reset
[May 11 19:48:25]: BMC-Watchdog Timer Reset
[May 11 19:48:35]: BMC-Watchdog Timer Reset

So loading the "ichwd.ko" is not any more necessary!

edit (forgot it to mention): please create a new file in /boot:

/boot/loader.conf.local with following setting in:

ipmi_load="YES"

If you put that setting in your loader.conf, it will be lost with the next firmware-update.

stephenw10

That's good stuff. What other treasures does IPMI hold? ;D

Thanks should go to Koen Zomers for taking the time to write up the procedure. Your write up seems equally comprehensive. Two options for using a watchdog is better than one.

Steve

_igor_

one other fine thing is that:

supermicro X7SBA-D525-HF:

ipmi-sensors
4: System Temp (Temperature): 37.00 C (-7.00/77.00): [OK]
71: CPU Temp (Temperature): 33.00 C (-8.00/90.00): [OK]
138: CPU FAN (Fan): 4840.00 RPM (585.00/29815.00): [OK]
205: SYS FAN (Fan): 5025.00 RPM (585.00/29815.00): [OK]
272: CPU Vcore (Voltage): 1.14 V (0.66/1.41): [OK]
339: Vichcore (Voltage): 1.04 V (0.82/1.18): [OK]
406: +3.3VCC (Voltage): 3.26 V (2.88/3.65): [OK]
473: VDIMM (Voltage): 1.53 V (1.33/1.66): [OK]
540: +5 V (Voltage): 4.96 V (4.32/5.60): [OK]
607: +12 V (Voltage): 12.30 V (10.60/13.20): [OK]
674: +3.3VSB (Voltage): 3.26 V (2.88/3.65): [OK]
741: VBAT (Voltage): 3.17 V (2.62/3.39): [OK]
808: Chassis Intru (Platform Chassis Intrusion): [OK]
875: PS Status (Power Supply): [Presence detected][Unrecognized State][Unrecognized State][Unrecognized State][Unrecognized State][Unrecognized State][Unrecognized State][Unrecognized State]

Cino

igor, great find!! Took me a while to get this to work, I had added ipmi.ko to my loader.conf.local a while back. Removed it and all is working now on my X7SPA-D510-HF.. The ipmi-sensors is great add-on!!


ipmi-sensors
4: System Temp (Temperature): 54.00 C (-7.00/77.00): [OK]
71: CPU Temp (Temperature): 61.00 C (-8.00/90.00): [OK]
138: CPU FAN (Fan): NA (NA/NA): [Unknown]
205: SYS FAN (Fan): 4840.00 RPM (585.00/29815.00): [OK]
272: CPU Vcore (Voltage): 1.16 V (0.66/1.41): [OK]
339: Vichcore (Voltage): 1.04 V (0.82/1.18): [OK]
406: +3.3VCC (Voltage): 3.26 V (2.88/3.65): [OK]
473: VDIMM (Voltage): 1.83 V (1.48/1.99): [OK]
540: +5 V (Voltage): 4.96 V (4.32/5.60): [OK]
607: +12 V (Voltage): 12.29 V (10.50/13.06): [OK]
674: +3.3VSB (Voltage): 3.26 V (2.88/3.65): [OK]
741: VBAT (Voltage): 3.15 V (2.62/3.39): [OK]
808: Chassis Intru (Platform Chassis Intrusion): [OK]
875: PS Status (Power Supply): [Presence detected][Unrecognized State][Unrecognized State][Unrecognized State][Unrecognized State][Unrecognized State][Unrecognized State][Unrecognized State]

btw, I have both watchdogd and bmc-watchdog running on my box. They both work, but I need to play with the options for bmc-watchdog. Watchdogd does a hard reset, but bmc-watchdog is doing a power cycle. Which I think is better for the hard drive but it still needs to clean the filesystem when it boots up.

Thanks again everyone!

MrKoen

Thanks igor for sharing your findings! Looking good. I'm going to give it a try in the next few days. Who can create a nice frontpage widget to parse and show the ipmi-sensors information? ;)

_igor_

cino: Thanks for the hint with ipmi="YES" in loader.conf. I forgot it to mention in my post. I added that setting there, it should be complete now.

Best practise is to set this in /boot/loader.conf.local. So it survives upgrades of pfsense.

Cino

@_igor_:

cino: Thanks for the hint with ipmi="YES" in loader.conf. I forgot it to mention in my post. I added that setting there, it should be complete now.

Best practise is to set this in /boot/loader.conf.local. So it survives upgrades of pfsense.

I'm sorry, I had to remove that line to get bmc_watchdog to work. When ipmi.ko is loaded, the watchdog doesn't work for me. ipmi-sensors would also return an error. I unloaded the driver and both programs started to work with no errors. So i removed it from my loader.conf.local, rebooted the box and everything works. Sorry for the confusing

Cino

@Koen:

Thanks igor for sharing your findings! Looking good. I'm going to give it a try in the next few days. Who can create a nice frontpage widget to parse and show the ipmi-sensors information? ;)

I copied page diag_system_activity.php and changed some of the code so it calls ipmi-sensors instead of top… It works :-) Upload it to /usr/local/www and you can access it via http://pfsense/diag_ipmi-sensors.php.

Scott created the page, and if i'm reading his copyright notice correct. We can make changes to it as long as we leave the notice there.

diag_ipmi-sensors.php.txt