Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Need driver for hardware watchdog

    Scheduled Pinned Locked Moved 2.0-RC Snapshot Feedback and Problems - RETIRED
    64 Posts 8 Posters 41.0k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • W Offline
      wallabybob
      last edited by

      You probably need to invoke a program to "tickle" the watchdog "often enough".

      See the FreeBSD man page for ichwd (http://www.freebsd.org/cgi/man.cgi?query=ichwd&apropos=0&sektion=0&manpath=FreeBSD+8.2-RELEASE&format=html ) and the links in the See Also section.

      1 Reply Last reply Reply Quote 0
      • stephenw10S Offline
        stephenw10 Netgate Administrator
        last edited by

        There are at least two watchdogs on this board.
        As I said in a previous message the watchdog invoked by the bios is likely the one on the Winbond SuperIO chip and NOT the ich watchdog. Thus the ICH driver will not 'pat' it.
        The drivers I posted are for 32bit only so you will need some for 64bit, unless you've already done that.  ;)

        Cino's report of success involved loading the ich drivers and starting the watchdog daemon. This will then set the watchdog and keep 'patting' it. You should leave the ipmi watchdog off in the bios.

        Good luck!  :)

        Steve

        1 Reply Last reply Reply Quote 0
        • M Offline
          MrKoen
          last edited by

          Thanks for your pointers guys. I'm fairly new to the worlds of Unix and Linux, so let me rephrase your comments to see if I understand them correctly.

          1. This particular board probably has two Watchdogs of which one of them is controlled by the BIOS and one can be controlled by the right driver/software
          2. I should not use the BIOS setting since the ichwd driver/software can not tickle that dog to keep it awake and thus it will reboot
          3. I should install a new server with FreeBSD 8.1 AMD64 and retrieve the ichwd.ko and ichwd.ko.symbols files from the /boot/kernel folder and copy them to my pfSense installation. After that I should try the kldload ichwd.ko again and I can kill the FreeBSD 8.1 server again.

          Makes me wonder.. what sets the timeout on the non-BIOS watchdog? Does it work like when the ichwd software sends the non-BIOS watchdog out of the door to guard the garden and whistle at it at some interval to keep it awake? And when it stops whistling at it, a chip on the mainboard activated by ichwd once it started detects the watchdog has fallen asleep because of the lack of whistles and reboots the system?

          1 Reply Last reply Reply Quote 0
          • stephenw10S Offline
            stephenw10 Netgate Administrator
            last edited by

            1. Both are hardware watchdogs and once switched on (by setting a bit on the chip) they will reset the board after a set interval. Both will have adjustable intervals, determined by the contents of a register on the chip. Both require a piece of software to periodically reset a bit in the chip resetting the timer.

            2. The BIOS settings set the reisters in the Winbond SuperIO chip to activate it's watchdog and set it's timer interval. You would then need to use the ipmi diver to 'tickle' it.
            The ICH watchdog is not set by the bios and is off by default. Loading the ICHWD drivers allows watchdogd (the watchdog daemon) to set the appropriate resisters to switch it on. I've not played with it but it would probably also allow setting the timer interval. (edit: with the -t switch, default is 16s).

            3. You will need 64bit versions, that may be the easiest way to get them. There is a 64bit version of pc-bsd which I believe will run as a live cd. You can copy the files from that.

            Steve

            1 Reply Last reply Reply Quote 0
            • M Offline
              MrKoen
              last edited by

              Still no go here. I couldn't find out where and how to get the live bootable FreeBSD CD, so I went a head and installed a FreeBSD 8.2 AMD64 installation within VirtualBox. After finally having done that, I used the FTP command to upload the ichwd.ko and ichwd.ko.symbols files to my FTP server. Next from the console on my pfSense installation I used the same FTP command to retrieve them from my FTP server (using binary mode). I placed the files within /boot/kernel. Now when I run kldload ichwd from within /boot/kernel, I still get the same error:

              KLD ichwd.ko: depends on kernel - not available or version mismatch
              linker_load_file: Unsupported file type
              kldload: can't load ichwd: Exec format error

              I also tried chmod +x ichwd.ko but that makes no difference.

              I have attached the ichwd files I retrieved from FreeBSD 8.2 AMD64 to this post. Anyone knows what else I can try?

              ichwd.ko.png
              ichwd.ko.symbols.png

              1 Reply Last reply Reply Quote 0
              • stephenw10S Offline
                stephenw10 Netgate Administrator
                last edited by

                Hmmm, I thought that 2.0 was based on 8.1.
                What does it show if you type:

                uname -a

                Obviously this has to match.

                Steve

                1 Reply Last reply Reply Quote 0
                • C Offline
                  Cino
                  last edited by

                  I personally haven't tried the AMD64 because of drivers. I may try it one of these days but I'll have to make sure I have drivers first as I dont want to lose my pico-lcd display.

                  PC-BSD 8.0 DVD: ftp://ftp.pcbsd.org/pub/archived/8.0/amd64/PCBSD8.0-x64-DVD.iso
                  PC-BSD 8.2 DVD: ftp://mirrors.isc.org/pub/pcbsd/8.2/amd64/PCBSD8.2-x64-DVD.iso

                  I have used 8.0 in the base to copy of some file, but not drivers.

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S Offline
                    stephenw10 Netgate Administrator
                    last edited by

                    Your links seem slightly broken there Cino.
                    Shouldn't we be using 8.1 anyway? My 2.0 box isn't on I can't check.

                    Link to PC-BSD 8.1 x64

                    Steve

                    Edit: And my link was also broken! Doesn't like ftp.

                    1 Reply Last reply Reply Quote 0
                    • M Offline
                      MrKoen
                      last edited by

                      Steve, uname -a shows the following here:

                      FreeBSD <fqdn of="" my="" pfsense="" box="">8.1-RELEASE-p3 FreeBSD 8.1-RELEASE-p3 #1: Fri Apr 29 21:17:50 EDT 2011    root@FreeBSD_8.0_pfSense_2.0-AMD64.snaps.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8  amd64

                      What does this actually tell me? Is it based on FreeBSD 8.1 or 8.0 since I see both being mentioned.

                      I tried the PC-BSD ISO, but that didn't work within my VirtualBox image. I'll give it another try with FreeBSD 8.1. Thanks for the link!

                      Cino, I was running pfSense 2.0 RC1 i386 within VMWare ESXi on my SuperMicro X7SPA-HF-D525 board. This was very unstable with crashes and hangs at least 5 times a day. So I tried an install of pfSense 2.0 RC1 AMD64 directly on the hardware. This is less unstable but still creates hangs at least twice a day. This is with it being updated up to the latest beta version. I tried returning to the stable pfSense 2.0 RC1 i386 release, but I couldn't get it gitsynced with the smos git to include the IPv6 support. I remember you stating in a different forum topic that your installation on similar hardware was stable. I would recommend keeping it like that. The only reason I'm trying to get this watchdog to work is because it is so unstable.</fqdn>

                      1 Reply Last reply Reply Quote 0
                      • stephenw10S Offline
                        stephenw10 Netgate Administrator
                        last edited by

                        It's based on 8.1.

                        The only thing I can say I've done personally is installed FreeBSD 8.1 x86 and copied files from that to pfSense 2.0Beta5.

                        Steve

                        1 Reply Last reply Reply Quote 0
                        • M Offline
                          MrKoen
                          last edited by

                          It worked! Thanks guys for helping me out! I indeed needed the files from the FreeBSD 8.1 release and not the FreeBSD 8.2 release. To show my appreciation for you guys helping me out and deliver something back for it to the community, I took the time to put all steps required in one tutorial for both the i386 as well as the AMD64 versions of pfSense 2.0 RC1 together with the downloads for both of them. Find the tutorial and downloads here:

                          http://www.zomers.eu/knowledge/pfSense/Pages/Configure-pfSense-2.0-RC1-to-use-Watchdog-functionality.aspx

                          1 Reply Last reply Reply Quote 0
                          • W Offline
                            wallabybob
                            last edited by

                            @Koen:

                            To show my appreciation for you guys helping me out and deliver something back for it to the community, I took the time to put all steps required in one tutorial for both the i386 as well as the AMD64 versions of pfSense 2.0 RC1 together with the downloads for both of them.

                            Great way to show your appreciation. Thanks.

                            1 Reply Last reply Reply Quote 0
                            • stephenw10S Offline
                              stephenw10 Netgate Administrator
                              last edited by

                              Great write up.  :)

                              Steve

                              1 Reply Last reply Reply Quote 0
                              • M Offline
                                MrKoen
                                last edited by

                                Thanks. I'm glad to be able to help others with this great product. And you guys know what the funny thing about all of this is? Since I have the watchdog up and running, my pfSense installation hasn't crashed or freezed anymore for a whole day already! It used to crash or freeze at least 5 times a day before I got this up and running. Can't explain it, but I'm very happy with it ;)

                                1 Reply Last reply Reply Quote 0
                                • C Offline
                                  Cino
                                  last edited by

                                  Really nice write up!! I made have to try 64bit one of days now :-) Just need one more driver and i'll be good to go

                                  Good to hear that your box is stable now

                                  1 Reply Last reply Reply Quote 0
                                  • M Offline
                                    MrKoen
                                    last edited by

                                    I want to express once more that following the steps to enable the watchdog really has made my pfSense system rock solid and stable now. It has been running for a week already without any problems, a single crash or single hang. I did a reinstall on a new disk last weekend and the crashes and hangs started right again. Once applying the watchdog driver again, the system is solid and stable again. Strange, but true. So if you're using the SuperMicro X7SPA-HF-D525 or equivalent board and experience an unstable pfSense installation, apply these easy steps.

                                    1 Reply Last reply Reply Quote 0
                                    • stephenw10S Offline
                                      stephenw10 Netgate Administrator
                                      last edited by

                                      Weird?  :-
                                      I sounds like the perhaps the ICH is not being correctly setup by either the bios or the standard ICH driver.
                                      Loading the ICHWD driver sets the registers to known values preventing some unstable condition.
                                      Speculation.  ;)

                                      Steve

                                      1 Reply Last reply Reply Quote 0
                                      • D Offline
                                        dzeanah
                                        last edited by

                                        Nice write-up.

                                        Does anyone know if this same procedure will work with a Netgate hamakua?

                                        1 Reply Last reply Reply Quote 0
                                        • stephenw10S Offline
                                          stephenw10 Netgate Administrator
                                          last edited by

                                          It should work on anything that has an Intel ICH (I/O Controller Hub). Except that it can be disabled by the manufacturer.
                                          See the man page here.

                                          Steve

                                          1 Reply Last reply Reply Quote 0
                                          • _ Offline
                                            _igor_
                                            last edited by

                                            stephenw10: thanks for your tutorial, but seems as if it doesn't work when your harddisks are configured as AHCI: ichwd.ko does not work/load as expected. So i searched and found a reasonable thing to solve that problem:

                                            First of all i installed freeipmi:

                                            pkg_add -r freeipmi

                                            That installs a bunch of nice tools to control your hardware like sensors, voltages and so on. But here only one program is interesting: bmc-watchdog.

                                            Next i created a shell-script to start/enable the bmc-watchdog, i called it "watchdog.sh":

                                            #!/bin/sh
# /usr/local/etc/rc.d/watchdog.sh
# First stop the running bios-watchdog:
/usr/local/sbin/bmc-watchdog -y
# Next start the watchdog with new settings
# -d start bmc-watchdog as daemon
# -i xx is the initial timer, which generates the reset
# -e xx is our timer which resets to initial timer
/usr/local/sbin/bmc-watchdog -d -i 16 -e 10

# alternative which should run too,
# but not in conjunction with bmc-watchdog
# not tested in my case :)
# /etc/rc.d/watchdogd forcestart

                                            after creating the watchdog.sh, please change permissions:

                                            chmod a+x /usr/local/etc/rc.d/watchdog.sh

                                            Now we configure freeipmi, freeipmi.conf is located in /usr/local/etc/freeipmi/
                                            The freeipmi.conf is rather big, but easy to understand. All options are disabled, so we have to enable some things:

                                            The only relevant part is this one:

                                            #####################################################################################################
#
# BMC-WATCHDOG OPTIONS
#
# The following options are specific to bmc-watchdog(8).  They will be
# ignored by other tools.
#
# bmc-watchdog-workaround-flags workaround1,workaround2,workaround3

bmc-watchdog-logfile /var/log/watchdog.log
#
# bmc-watchdog-no-logging DISABLE
#
#####################################################################################################

                                            You can test your script by running it:

                                            /usr/local/etc/rc.d/watchdog.sh

                                            Now get the info via "bmc-watchdog -g". It will output something like this:

                                            Timer Use:                  SMS/OS
                                            Timer:                      Stopped
                                            Logging:                    Enabled
                                            Timeout Action:              None
                                            Pre-Timeout Interrupt:      None
                                            Pre-Timeout Interval:        0 seconds
                                            Timer Use BIOS FRB2 Flag:    Clear
                                            Timer Use BIOS POST Flag:    Clear
                                            Timer Use BIOS OS Load Flag: Clear
                                            Timer Use BIOS SMS/OS Flag:  Set
                                            Timer Use BIOS OEM Flag:    Clear
                                            Initial Countdown:          16 seconds
                                            Current Countdown:          16 seconds

                                            Next of it reboot your pfsense and enter the BIOS-settings.

                                            Go to "Advanced", "IPMI Configuration" and enable the watchdog here, but ONLY here!
                                            Set it to power-recycle or hard-reset, and timeout to 5 min.

                                            That setting gives your pfsense enough time to get up and running after restart. Even when a file-system-check happens, it will be enough time!
                                            (tested with a geom_mirror after a hard reset - disks have 250GB.)

                                            After rebooting you can control the thing:

                                            ssh into your pfsense and see what "bmc-watchdog -g" outputs:

                                            Timer Use:                  BIOS POST
                                            Timer:                      Running
                                            Logging:                    Enabled
                                            Timeout Action:              Power Cycle
                                            Pre-Timeout Interrupt:      None
                                            Pre-Timeout Interval:        0 seconds
                                            Timer Use BIOS FRB2 Flag:    Clear
                                            Timer Use BIOS POST Flag:    Clear
                                            Timer Use BIOS OS Load Flag: Clear
                                            Timer Use BIOS SMS/OS Flag:  Set
                                            Timer Use BIOS OEM Flag:    Clear
                                            Initial Countdown:          300 seconds
                                            Current Countdown:          93 seconds

                                            When your pfsense is started completely (after the little start-sound), control again:

                                            bmc-watchdog -g
                                            Timer Use:                  BIOS POST
                                            Timer:                      Running
                                            Logging:                    Enabled
                                            Timeout Action:              Power Cycle
                                            Pre-Timeout Interrupt:      None
                                            Pre-Timeout Interval:        0 seconds
                                            Timer Use BIOS FRB2 Flag:    Clear
                                            Timer Use BIOS POST Flag:    Clear
                                            Timer Use BIOS OS Load Flag: Clear
                                            Timer Use BIOS SMS/OS Flag:  Set
                                            Timer Use BIOS OEM Flag:    Clear
                                            Initial Countdown:          16 seconds
                                            Current Countdown:          15 seconds

                                            e voila, all is running as expected!
                                            Test it with a "killall -9 bmc-watchdog"
                                            Your countdown goes to 0 and your pfsense reboots.

                                            Looking at your watchdog.log you see the timer-resets:

                                            [May 11 19:47:25]: BMC-Watchdog Timer Reset
                                            [May 11 19:47:35]: BMC-Watchdog Timer Reset
                                            [May 11 19:47:45]: BMC-Watchdog Timer Reset
                                            [May 11 19:47:55]: BMC-Watchdog Timer Reset
                                            [May 11 19:48:05]: BMC-Watchdog Timer Reset
                                            [May 11 19:48:15]: BMC-Watchdog Timer Reset
                                            [May 11 19:48:25]: BMC-Watchdog Timer Reset
                                            [May 11 19:48:35]: BMC-Watchdog Timer Reset

                                            So loading the "ichwd.ko" is not any more necessary!

                                            edit (forgot it to mention): please create a new file in /boot:

                                            /boot/loader.conf.local with following setting in:

                                            ipmi_load="YES"

                                            If you put that setting in your loader.conf, it will be lost with the next firmware-update.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.