Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    LAN adapter watchdog timeout when running heavy load on pfSense

    Scheduled Pinned Locked Moved Hardware
    18 Posts 9 Posters 33.9k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • G
      gschoch
      last edited by

      Hi

      given that pfSense is a real great product we decided to switch all our firewalls from IPCop and MonoWall to pfSense. The installation went smooth!

      Version: 1.0.1

      Environment:

      VIA EPIA-EK1000G Mini ITX with to LAN Cards (vr0 and vr1) integrated
      one additional Linksys LNE100TX (dc0)

      Problem:

      when running a larger traffic (e.g. Filetransfer) across the firewall (LAN<->WAN = about 40Mbps) than we get suddently

      #kernel vr0: watchdog timeout

      The problem can occur after 5 Seconds or 20 Minutes

      Tests 1:

      I changed vr0 (LAN) against the dc0 card

      Result: #kernel dc0: watchdog timeout

      Test 2:

      I assumed that the LAN environment could lead to an error. I tested several times the load test.

      Result: #kernel vr1: watchdog timeout

      That means that the WAN side disappeared

      Test 3:

      using only 2 external LAN-Cards (with an Adapter to the Mini ITX Board = Total 4 LAN Boards)

      Result: #kernel em0: watchdog timeout

      Additional hints:

      • we do have this Problem since using pfSense = we started a few monthes ago with RC 1
      • we had this problem as well with the older version of the VIA Mini ITX Board (800MHz)

      would be great if anybody had a hint for me!

      regards

      Günther

      1 Reply Last reply Reply Quote 0
      • G
        gschoch
        last edited by

        Hi again

        at least a little hint what I should test, replace, update?
        Does nobody had ever such problems?

        regards

        Günther

        1 Reply Last reply Reply Quote 0
        • H
          hoba
          last edited by

          I haven't noticed this yet, not even with a wrap when heavily overloading it. What cpu load do you see when doing that loadtest? Is the webgui/ssh still responsive during the test? Also do problems appear when these messages show up or does everything still work like expected besides the log entries?

          1 Reply Last reply Reply Quote 0
          • R
            rsw686
            last edited by

            Found a post on this http://forum.ev1servers.net/showthread.php?t=55633
            It suggests the following.

            a) turn ACPI support OFF
            b) turn PNP OS support OFF [in bios]

            if possible

            c) change the NIC to some other brand like 3com or Intel.

            1 Reply Last reply Reply Quote 0
            • G
              gschoch
              last edited by

              work like expected besides the log entries?

              the system does fail and has to be rebooted. If the WAN network cards fails than it's still possible to access the Firewall via Web-GUI or SSH. The console does work in all cases

              a) turn ACPI support OFF
              b) turn PNP OS support OFF [in bios]
              if possible
              c) change the NIC to some other brand like 3com or Intel.

              thank you for the link. At the end of this thread I found

              Disabling ACPI did not help.
              Dedicating IRQ for the NIC did not help.
              Changing the hardware (nic/mb) did help, havent seen the error for a week now. So 'watchdog timeouts' is mostly hardware related problem.

              I think I will give a try to the last solution

              regards

              Günther

              1 Reply Last reply Reply Quote 0
              • A
                abcdefg1675
                last edited by

                @gschoch:

                Problem:

                when running a larger traffic (e.g. Filetransfer) across the firewall (LAN<->WAN = about 40Mbps) than we get suddently

                #kernel vr0: watchdog timeout

                The problem can occur after 5 Seconds or 20 Minutes

                Im having problems similar to this. K6-500, 415MB ram. Regardless of what NICs I try (compaq dual-port server NIC, or generic cheapos), Pfsense locks up when I get close to 10-15Mbps, and occasionally at lower speeds with heavy use such as uTorrent. I suspect it may be a problem with the actual computer running pfsense (bad ram, bad cpu??). I ran Pfsense under VMware, on a 1.8Ghz "3000+" amd computer. Performance as far as throughput goes was no better than the K6-500, but I havent been able to reproduce any of the crashing I was experiancing.

                gschoch, do you have any way of monitoring temperatures (CPU, memory etc) in the routers? Ive had issues before with other PC's that wernt getting good ventilation, and hanging/crashing when getting well over 60-70+C.

                1 Reply Last reply Reply Quote 0
                • G
                  gschoch
                  last edited by

                  …, do you have any way of monitoring temperatures (CPU, memory etc) in the routers?
                  Ive had issues before with other PC's that wernt getting good ventilation,
                  and hanging/crashing when getting well over 60-70+C.

                  thank you for the answer. Yes we monitored and have about 6 pfSense boxes (but the same age or cooling or memory). All of them do have in common that they are based on VIA Mini ITX boards. All other components are different. But in all cases we do end up with this timeout errors. We will test now another set of NICs and if this does not help the Mini ITX boards have to be switched to another product.

                  regards

                  Günther

                  1 Reply Last reply Reply Quote 0
                  • G
                    gschoch
                    last edited by

                    Hi

                    just for others with the same problem:

                    we switched to a D-Link DFE-580TX Networkadapter with 4 Ports and since then we had nomore any watchdog timeout

                    regards

                    Günther

                    1 Reply Last reply Reply Quote 0
                    • U
                      unreal1024
                      last edited by

                      This is problem FreeBSD v. 6.1
                      Im using two card 3Com 3C905CX-TX-M, disable ACPI, disable sound blaster, change network cards, disable PNP OS, disable COM, LPT, change VGA card.
                      No change anything.

                      Im using pfSense v. 1.0.1:
                      (filtered…)

                      Dec 8 05:24:45 kernel: xl1: watchdog timeout
                      Dec 7 19:26:13 kernel: xl1: watchdog timeout
                      Dec 7 16:03:00 kernel: xl1: watchdog timeout
                      Dec 7 11:15:14 kernel: xl1: watchdog timeout
                      Dec 7 09:18:26 kernel: xl1: watchdog timeout
                      Dec 7 03:43:00 kernel: xl1: watchdog timeout
                      Dec 7 03:40:29 kernel: xl1: watchdog timeout
                      Dec 7 03:35:51 kernel: xl1: watchdog timeout
                      Dec 6 20:34:35 kernel: xl1: watchdog timeout
                      Dec 6 18:29:09 kernel: xl1: watchdog timeout
                      Dec 6 17:15:44 kernel: xl1: watchdog timeout
                      Dec 6 16:27:28 kernel: xl0: watchdog timeout
                      Dec 6 13:36:52 kernel: xl1: watchdog timeout
                      Dec 5 17:29:39 kernel: xl1: watchdog timeout
                      Dec 5 15:12:16 kernel: xl1: watchdog timeout
                      Dec 5 12:34:30 kernel: xl1: watchdog timeout
                      Dec 5 10:43:28 kernel: xl1: watchdog timeout
                      Dec 5 10:39:41 kernel: xl1: watchdog timeout

                      1 Reply Last reply Reply Quote 0
                      • S
                        SatireWolf
                        last edited by

                        for reference, broadcom based integrated chips tend to watchdog timeout under VERY heavy loads but they reset and go on with life. I've seen this happen on AMD Opteron and Intel Xeon platforms with multiple broadcom chips. I just use Intel NICS for everything critical and I don't have issues.

                        As for 3com's the 3c905b's were perhaps the best 10/100 NIC's ever built in my opinion. But when they did the die shrink and built the 3c905c's they were terrible. I even have problems with those in Windows boxes and Linux boxes!

                        My recommendation would be to stay away from newer 3com cards, period. There's a reason Intel has outsold, and blown them out of the water in the NIC market. Besides, Intel continuously contributes code for their NIC drivers. So they just 'work' T.M.

                        1 Reply Last reply Reply Quote 0
                        • S
                          sullrich
                          last edited by

                          Upgrade to http://www.pfsense.com/~sullrich/1.0.1-SNAPSHOT-12-06-2006/ and see if the problems persist.

                          1 Reply Last reply Reply Quote 0
                          • D
                            DWAyotte
                            last edited by

                            I had the same problem earlier today.

                            Dec 9 14:57:03 kernel: sk0: link state changed to UP
                            Dec 9 14:33:46 kernel: arplookup 231.57.128.57 failed: host is not on local network
                            Dec 9 14:08:17 php: : Hotplug event detected for sk0 but ignoring since interface is not set for DHCP
                            Dec 9 14:08:17 check_reload_status: rc.linkup starting
                            Dec 9 14:08:14 kernel: sk0: link state changed to DOWN
                            Dec 9 14:08:14 kernel: sk0: watchdog timeout
                            Dec 9 13:59:43 kernel: arp: 172.17.17.254 is on sk0 but got reply from 00:0b:db:65:8a:91 on em0
                            Dec 9 12:58:09 kernel: arp: 172.17.17.241 is on sk0 but got reply from 00:11:43:11:92:8b on em0
                            Dec 9 12:38:17 last message repeated 4 times
                            Dec 9 12:37:53 kernel: arp: 172.17.17.252 is on sk0 but got reply from 00:13:20:00:cb:cf on em0
                            Dec 9 12:36:57 last message repeated 9 times
                            Dec 9 12:36:37 kernel: arp: 172.17.17.241 is on sk0 but got reply from 00:11:43:11:92:8b on em0
                            Dec 9 12:21:55 kernel: em0: watchdog timeout – resetting
                            Dec 9 12:21:33 last message repeated 9 times
                            Dec 9 12:19:44 kernel: arp: 172.17.17.241 is on sk0 but got reply from 00:11:43:11:92:8b on em0
                            Dec 9 12:02:09 last message repeated 4 times
                            Dec 9 12:02:01 kernel: arp: 172.17.17.241 is on sk0 but got reply from 00:11:43:11:92:8b on em0
                            Dec 9 11:46:48 last message repeated 5 times
                            Dec 9 11:46:23 kernel: arp: 172.17.17.252 is on sk0 but got reply from 00:13:20:00:cb:cf on em0
                            Dec 9 11:45:39 kernel: arp: 172.17.17.247 is on sk0 but got reply from 00:06:5b:f3:2a:4b on em0

                            Intel Pro/1000 MT and 3COM 3C2000-T NIC's

                            I have just updated my box with your snaphsot link and also went into BIOS and disabled all unnecessary hardware.  I hope this fixes it.

                            1 Reply Last reply Reply Quote 0
                            • U
                              unreal1024
                              last edited by

                              I am upgrade pfSense 1.0.1 to 1.0.1-SNAPSHOT-12-08-2006, new install 1.0.1-SNAPSHOT-12-08-2006, but have same problem.

                              …filtered...
                              Dec 10 17:48:43 kernel: xl1: watchdog timeout
                              Dec 10 17:19:51 kernel: xl0: watchdog timeout
                              Dec 10 17:01:23 kernel: xl1: watchdog timeout
                              Dec 10 16:36:45 kernel: xl1: watchdog timeout
                              Dec 10 15:13:29 kernel: xl1: watchdog timeout
                              Dec 10 12:28:23 kernel: xl1: watchdog timeout
                              Dec 10 12:18:24 kernel: xl1: watchdog timeout

                              1 Reply Last reply Reply Quote 0
                              • U
                                unreal1024
                                last edited by

                                If you start receiving these messages regularly, read below:

                                FreeBSD network device drivers (dc, xl, sk etc.) utilize watchdog timers to keep some statistics about the device driver and to tackle deadlock situations that may arise because of hardware issues. This timer is set to some specific value by the device driver, and decremented by one once a second. If the timer expires and the network adapter
                                did not finish its job, the watchdog routine for this adapter is run.

                                Time watchdog timer routine simply prints a diagnostics message that is visible through /var/log/messages and restarts the network adapter.

                                Briefly, you can try the following to get rid of the problem:

                                Many PCI network adapters require a PCI slot that supports Bus Mastering. Some old motherboards
                                have this feature on only on their first PCI slot (pci0). So, plug your network adapter to the first
                                pci slot on your mainboard and see if that helps.

                                1 Reply Last reply Reply Quote 0
                                • U
                                  unreal1024
                                  last edited by

                                  Hi, support this networl card D-Link DFE-580TX Traffic shaper?

                                  1 Reply Last reply Reply Quote 0
                                  • J
                                    jeroen234
                                    last edited by

                                    don't steal a topic
                                    post it in the trafic shaper section of the forum
                                    if you had searched there you had found this post that will tell you witch interfaces suport alt q
                                    http://forum.pfsense.org/index.php/topic,1686.msg9789.html#msg9789

                                    1 Reply Last reply Reply Quote 0
                                    • U
                                      unreal1024
                                      last edited by

                                      Hi, exists patch for FreeBSD v. 6.1 for watchdog timeouts (Bus mastering PCI)?

                                      1 Reply Last reply Reply Quote 0
                                      • U
                                        unreal1024
                                        last edited by

                                        Hi, i am change 3Com card 3C905X to Planet with Realtek chipset (ENW-9503A) and disable acpi in to pfSense v. 1.0.1 (hint.acpi.0.disabled=1) and works perfectly!!!!!

                                        1 Reply Last reply Reply Quote 0
                                        • First post
                                          Last post
                                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.