Watchdog timeout -- resetting
-
I have PFSense Box build on HP ProLiant MicroServer Gen10 / AMD Opteron X3418 1.8GHz With INTEL Gigabit ET Dual Port (Intel
82 576).Onboard NICs are used for LAN and Intel card is used for WAN.
I'm running OpenVPN & Snort on the box.
My problem is that every (aprox.) week one of the two port on Intel NICs suddenly goes down and I cannot get it back up until I restart the whole machine.System Log:
Feb 3 13:57:04 dhcpleases bad name in /var/dhcpd/var/db/dhcpd.leases Feb 3 13:57:04 check_reload_status Reloading filter Feb 3 13:57:04 dhcpleases /etc/hosts changed size from original! Feb 3 13:57:04 php-fpm 10544 /rc.newwanip: rc.newwanip: on (IP address: <REDACTED>) (interface: WAN1[wan]) (real interface: igb0). Feb 3 13:57:04 php-fpm 10544 /rc.newwanip: rc.newwanip: Info: starting on igb0. Feb 3 13:57:03 check_reload_status Reloading filter Feb 3 13:57:03 check_reload_status rc.newwanip starting igb0 Feb 3 13:57:03 php-fpm 10544 /rc.linkup: Hotplug event detected for WAN1(wan) static IP (<REDACTED> ) Feb 3 13:57:02 kernel igb0: link state changed to UP Feb 3 13:57:02 check_reload_status Linkup starting igb0 Feb 3 13:57:02 check_reload_status Reloading filter Feb 3 13:57:02 php-fpm /rc.linkup: Hotplug event detected for WAN1(wan) static IP (<REDACTED> ) Feb 3 13:57:01 check_reload_status Linkup starting igb0 Feb 3 13:57:01 kernel igb0: link state changed to DOWN Feb 3 13:57:01 kernel igb0: TX(0) desc avail = 0,Next TX to Clean = 0 Feb 3 13:57:01 kernel igb0: Queue(0) tdh = 0, hw tdt = 984 Feb 3 13:57:01 kernel igb0: Watchdog timeout -- resettingI have two WAN connections and the other one is totally fine and newer went down. I tried to change cables used copper and used optics with transceivers but link will go down every time after a while.
I have all of the HW offloading disabled except Checksum but even with it disabled it was the same. I also tried to set
hw.igb.num_queues=1But again, nothing changed as far as I can tell.
Does someone have any idea what could be the culprit or is there maybe some incompatibility that i cannot find any mention about?
I did check with my ISP, he did some testing on his equipment but he couldn't find anything.
-
You might try booting verbose to see if you get any additional logs. You can also check the macstats in the sysctls for igb.
Generally though igb NICs are pretty good, if you see that it's probably something exceptional.If you're lucky that might be some odd traffic rather than a hardware issue.
Steve
-
Thank you I will check. One think that I know for sure is that both NICs - One on my box and one on IPS router sending data but neither of them receives anything...
-
Hmm, well that seems like something low level. Is there a switch in between? Just a bad cable?
Steve
-
@stephenw10 There is no switch, it is direct connection between pfsense box and I believe Mikrotik SXT. I tested both copper connection and optical (cat5 -> converter -> optic -> converter -> cat5) but I had same problem with both so I do not believe that it is problem with medium.
-
One additional note is that until I set static/permanent mac address in ARP for my gateway I got these problems (lost link) almost every day, after that It will only occurs once per week...
-
I have some new findings originally problem did not occur periodically, it was seemingly random but now 21 days in a row (three times in that time) connection dropped on Monday within 3 hours window. Se either there is something external going on, which my ISP cannot find or there something in the system that I cannot yet find or it may be just coincidence...
-
Check the crontab, install the Cron package for easy access to it. Anything scheduled for Mondays like that should appear there.
Steve
-
@stephenw10 Yep I already checked cron, sadly there doesn't seem to be any related items...
minute hour mday month wday who command 1,31 0-5 * * * root /usr/bin/nice -n20 adjkerntz -a 1 3 1 * * root /usr/bin/nice -n20 /etc/rc.update_bogons.sh 1 1 * * * root /usr/bin/nice -n20 /etc/rc.dyndns.update */60 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot 30 12 * * * root /usr/bin/nice -n20 /etc/rc.update_urltables 1 0 * * * root /usr/bin/nice -n20 /etc/rc.update_pkg_metadata 0 * * * * root /usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php cron >> /var/log/pfblockerng/pfblockerng.log 2>&1 0 12 4-10 * * root /usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php dcc >> /var/log/pfblockerng/extras.log 2>&1 */5 * * * * root /usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/pkg/snort/snort_check_cron_misc.inc 15 3,15 * * * root /usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/pkg/snort/snort_check_for_rule_updates.php */2 * * * * root /usr/bin/nice -n20 /sbin/pfctl -q -t snort2c -T expire 900 -
During latest connection drop I also grabed netstat -m just because...:
12457/4508/16965 mbufs in use (current/cache/total) 10428/2988/13416/1000000 mbuf clusters in use (current/cache/total/max) 10428/2981 mbuf+clusters out of packet secondary zone in use (current/cache) 1/174/175/524288 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/524288 9k jumbo clusters in use (current/cache/total/max) 0/0/0/39393 16k jumbo clusters in use (current/cache/total/max) 23979K/7799K/31778K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 sendfile syscalls 0 sendfile syscalls completed without I/O request 0 requests for I/O initiated by sendfile 0 pages read by sendfile as part of a request 0 pages were valid at time of a sendfile request 0 pages were requested for read ahead by applications 0 pages were read ahead by sendfile 0 times sendfile encountered an already busy page 0 requests for sfbufs denied 0 requests for sfbufs delayedI also tried to force-restart interface without restarting the whole machine. Using service netif restart igb0 did nothing. Is there a way to do hard reset of the whole card from within OS so I can quickly fix the problem until I find temporary solution?
-
Today I also found this: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=239240 It might be relevant to my problem.
-
That appears to be a FreeBSD 12 issue, are you running a pfSense 2.5 snapshot? If not it's probably unrelated.
Steve
-
@stephenw10 Oh, I have got the impression that it is also related to older versions. Thank you.
-
This post is deleted!
