4. Watchdog timeout -- resetting

Watchdog timeout -- resetting

  • R

    I have PFSense Box build on HP ProLiant MicroServer Gen10 / AMD Opteron X3418 1.8GHz With INTEL Gigabit ET Dual Port (Intel® 82 576).

    Onboard NICs are used for LAN and Intel card is used for WAN.

    I'm running OpenVPN & Snort on the box.
    My problem is that every (aprox.) week one of the two port on Intel NICs suddenly goes down and I cannot get it back up until I restart the whole machine.

    System Log:

    Feb 3 13:57:04	dhcpleases		bad name in /var/dhcpd/var/db/dhcpd.leases
Feb 3 13:57:04	check_reload_status		Reloading filter
Feb 3 13:57:04	dhcpleases		/etc/hosts changed size from original!
Feb 3 13:57:04	php-fpm	10544	/rc.newwanip: rc.newwanip: on (IP address: <REDACTED>) (interface: WAN1[wan]) (real interface: igb0).
Feb 3 13:57:04	php-fpm	10544	/rc.newwanip: rc.newwanip: Info: starting on igb0.
Feb 3 13:57:03	check_reload_status		Reloading filter
Feb 3 13:57:03	check_reload_status		rc.newwanip starting igb0
Feb 3 13:57:03	php-fpm	10544	/rc.linkup: Hotplug event detected for WAN1(wan) static IP (<REDACTED> )
Feb 3 13:57:02	kernel		igb0: link state changed to UP
Feb 3 13:57:02	check_reload_status		Linkup starting igb0
Feb 3 13:57:02	check_reload_status		Reloading filter
Feb 3 13:57:02	php-fpm		/rc.linkup: Hotplug event detected for WAN1(wan) static IP (<REDACTED> )
Feb 3 13:57:01	check_reload_status		Linkup starting igb0
Feb 3 13:57:01	kernel		igb0: link state changed to DOWN
Feb 3 13:57:01	kernel		igb0: TX(0) desc avail = 0,Next TX to Clean = 0
Feb 3 13:57:01	kernel		igb0: Queue(0) tdh = 0, hw tdt = 984
Feb 3 13:57:01	kernel		igb0: Watchdog timeout -- resetting

    I have two WAN connections and the other one is totally fine and newer went down. I tried to change cables used copper and used optics with transceivers but link will go down every time after a while.

    I have all of the HW offloading disabled except Checksum but even with it disabled it was the same. I also tried to set

    hw.igb.num_queues=1

    But again, nothing changed as far as I can tell.

    Does someone have any idea what could be the culprit or is there maybe some incompatibility that i cannot find any mention about?

    I did check with my ISP, he did some testing on his equipment but he couldn't find anything.

    0
  • Netgate Administrator

    You might try booting verbose to see if you get any additional logs. You can also check the macstats in the sysctls for igb.

    Generally though igb NICs are pretty good, if you see that it's probably something exceptional.If you're lucky that might be some odd traffic rather than a hardware issue.

    Steve

    0
  • R

    Thank you I will check. One think that I know for sure is that both NICs - One on my box and one on IPS router sending data but neither of them receives anything...

    0
  • Netgate Administrator

    Hmm, well that seems like something low level. Is there a switch in between? Just a bad cable?

    Steve

    0
    R
     1 Reply
  • R

    @stephenw10 There is no switch, it is direct connection between pfsense box and I believe Mikrotik SXT. I tested both copper connection and optical (cat5 -> converter -> optic -> converter -> cat5) but I had same problem with both so I do not believe that it is problem with medium.

    0
  • R

    One additional note is that until I set static/permanent mac address in ARP for my gateway I got these problems (lost link) almost every day, after that It will only occurs once per week...

    0
  • R

    I have some new findings originally problem did not occur periodically, it was seemingly random but now 21 days in a row (three times in that time) connection dropped on Monday within 3 hours window. Se either there is something external going on, which my ISP cannot find or there something in the system that I cannot yet find or it may be just coincidence...

    0
  • Netgate Administrator

    Check the crontab, install the Cron package for easy access to it. Anything scheduled for Mondays like that should appear there.

    Steve

    0
    R
     1 Reply
  • R

    @stephenw10 Yep I already checked cron, sadly there doesn't seem to be any related items...

    minute	hour	mday	month	wday	who	command	
1,31	0-5	*	*	*	root	/usr/bin/nice -n20 adjkerntz -a	 
1	3	1	*	*	root	/usr/bin/nice -n20 /etc/rc.update_bogons.sh	 
1	1	*	*	*	root	/usr/bin/nice -n20 /etc/rc.dyndns.update	 
*/60	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot	 
30	12	*	*	*	root	/usr/bin/nice -n20 /etc/rc.update_urltables	 
1	0	*	*	*	root	/usr/bin/nice -n20 /etc/rc.update_pkg_metadata	 
0	*	*	*	*	root	/usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php cron >> /var/log/pfblockerng/pfblockerng.log 2>&1	 
0	12	4-10	*	*	root	/usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php dcc >> /var/log/pfblockerng/extras.log 2>&1	 
*/5	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/pkg/snort/snort_check_cron_misc.inc	 
15	3,15	*	*	*	root	/usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/pkg/snort/snort_check_for_rule_updates.php	 
*/2	*	*	*	*	root	/usr/bin/nice -n20 /sbin/pfctl -q -t snort2c -T expire 900
    0
  • R

    During latest connection drop I also grabed netstat -m just because...:

    12457/4508/16965 mbufs in use (current/cache/total)
10428/2988/13416/1000000 mbuf clusters in use (current/cache/total/max)
10428/2981 mbuf+clusters out of packet secondary zone in use (current/cache)
1/174/175/524288 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/524288 9k jumbo clusters in use (current/cache/total/max)
0/0/0/39393 16k jumbo clusters in use (current/cache/total/max)
23979K/7799K/31778K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 sendfile syscalls
0 sendfile syscalls completed without I/O request
0 requests for I/O initiated by sendfile
0 pages read by sendfile as part of a request
0 pages were valid at time of a sendfile request
0 pages were requested for read ahead by applications
0 pages were read ahead by sendfile
0 times sendfile encountered an already busy page
0 requests for sfbufs denied
0 requests for sfbufs delayed

    I also tried to force-restart interface without restarting the whole machine. Using service netif restart igb0 did nothing. Is there a way to do hard reset of the whole card from within OS so I can quickly fix the problem until I find temporary solution?

    0
  • R

    Today I also found this: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=239240 It might be relevant to my problem.

    0
  • Netgate Administrator

    That appears to be a FreeBSD 12 issue, are you running a pfSense 2.5 snapshot? If not it's probably unrelated.

    Steve

    0
    R
     1 Reply
  • R

    @stephenw10 Oh, I have got the impression that it is also related to older versions. Thank you.

    0
  • R

    This post is deleted!
    0
Log in to reply
 

Products

Services

Support

News

Resources

Company

Our Mission

We provide leading-edge network security at a fair price - regardless of organizational size or network sophistication. We believe that an open-source security model offers disruptive pricing along with the agility required to quickly address emerging threats.

Subscribe to our Newsletter

Product information, software announcements, and special offers. See our newsletter archive for past announcements.

© Copyright 2019 Rubicon Communications, LLC | Privacy Policy