Watchdog timeout -- resetting

stephenw10

You might try booting verbose to see if you get any additional logs. You can also check the macstats in the sysctls for igb.

Generally though igb NICs are pretty good, if you see that it's probably something exceptional.If you're lucky that might be some odd traffic rather than a hardware issue.

Steve

r43K9o

Thank you I will check. One think that I know for sure is that both NICs - One on my box and one on IPS router sending data but neither of them receives anything...

stephenw10

Hmm, well that seems like something low level. Is there a switch in between? Just a bad cable?

Steve

r43K9o

@stephenw10 There is no switch, it is direct connection between pfsense box and I believe Mikrotik SXT. I tested both copper connection and optical (cat5 -> converter -> optic -> converter -> cat5) but I had same problem with both so I do not believe that it is problem with medium.

r43K9o

One additional note is that until I set static/permanent mac address in ARP for my gateway I got these problems (lost link) almost every day, after that It will only occurs once per week...

r43K9o

I have some new findings originally problem did not occur periodically, it was seemingly random but now 21 days in a row (three times in that time) connection dropped on Monday within 3 hours window. Se either there is something external going on, which my ISP cannot find or there something in the system that I cannot yet find or it may be just coincidence...

stephenw10

Check the crontab, install the Cron package for easy access to it. Anything scheduled for Mondays like that should appear there.

Steve

r43K9o

@stephenw10 Yep I already checked cron, sadly there doesn't seem to be any related items...

minute	hour	mday	month	wday	who	command	
1,31	0-5	*	*	*	root	/usr/bin/nice -n20 adjkerntz -a	 
1	3	1	*	*	root	/usr/bin/nice -n20 /etc/rc.update_bogons.sh	 
1	1	*	*	*	root	/usr/bin/nice -n20 /etc/rc.dyndns.update	 
*/60	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot	 
30	12	*	*	*	root	/usr/bin/nice -n20 /etc/rc.update_urltables	 
1	0	*	*	*	root	/usr/bin/nice -n20 /etc/rc.update_pkg_metadata	 
0	*	*	*	*	root	/usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php cron >> /var/log/pfblockerng/pfblockerng.log 2>&1	 
0	12	4-10	*	*	root	/usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php dcc >> /var/log/pfblockerng/extras.log 2>&1	 
*/5	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/pkg/snort/snort_check_cron_misc.inc	 
15	3,15	*	*	*	root	/usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/pkg/snort/snort_check_for_rule_updates.php	 
*/2	*	*	*	*	root	/usr/bin/nice -n20 /sbin/pfctl -q -t snort2c -T expire 900

r43K9o

During latest connection drop I also grabed netstat -m just because...:

12457/4508/16965 mbufs in use (current/cache/total)
10428/2988/13416/1000000 mbuf clusters in use (current/cache/total/max)
10428/2981 mbuf+clusters out of packet secondary zone in use (current/cache)
1/174/175/524288 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/524288 9k jumbo clusters in use (current/cache/total/max)
0/0/0/39393 16k jumbo clusters in use (current/cache/total/max)
23979K/7799K/31778K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 sendfile syscalls
0 sendfile syscalls completed without I/O request
0 requests for I/O initiated by sendfile
0 pages read by sendfile as part of a request
0 pages were valid at time of a sendfile request
0 pages were requested for read ahead by applications
0 pages were read ahead by sendfile
0 times sendfile encountered an already busy page
0 requests for sfbufs denied
0 requests for sfbufs delayed

I also tried to force-restart interface without restarting the whole machine. Using service netif restart igb0 did nothing. Is there a way to do hard reset of the whole card from within OS so I can quickly fix the problem until I find temporary solution?

r43K9o

Today I also found this: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=239240 It might be relevant to my problem.

stephenw10

That appears to be a FreeBSD 12 issue, are you running a pfSense 2.5 snapshot? If not it's probably unrelated.

Steve

r43K9o

@stephenw10 Oh, I have got the impression that it is also related to older versions. Thank you.

r43K9o

This post is deleted!