Firewall periodically loses internet connectivity (package related?)



  • Not sure if this is the proper place for this issue, but since I cannot narrow down the reason here it goes...

    pfSense:
    2.4.3-RELEASE-p1 (amd64)
    built on Thu May 10 15:02:52 CDT 2018
    FreeBSD 11.1-RELEASE-p10

    Router working fine for 2+ years and went through several package/OS upgrades. All of a sudden, since a few days ago, I get home after work and the internet is down. Happens each night. Nothing (browsers and cell phones) resolves (404). Only a reboot of pfSense fixes this, but it happens all over again the next day...

    Router still can check for latest OS version, and can ping from the router.

    The following packages are installed:

    Cron 0.3.7_2
    freeradius3 0.15.5_2
    ntopng 0.8.12
    pfBlockerNG 2.1.2_3
    RRD_Summary 2.0
    Service_Watchdog1.8.4
    snort 3.2.9.6_1
    Status_Traffic_Totals 1.2.4

    This is the general log from 7AM today to when I got back home. At 7AM the internet was working just fine so I assume something went wrong during the day.

    Jun 27 17:40:16 	php-fpm 	319 /index.php: Successful login for user 'admin' from: XXX.XXX.XXX.XXX
    Jun 27 17:40:00 	ntopng 		[Alert] [ENGAGED] Host <a href='/lua/host_details.lua?ifid=2&host=XXX.XXX.XXX.XXX'>XXX-XXX-XXX-XXX.isp.com</a> is a Flooder (26 flows sent in 3 sec)
    Jun 27 17:02:41 	php 		[pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
    Jun 27 17:00:01 	php 		[pfBlockerNG] Starting cron process.
    Jun 27 16:02:42 	php 		[pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
    Jun 27 16:00:01 	php 		[pfBlockerNG] Starting cron process.
    Jun 27 15:02:48 	php 		[pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
    Jun 27 15:00:00 	php 		[pfBlockerNG] Starting cron process.
    Jun 27 14:02:51 	php 		[pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
    Jun 27 14:00:00 	php 		[pfBlockerNG] Starting cron process.
    Jun 27 13:02:46 	php 		[pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
    Jun 27 13:00:01 	php 		[pfBlockerNG] Starting cron process.
    Jun 27 12:53:54 	root 		rc.update_bogons.sh is ending the update cycle.
    Jun 27 12:53:54 	root 		Bogons V6 file downloaded but not updating IPv6 bogons table because it is not in use.
    Jun 27 12:53:54 	root 		Bogons V4 file downloaded: no changes.
    Jun 27 12:53:53 	root 		rc.update_bogons.sh is beginning the update cycle.
    Jun 27 12:30:55 	php-cgi 	rc.update_urltables: /etc/rc.update_urltables: pfB_IP_pass_foreground does not need updating.
    Jun 27 12:30:55 	php-cgi 	rc.update_urltables: /etc/rc.update_urltables: pfB_IP_pass_background does not need updating.
    Jun 27 12:30:55 	php-cgi 	rc.update_urltables: /etc/rc.update_urltables: pfB_ASN_pass does not need updating.
    Jun 27 12:30:55 	php-cgi 	rc.update_urltables: /etc/rc.update_urltables: pfB_ASN_block does not need updating.
    Jun 27 12:30:55 	php-cgi 	rc.update_urltables: /etc/rc.update_urltables: pfB_IP_block does not need updating.
    Jun 27 12:30:55 	php-cgi 	rc.update_urltables: /etc/rc.update_urltables: pfB_malicious_malware does not need updating.
    Jun 27 12:30:55 	php-cgi 	rc.update_urltables: /etc/rc.update_urltables: pfB_ads_spam does not need updating.
    Jun 27 12:30:55 	php-cgi 	rc.update_urltables: /etc/rc.update_urltables: pfB_PS_v4 does not need updating.
    Jun 27 12:30:55 	php-cgi 	rc.update_urltables: /etc/rc.update_urltables: pfB_Top_v6 does not need updating.
    Jun 27 12:30:55 	php-cgi 	rc.update_urltables: /etc/rc.update_urltables: pfB_Top_v4 does not need updating.
    Jun 27 12:30:55 	php-cgi 	rc.update_urltables: /etc/rc.update_urltables: pfB_SAmerica_v6 does not need updating.
    Jun 27 12:30:55 	php-cgi 	rc.update_urltables: /etc/rc.update_urltables: pfB_SAmerica_v4 does not need updating.
    Jun 27 12:30:55 	php-cgi 	rc.update_urltables: /etc/rc.update_urltables: pfB_Oceania_v6 does not need updating.
    Jun 27 12:30:55 	php-cgi 	rc.update_urltables: /etc/rc.update_urltables: pfB_Oceania_v4 does not need updating.
    Jun 27 12:30:55 	php-cgi 	rc.update_urltables: /etc/rc.update_urltables: pfB_NAmerica_v6 does not need updating.
    Jun 27 12:30:55 	php-cgi 	rc.update_urltables: /etc/rc.update_urltables: pfB_NAmerica_v4 does not need updating.
    Jun 27 12:30:55 	php-cgi 	rc.update_urltables: /etc/rc.update_urltables: pfB_Europe_v6 does not need updating.
    Jun 27 12:30:55 	php-cgi 	rc.update_urltables: /etc/rc.update_urltables: pfB_Europe_v4 does not need updating.
    Jun 27 12:30:55 	php-cgi 	rc.update_urltables: /etc/rc.update_urltables: pfB_Asia_v6 does not need updating.
    Jun 27 12:30:55 	php-cgi 	rc.update_urltables: /etc/rc.update_urltables: pfB_Asia_v4 does not need updating.
    Jun 27 12:30:55 	php-cgi 	rc.update_urltables: /etc/rc.update_urltables: pfB_Africa_v6 does not need updating.
    Jun 27 12:30:55 	php-cgi 	rc.update_urltables: /etc/rc.update_urltables: pfB_Africa_v4 does not need updating.
    Jun 27 12:30:55 	php-cgi 	rc.update_urltables: /etc/rc.update_urltables: pfB_DNSBLIP does not need updating.
    Jun 27 12:30:55 	php-cgi 	rc.update_urltables: /etc/rc.update_urltables: Starting URL table alias updates
    Jun 27 12:30:00 	php-cgi 	rc.update_urltables: /etc/rc.update_urltables: Sleeping for 55 seconds.
    Jun 27 12:30:00 	php-cgi 	rc.update_urltables: /etc/rc.update_urltables: Starting up.
    Jun 27 12:02:49 	php 		[pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
    Jun 27 12:00:11 	php 		[pfBlockerNG] Starting cron process.
    Jun 27 11:02:49 	php 		[pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
    Jun 27 11:00:10 	php 		[pfBlockerNG] Starting cron process.
    Jun 27 10:02:45 	php 		[pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
    Jun 27 10:00:00 	php 		[pfBlockerNG] Starting cron process.
    Jun 27 09:02:55 	php 		[pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
    Jun 27 09:00:13 	php 		[pfBlockerNG] Starting cron process.
    Jun 27 08:02:47 	php 		[pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
    Jun 27 08:00:01 	php 		[pfBlockerNG] Starting cron process.
    Jun 27 07:02:49 	php 		[pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
    Jun 27 07:00:00 	php 		[pfBlockerNG] Starting cron process. 
    

    I see nothing abnormal but I am not an expert (hence why I am posting here). However one line caught my eye:

    [Alert] [ENGAGED] Host <a href='/lua/host_details.lua?ifid=2&host=XXX.XXX.XXX.XXX'>XXX-XXX-XXX-XXX.isp.com</a> is a Flooder (26 flows sent in 3 sec)
    

    Related?



  • @pftdm007 Sounds like a memory leak thing. Maybe wanna check out, there was a thing about increasing some table size blah-blah.

    Storage, ram, cpu utilizations not maxing out?



  • Thanks for replying. AFAIK RAM is never maxed out. Runs around 30 to 35% of the installed 12GB. Same for CPU, I rarely see anything above 5 to 10% utilization.

    States table has 812000 entries, to which only 2000 or so are used at any given time.

    I left my house friday night for the weekend and came back yesterday. When I left, I had disabled fpblockerNG and DNSBL. When I came back home, everything was working fine. I activated pfblockerNG and DNSBL, went to bed, the internet was still working. This morning, the internet was down again.

    I am 99% convinced pfblockerNG or DNSBL are to blame. I tried resetting the states table, do a force reload on pfblocker and DNSBL (although they were deactivated I thought that could maybe help) and had to reboot the firewall once again.

    Next step is to find the reason for this. Anybody has an idea on how to troubleshoot this?

    To me it looks like a DNS resolution issue more than actual connectivity to the web since pfsense still can access the outside world but nothing from the LAN can reach out... I may be wrong. I also thought Unbound was crashing but it appears to be always run fine...