pfSense Sporadic unable to get to internet.
-
Hello,
My pfSense install has been running well for around the last month or so. However I notice that now and again I am unable to get any internet, something that looks to be a DNS issues as per what the browsers report.
All the services are green and good to go so far as the Status section, though I did try restarting the DNS and DHCP service just in case, to no avail. It is clear however that even pfSense has issues getting out to the internet when these outages happen as it is unable to collect update information. I have checked my main ISP router by hooking directly to it and all is well, but as soon as I am behind the pfSense, no go.
Only a full reboot of the pfSense system will remedy the problem and all is well again, until it isn't. I also tried a reboot with check filesystem, just to make sure none of the system files had become damaged. However it reports all is well.
Some notes
*pfSense is set to be in the DMZ of my ISP router
*Current pfSense version 2.6.0System Specs
System: HP Thin Client T730
CPU: AMD RX-427BB with AMD Radeon(tm) R7 Graphics
RAM: 16GB
SSD: WD Blue M.2 250GB
NIC: Intel I350 -T4Please see below image of the current running services
I also noticed upon reboot that I had the following message regarding Netmap. I think I used to get this before and this was normal, but I am not confident.
When the system was fouling, I took a copy of the first page of the System Log to see if that yields any fruit for the experts on here. Please see attached, it was long so did not want to clutter this post.
Thanks in advance for any help given. I would love to make this as reliable as possible.
-
@panzerscope
Hey there,
excuse my poor input for I am definitly no pfsense pro...
What struck me by looking at your pfsense log:- unbound as DNS resolver seems to stop
- at the same time you have DNS resolver AND (??) forwarder running...why both?
- might be a known unbound issue, when DHCP registration is active...unbound then stops and starts again everytime a new client tries to get its IP?
Just a guess...
-
@the-other said in pfSense Sporadic unable to get to internet.:
@panzerscope
Hey there,
excuse my poor input for I am definitly no pfsense pro...
What struck me by looking at your pfsense log:- unbound as DNS resolver seems to stop
- at the same time you have DNS resolver AND (??) forwarder running...why both?
- might be a known unbound issue, when DHCP registration is active...unbound then stops and starts again everytime a new client tries to get its IP?
Just a guess...
Thanks for the input! Well I have just checked at DNS Forwarding is not enabled, I only have the DNS Resolver enabled.
Fair point about the unbound DNS issues, though unsure what I would do to test/remedy the issue. I would assume that it could be related to pfblockerNG and its DNS based filtering via DNSBL. Though not sure.
-
@panzerscope
Hey,
is the option dhcp registration under Services > DNS resolver active? Might want to deactivate that and give it a try.Yeah, I also assume that it is a DNS related problem since your log shows an issue with resolving the needed address for sending e-mail...
I get the same error once in a while: unbound seems to stop, DNS not working, most of the time it starts again by itself, once in a while it needs a manual kick in the butt to start again...and so far I could not solve that...happens about twice a month.
-
Thanks,I have just checked and DNS registration is disabled. This is the DNS Resolver page currently.
-
I see a number of issues from the entries in the log snippet you posted.
First, you have the DNS Forwarder (
dnsmasq) being monitored by the Service Watchdog package. I see multiple times in your logs where Service Watchdog thinks the DNS Forwarder is "down" and restarts it. Because the Forwarder and Resolver both want to run on the same port, you have problems. That leads to DNS failures. The DNS Resolver on pfSense isunbound. The DNS Forwarder isdnsmasq.Second big issue is that you have Snort configured in the Service Watchdog package. I am the Snort package developer/maintainer, and I have posted here on the pfSense forums over and over that a user should NEVER configure Service Watchdog to monitor Snort. Service Watchdog does not understand how Snort works and will attempt to restart it when unnecessary. That can eventually lead to many duplicate Snort processes running on the same interface.
Finally, I see a number of netmap device errors related to running Snort in Inline IPS Mode. It might be due to duplicate processes created by the Service Watchdog package not understanding how Snort works, or it might be due to your particular NIC variant not being 100% netmap compliant.
-
@bmeeks said in pfSense Sporadic unable to get to internet.:
Snort configured in the Service Watchdog package
But he doesn't even show that as a running service, he has Suricata listed.. So trying to run both?, switched to it vs snort, but left short in the watchdog?
-
@johnpoz said in pfSense Sporadic unable to get to internet.:
@bmeeks said in pfSense Sporadic unable to get to internet.:
Snort configured in the Service Watchdog package
But he doesn't even show that as a running service, he has Suricata listed.. So trying to run both?, switched to it vs snort, but left short in the watchdog?
Yeah, I'm thinking Service Watchdog has some improper entries.
I just immediately looked into the posted log attachment and did not examine all the installed packages. But yes, you are correct. He has Suricata installed now, but Service Watchdog is trying to start Snort.
-
Thanks very much for your comments and help. So it is true that I switched from Snort to Suricata, thus must have ended up being a left over entry in the Service Watchdog.
I have removed Snort as well as the DNS Forwarder from the Service Watchdog. Currently my watchdog list looks like the following.
Does this list look appropriate, will retaining the unbound DNS Resolver in the list cause any issues ? Once I have this cleared up I will reboot the pfsense box and see if I get these issues again further down the line.
-
@panzerscope said in pfSense Sporadic unable to get to internet.:
Thanks very much for your comments and help. So it is true that I switched from Snort to Suricata, thus must have ended up being a left over entry in the Service Watchdog.
I have removed Snort as well as the DNS Forwarder from the Service Watchdog. Currently my watchdog list looks like the following.
Does this list look appropriate, will retaining the unbound DNS Resolver in the list cause any issues ? Once I have this cleared up I will reboot the pfsense box and see if I get these issues again further down the line.
Personally, there is really no need to run the Service Watchdog package. At best, if you have services randomly stopping, it is a band aid. You need to identify why the services are randomly stopping and fix that root cause. Service Watchdog is not a package I would consider installing.
unboundmakes a great resolver for pfSense, but it begins to get a bit strained when you use a package like pfBlockerNG-devel to create and maintain large DNS blacklists (via the DNSBL addon, for example). -
@bmeeks said in pfSense Sporadic unable to get to internet.:
@panzerscope said in pfSense Sporadic unable to get to internet.:
Thanks very much for your comments and help. So it is true that I switched from Snort to Suricata, thus must have ended up being a left over entry in the Service Watchdog.
I have removed Snort as well as the DNS Forwarder from the Service Watchdog. Currently my watchdog list looks like the following.
Does this list look appropriate, will retaining the unbound DNS Resolver in the list cause any issues ? Once I have this cleared up I will reboot the pfsense box and see if I get these issues again further down the line.
Personally, there is really no need to run the Service Watchdog package. At best, if you have services randomly stopping, it is a band aid. You need to identify why the services are randomly stopping and fix that root cause. Service Watchdog is not a package I would consider installing.
unboundmakes a great resolver for pfSense, but it begins to get a bit strained when you use a package like pfBlockerNG-devel to create and maintain large DNS blacklists (via the DNSBL addon, for example).That is a fair point to be honest, best to fix the problem over the service continuously restarting the service. With that in mind, I have gone ahead and removed Service Watchdog altogether. Stops it from being a factor after all.
-
I also meant to ask, specifically regarding this screenshot where it makes references to netmap. Is this normal ? I just cannot recall if I got this before. I know I did solve all my netmap issues when switching to the Intel I350 -T4 NIC. A discussion previously had here: https://forum.netgate.com/topic/171570/editing-loader-conf
Thanks again!
-
@panzerscope why would you have pcscd in watchdog - are you actually using it? Its got a memory leak, there are multiple threads about it. And it was set to not run on default I do believe a update or so back, etc.
I don't even have the service watchdog package installed..
-
@panzerscope said in pfSense Sporadic unable to get to internet.:
I also meant to ask, specifically regarding this screenshot where it makes references to netmap. Is this normal ? I just cannot recall if I got this before. I know I did solve all my netmap issues when switching to the Intel I350 -T4 NIC. A discussion previously had here: https://forum.netgate.com/topic/171570/editing-loader-conf
Thanks again!
Yes, those are purely informational startup messages logged by the netmap device. They show that your NIC is providing 4 TX and 4 RX netmap queues (or rings).
The messages I saw in your logs that indicated problems were the ones that said something along the lines of "... netmap_reinint ...". I don't recall that exact wording off the top of my head. But those messages in your previous system log showing netmap issues indicate that multiple threads were stepping on each other's netmap buffer (or ring) areas.
-
Yeah, you do not want pcscd running at all. It's disabled by default in 2.6.
https://redmine.pfsense.org/issues/11933And, yeah, you probably don't need/want the services watchdog running at all. You should have a specific reason for enabling that for any service.
Steve
-
Thanks all for your help. I just wanted to come back and things seem to now be resolved due to the above steps. Fingers crossed it stays that way. Hopefully some other newb will find this useful in the future.
