Intermittent connection issue

kevindd992002

@johnpoz said in Intermittent connection issue:

Your issue is you had 21% packet loss that triggered a gateway down event.. So no shit unbound would not be able to resolve during that period..

Yes, I agree 100%. Like I said, this could just be a coincidence with the power-up event of my desktop PC.

Do you have any comments on my observation when using unbound w/ forwarding vs. without? I have not experienced a single occurrence of the issue (except when the gateway went down) when I enabled forwarding. So that tells us that unbound w/o forwarding is the issue here but I can't point out why because I have another pfsense box on the same ISP that uses unbound w/o forwarding flawlessly.

johnpoz

As I have already gone over - if your line is having packet loss, then yes you can have an issue with resolving something more than say a forward. You have to talk to multiple servers all over the internet to resolve.. With a forward your just asking that 1 guy for what the answer is..

Upping your logging level in unbound (in the advanced section of unbound), also logging queries and answers will give you some insight to what might be the problem with resolving specific sites.

In the options box

server:
log-queries: yes
log-replies: yes

Look at your cache for your unbound for any problem sites that are not resolving.. If unbound is restarting you loose your cache.. Just because you haven't seen packet loss issues in the past, doesn't mean your not having them.. Your path to 8.8.8.8 is not the whole internet... It's an anycast address.. There are MULTIPLE paths to get to that address.

If your having issues with unbound resolving something - you have to troubleshoot the resolving issue.. Which is why setup your log to log more info, and log the queries.. and the answers..

On a problematic connection forwarding can be less likely to see problems than resolving. Especially if your restarting unbound and loosing your local cache. Especially if you have issues to talking to specific NS, which unbound keeps track of and doesn't try to use via its infra info.. But when the cache is lost on a restart, all of that info is lost as well..

kevindd992002

@johnpoz said in Intermittent connection issue:

As I have already gone over - if your line is having packet loss, then yes you can have an issue with resolving something more than say a forward. You have to talk to multiple servers all over the internet to resolve.. With a forward your just asking that 1 guy for what the answer is..

Upping your logging level in unbound (in the advanced section of unbound), also logging queries and answers will give you some insight to what might be the problem with resolving specific sites.

In the options box
server:
log-queries: yes
log-replies: yes
Look at your cache for your unbound for any problem sites that are not resolving.. If unbound is restarting you loose your cache.. Just because you haven't seen packet loss issues in the past, doesn't mean your not having them.. Your path to 8.8.8.8 is not the whole internet... It's an anycast address.. There are MULTIPLE paths to get to that address.

If your having issues with unbound resolving something - you have to troubleshoot the resolving issue.. Which is why setup your log to log more info, and log the queries.. and the answers..

On a problematic connection forwarding can be less likely to see problems than resolving. Especially if your restarting unbound and loosing your local cache. Especially if you have issues to talking to specific NS, which unbound keeps track of and doesn't try to use via its infra info.. But when the cache is lost on a restart, all of that info is lost as well..

Like I said though, it's not even really the resolving part that's the issue. When I'm using unbound and the issue is present, I don't receive responses when I ping the IP address of www.google.com (so no resolution involved here, but unbound is the one causing it). When I switch to a forwarder, I don't encounter this issue.

So the problem is when using unbound, somehow pfsense cannot reach the multiple NS that it's trying to query from. But yeah, upping the logging level wouldn't hurt for me to try.

Is there such thing where an ISP doesn't work well with unbound setup on their customer's premises? Maybe bad routes to some NS servers or something? I'm just trying to think out of the box here.

Also, is it recommended to disable all these options so that unbound will not restart? If so, how can I resolve my local clients by their FQDN?

johnpoz

@kevindd992002 said in Intermittent connection issue:

(so no resolution involved here, but unbound is the one causing it)

unbound has ZERO!!! Let me repeat that ZERO!!! to do with you pinging some IP.. 8.8.8.8 is not resolved, so you resolver has ZERO to do with it... If you can not ping 8.8.8.8 then you have a connectivity issue and ZERO!!! Again ZERO to do with any forwarder or resolver you would be running..

Yes I would recommend you turn off registering dhcp or vpn in unbound - that causes a restart of it. Static is fine.

kevindd992002

@johnpoz said in Intermittent connection issue:

@kevindd992002 said in Intermittent connection issue:

(so no resolution involved here, but unbound is the one causing it)

unbound has ZERO!!! Let me repeat that ZERO!!! to do with you pinging some IP.. 8.8.8.8 is not resolved, so you resolver has ZERO to do with it... If you can not ping 8.8.8.8 then you have a connectivity issue and ZERO!!! Again ZERO to do with any forwarder or resolver you would be running..

Yes I would recommend you turn off registering dhcp or vpn in unbound - that causes a restart of it. Static is fine.

Again, I know that pinging an IP address DOES NOT involve DNS resolution, I'm not a beginner here. I don't know how else to say what I'm observing but like I said when I switch to unbound the issue randomly shows itself but when I use forwarding I do not experience the issue. So as I see it, unbound is bugging the whole pfsense box that it's acting up in intermittently in reaching external servers.

If I turn off dhcp registration, how do I resolve my internal clients?

kevindd992002

So I switched to unbound again while using my laptop and NOTHING ELSE. It was working for maybe around 30 mins until I experienced the issue again. Here's what I see in my cache:

alt text

When it was working, I had 0 Timeout A's for all those servers. And then it just started happening.

As for the logging level for DNS Resolver, what level should I put it? Increase it to level 5 right away? I've noticed that when I do that, I cannot see all of the logs under System Logs even though I increase the log filter quantity to an insane amount. That just means that there's too much data.

johnpoz

See those timeouts - that means your having issues talking to those.. That is going to cause you problems with resolution!! Those should pretty much be all ZEROS..

Looking at mine with 417 I have 1 entry with 2, and 1 other with 1

157.55.133.11 	o365filtering.com.    449 	0 	94 	376 	752 	2 	2 	0
156.154.64.10 	amazonaws.com. 	      441 	0 	94 	376 	752 	1 	0 	0

As to too much data - the data is there it has just rolled over.. Send it to a syslog if you want to be able to parse it easier.. But yeah having that many timeouts is going to be problematic..

kevindd992002

@johnpoz said in Intermittent connection issue:

See those timeouts - that means your having issues talking to those.. That is going to cause you problems with resolution!! Those should pretty much be all ZEROS..

Looking at mine with 417 I have 1 entry with 2, and 1 other with 1
157.55.133.11 	o365filtering.com.    449 	0 	94 	376 	752 	2 	2 	0
156.154.64.10 	amazonaws.com. 	      441 	0 	94 	376 	752 	1 	0 	0 
As to too much data - the data is there it has just rolled over.. Send it to a syslog if you want to be able to parse it easier.. But yeah having that many timeouts is going to be problematic..

Exactly! So it looks like my ISP's network is dropping packets when I'm using unbound but everything works properly when I'm forwarding to their DNS servers. Does that make sense? Or is there something wrong with the unbound service on my pfsense box?

johnpoz

Yeah its quite possible for your ISP to have issues with sending 53 or or just plain bad peering, etc.

How do you know your not having to retrans to 8.8.8.8 for your dns to work, and again 8.8.8.8 is anycast.. So you could be getting answers from any of them on that anycast network.. So it could be less noticeable... Or their dns..

If your having issues with unbound - use dnsmasq and forward...

Not sure what pfsense is suppose to do about your shitty isp?

kevindd992002

@johnpoz said in Intermittent connection issue:

Yeah its quite possible for your ISP to have issues with sending 53 or or just plain bad peering, etc.

How do you know your not having to retrans to 8.8.8.8 for your dns to work, and again 8.8.8.8 is anycast.. So you could be getting answers from any of them on that anycast network.. So it could be less noticeable... Or their dns..

If your having issues with unbound - use dnsmasq and forward...

Not sure what pfsense is suppose to do about your shitty isp?

Problem is, how do I tell this to them? Or is this even a valid concern? All they will be telling me is that I need to use their own DNS servers.

What do you mean? I'm not using 8.8.8.8 for DNS to work. 8.8.8.8 is simply a monitor IP in my pfsense box, for my modem's gateway. How is that significant here? I'm curious.

What is the difference between using unbound with forwarding and dnsmasq with forwarding?

kevindd992002

Ok, so I decided to disable DHCP registration and OpenVPN client registration from unbound since 4 days ago and everything seemed to be working perfectly now! I tried turning on my desktop and no issues at all! Weird thing is that when I had this issue I did not get a lot of unbound restarts as shown in the logs above so I still don't know the cause. There could be a device (when DHCP registration was enabled) in my network that constantly renewing its lease, I don't know, but I should've seen the restarts in the unbound logs. As for the Openvpn client registration, I think that's only for server-to-client OpenVPN connections which I do not use because my OpenVPN connection is server-to-server (site-to-site).

But yeah, so far so good. Now I'm thinking of implementing a pi-hole DNS server (using the container in docker hub) in my environment. What advantages will I get with using it? Is it just the beautiful interface/graphs that it offers? From what I've read so far, the pi-hole will be the DNS server for the clients (assigned via DHCP) and then it just forwards the requests to pfsense's unbound, right?

johnpoz

The different with dnsmasq and unbound for forwarding is dnsmasq out of the box forwards to ALL of the forwarders you have set at the same time, and uses just uses the first answer. I don't think you can tell unbound to do that.. Would have to check the unbound docs - but there is no way to set that in the gui of pfsense. This can be good for when you have bad peering isp or issues talking via some paths, etc.

Well restarting unbound never helps because it flushes your cache. But you do have a lot of timeouts you showed - so something not great with your isp either which could be problems.

For all we know your isp connections has gotten better, and has zero to do with unbound restarting.

Again when you have issues with dns - you can not just assume the problem is X, you need to troubleshoot the exact issue your seeing... Not just dns not working sometimes.. Pick something that didn't work and find out why.. Are you still seeing a lot of timeouts in your infra info?

You can also make sure you setup prefetch with unbound, this can help with problematic issues because it will look up stuff in the background before the ttl expire and it flushes out of the cache. Also setting serve 0 ttl can really help as well. Since now even if the cache expired, it would serve up the last entry when client asks for it - and then look it up in the background again.

kevindd992002

@johnpoz said in Intermittent connection issue:

The different with dnsmasq and unbound for forwarding is dnsmasq out of the box forwards to ALL of the forwarders you have set at the same time, and uses just uses the first answer. I don't think you can tell unbound to do that.. Would have to check the unbound docs - but there is no way to set that in the gui of pfsense. This can be good for when you have bad peering isp or issues talking via some paths, etc.

Well restarting unbound never helps because it flushes your cache. But you do have a lot of timeouts you showed - so something not great with your isp either which could be problems.

For all we know your isp connections has gotten better, and has zero to do with unbound restarting.

Again when you have issues with dns - you can not just assume the problem is X, you need to troubleshoot the exact issue your seeing... Not just dns not working sometimes.. Pick something that didn't work and find out why.. Are you still seeing a lot of timeouts in your infra info?

You can also make sure you setup prefetch with unbound, this can help with problematic issues because it will look up stuff in the background before the ttl expire and it flushes out of the cache. Also setting serve 0 ttl can really help as well. Since now even if the cache expired, it would serve up the last entry when client asks for it - and then look it up in the background again.

I see. I thought both dnsmasq and unbound forwarding works that way.

For some reason, I have 0 timeouts now. The one I showed you with a lot of timeouts was when I had DHCP and OpenVPN client registrations in unbound enabled.

Yeah, that's possible. It could be that the ISP connection has gotten better and is another coincidence. I really don't know but I was trying to pinpoint the issue to one specific module on my whole infra but I still can't isolate it.

Ok, so I'll enable these three, I guess:

How does the Serve Expired setting help though? So if the record has a TTL of already, it will still serve it to the client and update the cache in the background. What if the record is really no longer valid, how would serving an invalid record to the client help? I'm trying to understand how that setting works.

kevindd992002

Nevermind! I enabled those three options for both my infras and this problematic infra started getting DNS timeouts again! Tried to ping 8.8.8.8 from Diagnostics -> Ping using the WAN interface and 100% packet loss. When I think that one isolation step solves the problem, it bites me in the back after a few days of trying it. This is insane.

johnpoz

And lets go over this again - restarting unbound to change a feature has ZERO to do you with pinging anything by IP.. But if your having packetloss then resolving anything or even forwarding is going to be problematic at best.

kevindd992002

@johnpoz said in Intermittent connection issue:

And lets go over this again - restarting unbound to change a feature has ZERO to do you with pinging anything by IP.. But if your having packetloss then resolving anything or even forwarding is going to be problematic at best.

Yes, I completely agree. I just stated what I did and observed, I didn't say that the unbound restart directly caused me having packet losses.

I still don't get it though. I don't know what to tell these ISP people because all along I still believe that the issue is with their network. I just can't prove it.

bjohe

@kevindd992002, this is a very interesting thread. I've very similar experience with my pfsense. Some of my testing could give input to further investigations.
The problem is almost identical to what you have described. Occationally I cannot access webpages. This typically happens first time starting to browse Internet. It is any type of website, it could be a frequently accessed site or an one time visit. But It also occurs in the middle of when I'm browsing e.g. after a break.
But if I open up several tabs and start to access different websites, and constantly refresh and try to access them I finally start to connect to one or several of the websites. However there could still be one of them that is not loading.

For e.g. when watching Netflix I could have and issue at the beginning when I start loading the first movie or even connecting to Netflix. However once the movie has started I cannot recall I have ever experienced a loading or connection issues. The same is true for Citrix when logging on to a remote desktop. I could have issues to connect to begin with but once connected I cannot recall any loading/connection issues.

I replaced my old Netgear SRX5308 with pfSense on a box from Protecli (amd64) Intel Celeron CPU J3160 @ 1.60GHz 4 CPUs: 1 package x 4 core, AES-NI CPU Crypto: Yes (active).
The old SRX5308 never had any connection issues however it was slow and capped my fiber connection. I have installed pfSense 3 times to make sure I have a default installation. Last 3 months I have been searching and reading Internet for trouble shoooting tips. I have tested several tips and none have solved the issue so far.

kevindd992002

I've had spare time and proceeded with the reinstallation of pfsense from scratch and since then I had 0 problems with unbound and no packet loss, so far. I'm not entirely sure if the issue was caused by some bug in pfsense (doubtful) but like @johnpoz said it could be just my ISP fixing something in the backend and was just coincidental to when I reinstalled pfsense. So far so good though, I'll definitely post back if I encounter the same issue (which I certainly hope I won't).

kevindd992002

@johnpoz said in Intermittent connection issue:

The different with dnsmasq and unbound for forwarding is dnsmasq out of the box forwards to ALL of the forwarders you have set at the same time, and uses just uses the first answer. I don't think you can tell unbound to do that.. Would have to check the unbound docs - but there is no way to set that in the gui of pfsense. This can be good for when you have bad peering isp or issues talking via some paths, etc.

Well restarting unbound never helps because it flushes your cache. But you do have a lot of timeouts you showed - so something not great with your isp either which could be problems.

For all we know your isp connections has gotten better, and has zero to do with unbound restarting.

Again when you have issues with dns - you can not just assume the problem is X, you need to troubleshoot the exact issue your seeing... Not just dns not working sometimes.. Pick something that didn't work and find out why.. Are you still seeing a lot of timeouts in your infra info?

You can also make sure you setup prefetch with unbound, this can help with problematic issues because it will look up stuff in the background before the ttl expire and it flushes out of the cache. Also setting serve 0 ttl can really help as well. Since now even if the cache expired, it would serve up the last entry when client asks for it - and then look it up in the background again.

@johnpoz

Does this mean that unbound, by default, when set to forwarding mode queries the DNS servers you set in General sequentially and that you cannot change this behavior?

bjohe

@kevindd992002 and @johnpoz
This is my understanding from the documentation. "unbound will use the system DNS servers from System > General Setup or those received from a dynamic WAN, rather than using the root servers directly"

When I changed my DNS resolver setting to "Enable Forwarding Mode" most of my intermittent connection issues have disappeared. Sometimes I still have to wait a sec or two before the webpage loads completely, but In general the pages load immediately and there are no long periods of total non-connection.

I have not tried using the DNS Forwarder (dnsmasq). The documentation on “DNS Query Forwarding” mentions that DNSSEC need to be disabled but I have not done this. Maybe it works for me as I did check that the DNS servers I choose were supporting DNSSEC.