Intermittent connection issue
-
Your pings to 8.8.8.8 not being answered - ok, contact your ISP about it.. Has zero to do with pfsense, ZERO..
And this again has zero to do with unbound and your title.
"Non-forwarding Resolver intermittent operation"Unbound could give 2 shits about 8.8.8.8 not answering when it resolves, ie non forwarding mode..
-
@johnpoz said in Non-forwarding Resolver intermittent operation:
Your pings to 8.8.8.8 not being answered - ok, contact your ISP about it.. Has zero to do with pfsense, ZERO..
And this again has zero to do with unbound and your title.
"Non-forwarding Resolver intermittent operation"Unbound could give 2 shits about 8.8.8.8 not answering when it resolves, ie non forwarding mode..
@johnpoz from what I've understood is that @kevindd992002 did contact the ISP, but their tech is blaming pfSense and won't take any action. I can understand the frustration there. Even more frustrating is that in trying to prove it's not pfSense, it's only making things more confusing.
I would look at some of the suggestions in that link I sent. Also, putting the Asus in place of pfSense might be another step?
Oh, and something that might also give you more insight then ping is a trace route.
Raffi
-
8.8.8.8 not being answered and not really a huge deal with unbound is something that I agree with, ok, but that was just one sample server. As you can see with the packet capture, almost all servers that I was trying to ping had packet loss one way or another. So with unbound (non forwarding), I'm sure the query to the root hints servers is affected too as it seems that all destination servers are affected when the issue is happening.
As for the title, yes I'm sorry about that. I should've created a new thread that is not specific to unbound but it just got derailed and hard to abandon now. But if possible, the mods can put it in the correct forum section and let me edit the title.
@Raffi_ , yes I'll check the suggestions in that link tomorrow :) It's already 1:30AM here and will have to continue tomorrow.
And yes, I didn't forget about removing pfsense from the mix and try either just the asus router or directly connecting to the modem. Those steps will come eventually.
-
And show the tech that pings were sent and not answer - this has ZERO to do with what sent them... If the tech will not believe that.. Then they are an idiot..
Still not seeing what the problem is.. Like I said its all over the board.. If X is not answering a dns query.. Where is that listed? Was the dns query sent, then again not anything to do with pfsense..
What else is there to talk about... Simple sniff, if shows traffic being sent, and nothing answered it has nothing to do with pfsense.. There is nothing left to do if you sniff and see traffic being put on the wire, and nothing coming back - its outside the control of what put it on the wire.
You would have another thing if nothing being sent on the wire, or you see an answer but not being processed.
-
@johnpoz said in Non-forwarding Resolver intermittent operation:
And show the tech that pings were sent and not answer - this has ZERO to do with what sent them... If the tech will not believe that.. Then they are an idiot..
Still not seeing what the problem is.. Like I said its all over the board.. If X is not answering a dns query.. Where is that listed? Was the dns query sent, then again not anything to do with pfsense..
What else is there to talk about... Simple sniff, if shows traffic being sent, and nothing answered it has nothing to do with pfsense.. There is nothing left to do if you sniff and see traffic being put on the wire, and nothing coming back - its outside the control of what put it on the wire.
You would have another thing if nothing being sent on the wire, or you see an answer but not being processed.
Again, I'm simply asking help in what to tell the tech in trying to rule out pfsense. They are idiots, I agree, but I don't have a choice but to deal with them and convince them that the issue on their side.
-
@kevindd992002 I'm interested to know how the non pfSense experiments go.
If push comes to shove and they don't want to help you fix your problem, you nicely explain to them that you'll switch to their competitor (assuming there is one). You'll be surprised at how they suddenly become more interested in helping you solve it.
-
You have already proven the issue is there end, when you show a ping going out and not getting an answer.. Not sure what else you can do.. Do the same thing when just a PC connected to their device..
-
@Raffi_ said in Non-forwarding Resolver intermittent operation:
@kevindd992002 I'm interested to know how the non pfSense experiments go.
If push comes to shove and they don't want to help you fix your problem, you nicely explain to them that you'll switch to their competitor (assuming there is one). You'll be surprised at how they suddenly become more interested in helping you solve it.
Yeah, me too. I'll try accomplishing both direct-to-modem and using the Asus router tests this week and will report back. Thanks.
Yeah, I know how ISP's react when you say that. They panic and makes things solved faster. There are competitors, for sure, but in my condo there's only one offering FTTH connections, the one that I'm using now. The others are using crappy phone copper cables which are very substandard. And I just switched from that copper ISP to this fiber ISP since January 2019 so not long ago.
-
Where your going to have a problem is pinging stuff off their network and getting packet loss - they can always just say not their network..
You need to have pings going to their network, and not getting back answers.. Also many (all really) an ISP do not promise zero loss, so unless you have significant packet loss - good luck.. TCP can work just fine with a small amount of packet loss.. Do they have anything in their SLA about amount of packet loss?
If you call and say yeah over 10k pings I saw .01% loss - they will just laugh and say, ok so? But if you show that you have sustained loss say 5% then you might have something to complain about.
So I am going to say it again, a few packets here or there loss is not going to be an issue.. And not the root of your problem with issues with resolving stuff.. Is not like dns only does 1 query, and if no answer just says F it, doesn't work... DNS will send multiple queries before it gives up, and can even switch to tcp vs udp, etc. So for packet loss to be a problem with dns resolution it really needs to be a significant issue.
Also - for resolving, not forwarding the different NS will be tried - for example roots have 13 different NS, if one does not respond another will be tried.. Unbound keeps track of which ns respond faster, etc. And will use them more then ones that are less responsive.. Look at your infra cache..
If your having dns resolving issues - you need to troubleshoot that specific issue.. Not just that you lost some pings to 8.8.8.8
-
@johnpoz said in Non-forwarding Resolver intermittent operation:
Where your going to have a problem is pinging stuff off their network and getting packet loss - they can always just say not their network..
You need to have pings going to their network, and not getting back answers.. Also many (all really) an ISP do not promise zero loss, so unless you have significant packet loss - good luck.. TCP can work just fine with a small amount of packet loss.. Do they have anything in their SLA about amount of packet loss?
If you call and say yeah over 10k pings I saw .01% loss - they will just laugh and say, ok so? But if you show that you have sustained loss say 5% then you might have something to complain about.
So I am going to say it again, a few packets here or there loss is not going to be an issue.. And not the root of your problem with issues with resolving stuff.. Is not like dns only does 1 query, and if no answer just says F it, doesn't work... DNS will send multiple queries before it gives up, and can even switch to tcp vs udp, etc. So for packet loss to be a problem with dns resolution it really needs to be a significant issue.
I'm going to ask this again too: Would a 10-minute ping to google.com with ALL RTO's not considered an issue for you? I'm really having a hard time thinking why you wouldn't consider that an issue.
Like I said, I CARE LESS for few RTO's because they will be retried anyway, I agree with you completely. But if you start getting 100% RTO for a span of even just one minute and your clients cannot browse the Internet 100%, then where in the world is that not an issue?
-
I will record a video of the issue when I get the chance and post it here as proof.
-
10 minutes yeah that is a problem - But maybe outside the isp control, maybe its somewhere past where the traffic leaves the ISP.. You can get them to troubleshoot it is happening all the time and you can not get to google. But 8.8.8.8 is not google.
And 8.8.8.8 does not have anything to do with overall dns, unless its the authoritative NS for what your looking for and can not talk to.. Which seems really odd, since its an anycast.. Unless you are forwarding to it, which per your title you are not.
If you can show an outage pinging 8.8.8.8 for 10 minutes - then for sure you can bring that to your ISP attention and say hey - WTF... But unless you can show them that it happens more than once in a while your going to have a hard time getting their attention.
edit: Do not post any stupid videos.. JFC nobody is going to watch such nonsense... You can either resolve something or you can not, you can either ping something you can not... Show the sniff of the traffic, show the traceroute to the IP.. So the sniffs of the resolving action and getting no response, etc.
BTW - here are the NS involved in resolving google.com.. Notice 8.8.8.8 not there
[2.4.4-RELEASE][admin@sg4860.local.lan]/: unbound-control -c /var/unbound/unbound.conf lookup google.com The following name servers are used for lookup of google.com. ;rrset 78708 4 0 2 0 google.com. 78708 IN NS ns2.google.com. google.com. 78708 IN NS ns1.google.com. google.com. 78708 IN NS ns3.google.com. google.com. 78708 IN NS ns4.google.com. ;rrset 78623 1 0 1 0 ns4.google.com. 78623 IN A 216.239.38.10 ;rrset 78623 1 0 1 0 ns4.google.com. 78623 IN AAAA 2001:4860:4802:38::a ;rrset 78623 1 0 1 0 ns3.google.com. 78623 IN A 216.239.36.10 ;rrset 78623 1 0 1 0 ns3.google.com. 78623 IN AAAA 2001:4860:4802:36::a ;rrset 78623 1 0 1 0 ns1.google.com. 78623 IN A 216.239.32.10 ;rrset 78623 1 0 1 0 ns1.google.com. 78623 IN AAAA 2001:4860:4802:32::a ;rrset 78623 1 0 1 0 ns2.google.com. 78623 IN A 216.239.34.10 ;rrset 78623 1 0 1 0 ns2.google.com. 78623 IN AAAA 2001:4860:4802:34::a Delegation with 4 names, of which 0 can be examined to query further addresses. It provides 8 IP addresses. 2001:4860:4802:34::a expired, rto 154191216 msec, tA 0 tAAAA 0 tother 0. 216.239.34.10 rto 223 msec, ttl 776, ping 7 var 54 rtt 223, tA 0, tAAAA 0, tother 0, EDNS 0 probed. 2001:4860:4802:32::a rto 376 msec, ttl 776, ping 0 var 94 rtt 376, tA 0, tAAAA 0, tother 0, EDNS 0 assumed. 216.239.32.10 not in infra cache. 2001:4860:4802:36::a rto 376 msec, ttl 776, ping 0 var 94 rtt 376, tA 0, tAAAA 0, tother 0, EDNS 0 assumed. 216.239.36.10 rto 311 msec, ttl 776, ping 3 var 77 rtt 311, tA 0, tAAAA 0, tother 0, EDNS 0 probed. 2001:4860:4802:38::a rto 376 msec, ttl 776, ping 0 var 94 rtt 376, tA 0, tAAAA 0, tother 0, EDNS 0 assumed. 216.239.38.10 rto 252 msec, ttl 776, ping 4 var 62 rtt 252, tA 0, tAAAA 0, tother 0, EDNS 0 probed. [2.4.4-RELEASE][admin@sg4860.local.lan]/:
-
@johnpoz said in Non-forwarding Resolver intermittent operation:
10 minutes yeah that is a problem - But maybe outside the isp control, maybe its somewhere past where the traffic leaves the ISP.. You can get them to troubleshoot it is happening all the time and you can not get to google. But 8.8.8.8 is not google.
And 8.8.8.8 does not have anything to do with overall dns, unless its the authoritative NS for what your looking for and can not talk to.. Which seems really odd, since its an anycast.. Unless you are forwarding to it, which per your title you are not.
If you can show an outage pinging 8.8.8.8 for 10 minutes - then for sure you can bring that to your ISP attention and say hey - WTF... But unless you can show them that it happens more than once in a while your going to have a hard time getting their attention.
edit: Do not post any stupid videos.. JFC nobody is going to watch such nonsense... You can either resolve something or you can not, you can either ping something you can not... Show the sniff of the traffic, show the traceroute to the IP.. So the sniffs of the resolving action and getting no response, etc.
I know 8.8.8.8 is not google.com. These are two different servers that I showed in my tests above. I'm not sure why you're not following.
Well, I thought a video will help you believe me that there's a problem. If you don't want it, then fine. Obviously, your network troubleshooting skills are way better than mine but I make it to the point to give any information I deem necessary for everyone to check. This is why I'm asking for guidance.
I thought we're already past the point where we're not considering this to be a DNS issue anymore? If it was, then the resolution part is where I'll have issues but when the issue happens pinging random servers show RTO's as well. I'm not mentioning here that it is a DNS issue. 8.8.8.8 was just the monitor IP I have for the WAN gateway from the very start so that's what I showed everyone in this forum.
-
Here's a traceroute that I sent them a few months ago: https://pastebin.com/JqPx326v
That looks like the RTO starts from the hop that's within the ISP network. Is that enough evidence for them to conclude that the problem is in their network?
And then 3 minutes after the issue, it got resolved and the traceroute results became like this: https://pastebin.com/XYbNMiWy
-
You got to the end point in that trace.. That all hops along the way do not naswer does not always mean anything..
Not sure what you think that shows as a problem?
Tracing route to www.pfsense.org [208.123.73.69] over a maximum of 30 hops: 1 <1 ms <1 ms <1 ms 192.168.9.253 2 10 ms 9 ms 16 ms 50.4.132.1 3 11 ms 17 ms 10 ms 76.73.191.106 4 9 ms 9 ms 8 ms 76.73.164.142 5 12 ms 10 ms 9 ms 76.73.164.154 6 13 ms 10 ms 10 ms 76.73.191.242 7 11 ms 21 ms 10 ms 143.59.95.224 8 30 ms 15 ms 18 ms 75.76.35.8 9 * 13 ms 11 ms 4.16.38.157 10 * * * Request timed out. 11 36 ms 46 ms 37 ms 4.14.49.2 12 41 ms 35 ms 35 ms 64.20.229.158 13 36 ms 35 ms 35 ms 66.219.34.194 14 34 ms 38 ms 35 ms 208.123.73.4 15 39 ms 35 ms 39 ms 208.123.73.69
So from that trace I guess I am having issues getting to www.pfsense.org?
same goes for cnn seems
$ tracert -d www.cnn.com Tracing route to turner-tls.map.fastly.net [151.101.185.67] over a maximum of 30 hops: 1 <1 ms 3 ms <1 ms 192.168.9.253 2 10 ms 11 ms 16 ms 50.4.132.1 3 19 ms 20 ms 8 ms 76.73.191.106 4 11 ms 10 ms 9 ms 76.73.164.142 5 13 ms 10 ms 11 ms 76.73.164.154 6 10 ms 11 ms 11 ms 76.73.191.242 7 10 ms 10 ms 10 ms 143.59.95.224 8 13 ms 9 ms 10 ms 75.76.35.8 9 * * * Request timed out. 10 12 ms 10 ms 10 ms 151.101.185.67
-
@johnpoz said in Non-forwarding Resolver intermittent operation:
You got to the end point in that trace.. That all hops along the way do not naswer does not always mean anything..
Not sure what you think that shows as a problem?
Tracing route to www.pfsense.org [208.123.73.69] over a maximum of 30 hops: 1 <1 ms <1 ms <1 ms 192.168.9.253 2 10 ms 9 ms 16 ms 50.4.132.1 3 11 ms 17 ms 10 ms 76.73.191.106 4 9 ms 9 ms 8 ms 76.73.164.142 5 12 ms 10 ms 9 ms 76.73.164.154 6 13 ms 10 ms 10 ms 76.73.191.242 7 11 ms 21 ms 10 ms 143.59.95.224 8 30 ms 15 ms 18 ms 75.76.35.8 9 * 13 ms 11 ms 4.16.38.157 10 * * * Request timed out. 11 36 ms 46 ms 37 ms 4.14.49.2 12 41 ms 35 ms 35 ms 64.20.229.158 13 36 ms 35 ms 35 ms 66.219.34.194 14 34 ms 38 ms 35 ms 208.123.73.4 15 39 ms 35 ms 39 ms 208.123.73.69
So from that trace I guess I am having issues getting to www.pfsense.org?
Of course not! Some routers are setup to not respond to ICMP requests, I know that.
But how do you explain my first and second traceroute before and after (three minutes interval) the issue? Is it because the route to the same server changed in a span of 3 minutes?
-
@kevindd992002 said in Non-forwarding Resolver intermittent operation:
Here's a traceroute that I sent them a few months ago: https://pastebin.com/JqPx326v
That looks like the RTO starts from the hop that's within the ISP network. Is that enough evidence for them to conclude that the problem is in their network?
And then 3 minutes after the issue, it got resolved and the traceroute results became like this: https://pastebin.com/XYbNMiWy
@kevindd992002 That's not really proof of an issue on their network. Not all hops along the route will always respond. It's common to have hops that don't respond along the route. As long as at the end you get to the server, that's what matters. Also, the hops are not taking very long so that also looks OK.
-
@Raffi_ said in Non-forwarding Resolver intermittent operation:
@kevindd992002 said in Non-forwarding Resolver intermittent operation:
Here's a traceroute that I sent them a few months ago: https://pastebin.com/JqPx326v
That looks like the RTO starts from the hop that's within the ISP network. Is that enough evidence for them to conclude that the problem is in their network?
And then 3 minutes after the issue, it got resolved and the traceroute results became like this: https://pastebin.com/XYbNMiWy
@kevindd992002 That's not really proof of an issue on their network. Not all hops along the route will always respond. It's common to have hops that don't respond along the route. As long as at the end you get to the server, that's what matters. Also, the hops are not taking very long so that also looks OK.
Right, that's what I thought. I just posted the screenshots here in case you guys see something out of the ordinary.
-
@kevindd992002 said in Non-forwarding Resolver intermittent operation:
Yeah, I know how ISP's react when you say that. They panic and makes things solved faster. There are competitors, for sure, but in my condo there's only one offering FTTH connections, the one that I'm using now. The others are using crappy phone copper cables which are very substandard. And I just switched from that copper ISP to this fiber ISP since January 2019 so not long ago.
Off topic, but we're actually in the copper test industry. Believe it or not, they have technology now that is able to get close to Gigabit speeds on those old copper lines if the ISP is willing to invest in it. I'm curious are you in Australia? Here in the US, the old phone lines have been mostly abandoned in terms of further investment.
-
@Raffi_ said in Non-forwarding Resolver intermittent operation:
@kevindd992002 said in Non-forwarding Resolver intermittent operation:
Yeah, I know how ISP's react when you say that. They panic and makes things solved faster. There are competitors, for sure, but in my condo there's only one offering FTTH connections, the one that I'm using now. The others are using crappy phone copper cables which are very substandard. And I just switched from that copper ISP to this fiber ISP since January 2019 so not long ago.
Off topic, but we're actually in the copper test industry. Believe it or not, they have technology now that is able to get close to Gigabit speeds on those old copper lines if the ISP is willing to invest in it. I'm curious are you in Australia? Here in the US, the old phone lines have been mostly abandoned in terms of further investment.
I can imagine. I was mostly talking about how sub-standard the copper wires are here in our condo. Even the copper ISP's themselves tell me that the copper wires that the contractors used in this condo are crap. The copper wires in the building's cabinet are worse than how spaghetti looks like. And no one wants to invest to replace those. I'm in the Philippines, so a third-world country, but Internet service here came a long way already. My two service plans are 35 down/35 up (around $31) and 300 down/300 up (around $87).
-
The Asus router as the main router and without pfsense has been issue-free for the last two days. It's still too early to tell but I'll continue monitoring during the weekend (the time when the issue usually occurs most) before I come to a conclusion. If it does run flawless until Monday though, I'm not sure how to continue troubleshooting pfsense except to uninstall and reinstall it from scratch. I mean that's an easy task when I just need to reload the config but if I am to go that route I would want to not carry over any settings from the config (which might be corrupted or something, for all we know).
-
@kevindd992002 Interesting. Ok, that sounds like a good plan. Yea give it a little while to see how it goes. We'll see what the next step is from there. Have a good weekend.
Raffi -
After 5 days of continuously using the ASUS router, I've never had any single occurrence of the issue! That isolates the ASUS router, cables, and ISP modem from being the root cause of the issue.
I've decided, just now, to switch to pfsense and as soon as I've plugged it in and waited for everything to go green in the Dashboard, I experienced the issue. It's got to be either the pfsense software itself or the physical hardware that hosts pfsense (although I doubt this). What can you recommend as a next step here?
-
And your asus router was actual resolving for dns?
You title says non forwarding problems.. I find it unlikely that your asus router was resolving for dns vs forwarding..
Do you understand what the difference is?
-
@johnpoz said in Non-forwarding Resolver intermittent operation:
And your asus router was actual resolving for dns?
You title says non forwarding problems.. I find it unlikely that your asus router was resolving for dns vs forwarding..
Do you understand what the difference is?
Yes, I understand the difference between DNS resolver and DNS forwarder. I've already established this a few posts above. How can I rename the title for this whole thread and move it to the correct section? So that we can all be over the technicalities. Are my test results still not convincing for you that pfsense is causing my issue? What can I do to convince you?
The ASUS router is NOT a DNS resolver. It is a DNS forwarder and I was forwarding to the OpenDNS servers. That's the only main difference I see: pfsense was set as a DNS resolver (using root hints and not forwarding) while the ASUS router does not have this feature and is simply doing DNS forwarding.
-
So then - troubleshoot your dns problem when your "resolving" Or set pfsense to forward to opendns like your asus was doing.
-
@kevindd992002 said in Non-forwarding Resolver intermittent operation:
How can I rename the title for this whole thread and move it to the correct section?
FYI, you can change the title of this topic by going up to your very first post, then click on the 3 dots and go to edit. Then above the text box where it shows the title, you can go in and edit it to help avoid any confusion. Maybe something more generic like intermittent connection issue or whatever you like. As for moving it to the correct section, that would have to be done by someone else. You can start a new topic in the correct section if you'd like, but I think we already made some progress here.
After 5 days of continuously using the ASUS router, I've never had any single occurrence of the issue! That isolates the ASUS router, cables, and ISP modem from being the root cause of the issue.
I've decided, just now, to switch to pfsense and as soon as I've plugged it in and waited for everything to go green in the Dashboard, I experienced the issue. It's got to be either the pfsense software itself or the physical hardware that hosts pfsense (although I doubt this). What can you recommend as a next step here?
Here is what I would suggest doing if you haven't already.
- Make a backup of your current pfSense config.
- Do a fresh install of the latest pfSense (2.4.4-RELEASE-p3).
- Configure your interfaces as needed.
- Do not install any additional packages.
- Leave all the default settings. pfSense is pretty darn secure out of the box, certainly more secure then the Asus anyway.
Then let's see where that gets you.
Raffi
-
@johnpoz said in Non-forwarding Resolver intermittent operation:
So then - troubleshoot your dns problem when your "resolving" Or set pfsense to forward to opendns like your asus was doing.
I thought even you agreed that this is not a DNS problem? So why would I concentrate on troubleshooting DNS? I'm pinging an IP address when the issue happens so that in itself tells everyone already that it isn't a DNS issue.
@Raffi_ said in Non-forwarding Resolver intermittent operation:
@kevindd992002 said in Non-forwarding Resolver intermittent operation:
How can I rename the title for this whole thread and move it to the correct section?
FYI, you can change the title of this topic by going up to your very first post, then click on the 3 dots and go to edit. Then above the text box where it shows the title, you can go in and edit it to help avoid any confusion. Maybe something more generic like intermittent connection issue or whatever you like. As for moving it to the correct section, that would have to be done by someone else. You can start a new topic in the correct section if you'd like, but I think we already made some progress here.
After 5 days of continuously using the ASUS router, I've never had any single occurrence of the issue! That isolates the ASUS router, cables, and ISP modem from being the root cause of the issue.
I've decided, just now, to switch to pfsense and as soon as I've plugged it in and waited for everything to go green in the Dashboard, I experienced the issue. It's got to be either the pfsense software itself or the physical hardware that hosts pfsense (although I doubt this). What can you recommend as a next step here?
Here is what I would suggest doing if you haven't already.
- Make a backup of your current pfSense config.
- Do a fresh install of the latest pfSense (2.4.4-RELEASE-p3).
- Configure your interfaces as needed.
- Do not install any additional packages.
- Leave all the default settings. pfSense is pretty darn secure out of the box, certainly more secure then the Asus anyway.
Then let's see where that gets you.
Raffi
Changed the title already, thanks.
I was afraid that the only step I'm left with is to reinstall. But yeah, I can do that, but I'll probably not be able to until next week because I don't have a serial-to-USB adapter with me right now. I'll report back when I'm done with this but I'll continue monitoring while using the current pfsense installation.
-
@Raffi_ , since I didn't have time to start pfsense from scratch yet, I decided to just reset to factory defaults and restore the config. Same thing still happens.
One semi-consistent thing that I observed though is that the issue happens when I turn on my desktop computer from a state where it is turned off for a couple of hours. I say "semi-consistent" because when I restart or shut it down now and turn it back on immediately, the issue cannot be recreated. I've been observing this for a couple of days and it's as if that it is the one bogging down the whole network. Then I wait for 5 to 10 minutes and things come back to normal.
There's no special network configuration with my desktop computer aside from these things:
- Uses NUT (Network UPS Tools) to connect to the UPS master connected to pfsense.
- Wireshark/NPCap is installed but not running (the Npcap Loopback Adapter is installed)
- Hyper-V Manager is installed and the default virtual switch (has its own internal subnet) is configured.
- A mapped drive connecting to the Synology NAS on the other end of the OpenVPN tunnel is configured.
Can you think of anything out of those that could cause this?
Could it perhaps be a broadcast storm caused by the desktop PC? If so, how can I confirm this?
-
@kevindd992002 said in Intermittent connection issue:
how can I confirm this?
Sniff and see - be it on any device on that same L2 or pfsense... If you want to know what is going on - just look.
-
I agree with what John said. If you suspect something like the desktop traffic being an issue, run a packet capture on pfsense using the LAN interface.
I would suggest starting a packet capture on pfsense LAN before you turn on that PC which is causing the issues. Then while the capture is running, turn on the PC (after it's been off long enough to cause trouble) and see what's happening. Take note of the time when you first connect the PC so you can figure out at what point in the capture the PC was connected. I think it's a little coincidental that the default DHCP lease time in pfsense is 2 hours and the time it takes for your PC to stay disconnected before it starts causing trouble matches up with that.
Have you tried manually configuring a fixed IP on that PC instead of requesting a DHCP lease?
Raffi
-
@Raffi_ said in Intermittent connection issue:
I agree with what John said. If you suspect something like the desktop traffic being an issue, run a packet capture on pfsense using the LAN interface.
I would suggest starting a packet capture on pfsense LAN before you turn on that PC which is causing the issues. Then while the capture is running, turn on the PC (after it's been off long enough to cause trouble) and see what's happening. Take note of the time when you first connect the PC so you can figure out at what point in the capture the PC was connected. I think it's a little coincidental that the default DHCP lease time in pfsense is 2 hours and the time it takes for your PC to stay disconnected before it starts causing trouble matches up with that.
Have you tried manually configuring a fixed IP on that PC instead of requesting a DHCP lease?
Raffi
Ok, I'll try that and report back.
No, I have not tried a static IP on the PC but it has an IP reservation from the DHCP server, like all my other servers/PC's that I need to RDP from outside do. But yeah, I'll see if a static IP makes a difference to rule out DHCP.
-
Keep in mind that default packet capture on pfsense is set to 100, so you will most likely want to adjust that.. 0 would be a good setting.. Just don't forget its running ;)
-
Alright, so after some more testing I can say it's not the desktop causing the issue as it was more like a coincidence that happened. I've did more steps regarding DNS and the results were reproducible this time:
- Changed from regular DNS Resolver to DNS Resolver with forwarding (set to forward to 192.168.100.1 which is the modem's IP/gateway). Ping'ed 208.78.70.23 (one IP I saw in Status -> DNS Resolver) from pfsense's ping utility and there were no issues. Internet is back up and running.
- Changed back to regular DNS Resolver. Ping'ed the same IP from pfsense's ping utility and no response from 192.168.100.1.
So it looks like the problem is DNS but not the resolution part of DNS per se. For some reason, when DNS Resolver without forwarding is set the WAN gateway doesn't respond properly (or responds intermittently) as well, at least that's what's happening during my testing. So if you try to resolve with DNS Resolver, it's "as if" it is not resolving properly because of course it doesn't get any responses from the root hints servers consistently.
I know, the issue doesn't make sense but will getting a WAN packet capture while recreating the issue help in this situation? Or both WAN and LAN?
EDIT:
Also, I get a ton of Timeout A's under Status -> DNS Resolver when forwarding is NOT set. When forwarding, I get 0.
-
@kevindd992002 said in Intermittent connection issue:
Alright, so after some more testing I can say it's not the desktop causing the issue as it was more like a coincidence that happened. I've did more steps regarding DNS and the results were reproducible this time:
- Changed from regular DNS Resolver to DNS Resolver with forwarding (set to forward to 192.168.100.1 which is the modem's IP/gateway). Ping'ed 208.78.70.23 (one IP I saw in Status -> DNS Resolver) from pfsense's ping utility and there were no issues. Internet is back up and running.
- Changed back to regular DNS Resolver. Ping'ed the same IP from pfsense's ping utility and no response from 192.168.100.1.
So it looks like the problem is DNS but not the resolution part of DNS per se. For some reason, when DNS Resolver without forwarding is set the WAN gateway doesn't respond properly (or responds intermittently) as well, at least that's what's happening during my testing. So if you try to resolve with DNS Resolver, it's "as if" it is not resolving properly because of course it doesn't get any responses from the root hints servers consistently.
I know, the issue doesn't make sense but will getting a WAN packet capture while recreating the issue help in this situation? Or both WAN and LAN?
EDIT:
Also, I get a ton of Timeout A's under Status -> DNS Resolver when forwarding is NOT set. When forwarding, I get 0.
Ooooohhh I can't believe it didn't hit me sooner. This is a pretty well known issue with the DNS resolver (unbound) in pfSense. Here is one thread on it that I ran into when I had similar issues. https://forum.netgate.com/topic/120838/unbound-appears-to-restart-frequently-and-fails-to-resolve-domains-sometimes.
There are a number of solutions to this. As you found, use DNS forwarding. I would suggest forwarding to servers like Cloudflare (1.1.1.1 and 1.0.0.1) or Google (8.8.8.8 and 8.8.4.4). If you are going to use pfSense for resolving, go into the DNS resolver settings and uncheck the DHCP Registration option. That would explain exactly the issue you were seeing. When a PC is connected after 2 hours, it would request a new DHCP lease. When that lease is handed out, it also causes Unbound to restart because Unbound has to update its info since DHCP leases are being registered. This was a major pain for me. If you also have things that slow down the unbound startup process like lots of pfblocker IP lists, that will make matters worse.My current setup has DHCP registration disabled in the DNS Resolver options. I also have forwarding enabled to 1.1.1.1 and 1.0.0.1. Ideally, I wanted to have DNS resolution done by pfSense without having to forward, but it was not always reliable. I think the better solution would be to have a DNS resolver completely separate from pfSense, like Pi hole. I hear good things about it but never had a chance to play around with it.
-
@Raffi_ said in Intermittent connection issue:
@kevindd992002 said in Intermittent connection issue:
Alright, so after some more testing I can say it's not the desktop causing the issue as it was more like a coincidence that happened. I've did more steps regarding DNS and the results were reproducible this time:
- Changed from regular DNS Resolver to DNS Resolver with forwarding (set to forward to 192.168.100.1 which is the modem's IP/gateway). Ping'ed 208.78.70.23 (one IP I saw in Status -> DNS Resolver) from pfsense's ping utility and there were no issues. Internet is back up and running.
- Changed back to regular DNS Resolver. Ping'ed the same IP from pfsense's ping utility and no response from 192.168.100.1.
So it looks like the problem is DNS but not the resolution part of DNS per se. For some reason, when DNS Resolver without forwarding is set the WAN gateway doesn't respond properly (or responds intermittently) as well, at least that's what's happening during my testing. So if you try to resolve with DNS Resolver, it's "as if" it is not resolving properly because of course it doesn't get any responses from the root hints servers consistently.
I know, the issue doesn't make sense but will getting a WAN packet capture while recreating the issue help in this situation? Or both WAN and LAN?
EDIT:
Also, I get a ton of Timeout A's under Status -> DNS Resolver when forwarding is NOT set. When forwarding, I get 0.
Ooooohhh I can't believe it didn't hit me sooner. This is a pretty well known issue with the DNS resolver (unbound) in pfSense. Here is one thread on it that I ran into when I had similar issues. https://forum.netgate.com/topic/120838/unbound-appears-to-restart-frequently-and-fails-to-resolve-domains-sometimes.
There are a number of solutions to this. As you found, use DNS forwarding. I would suggest forwarding to servers like Cloudflare (1.1.1.1 and 1.0.0.1) or Google (8.8.8.8 and 8.8.4.4). If you are going to use pfSense for resolving, go into the DNS resolver settings and uncheck the DHCP Registration option. That would explain exactly the issue you were seeing. When a PC is connected after 2 hours, it would request a new DHCP lease. When that lease is handed out, it also causes Unbound to restart because Unbound has to update its info since DHCP leases are being registered. This was a major pain for me. If you also have things that slow down the unbound startup process like lots of pfblocker IP lists, that will make matters worse.My current setup has DHCP registration disabled in the DNS Resolver options. I also have forwarding enabled to 1.1.1.1 and 1.0.0.1. Ideally, I wanted to have DNS resolution done by pfSense without having to forward, but it was not always reliable. I think the better solution would be to have a DNS resolver completely separate from pfSense, like Pi hole. I hear good things about it but never had a chance to play around with it.
I'm aware that DHCP registration restarts unbound but it does it just for a few seconds and everything should be back to normal. It doesn't explain what I was seeing with my desktop because that machine has an DHCP IP reservation. I thought those won't restart unbound? Also, if the unbound restart is the issue I won't be experiencing the issue for more than a few seconds. But in my case, I experience it for around 5 minutes or so, every single time.
Another weird thing is that I have the same exact setup on my network in the other end of the tunnel. That network also uses unbound without forwarding. And that network is 10x bigger than the network I'm trying to troubleshoot here. The network in question in this thread is made up of just 1 desktop, 1 nas, 1 laptop, 2 mobile devices, 1 nvidia shield, and 1 ps4. That's it.
Also, if it was a matter of unbound not working, I should be able to ping a public IP address (that's known to be pingable, of course) using pfsense. But like I said above, as long as I use unbound without forwarding, the issue of me not being able to ping presents itself right away.
-
Right, that was the thing that I couldn't figure out as well. Even if unbound was restarting and DNS was not working, I would still expect pings to 8.8.8.8 to work.
Do you have "Register DHCP static mappings" enabled in the DNS Resolver settings? I would expect that will cause unbound to restart as well. To be clear, I don't think the DHCP/Static IP requests are the cause of the problem, but I think they seem to be triggering the problem. You pretty much have it isolated down to unbound. Enabling forwarding bypasses unbound.
Bypassing unbound = working.
Not bypassing unbound = not working
Is that correct?It sounds like you haven't actually checked whether the action of connecting the PC was causing unbound to restart. Check the DNS Resolver log under system logs. It could be that unbound is not behaving as expected so look into that. Is unbound actually restarting quickly as expected? Is it hanging and causing pfSense to hang as well? pfSense itself will use unbound if that's the only option available for DNS resolution. There is even a setting to disable that default behavior in the general setup.
I'm not an expert on the topic of unbound but I would suggest looking into that though. Make sure that unbound is running and there aren't multiple instances of it or something crazy like that. I'm not being sarcastic here, I honestly do not know how pfSense would behave if the DNS server it was relying on to do its job was not working. -
@Raffi_ said in Intermittent connection issue:
Right, that was the thing that I couldn't figure out as well. Even if unbound was restarting and DNS was not working, I would still expect pings to 8.8.8.8 to work.
Do you have "Register DHCP static mappings" enabled in the DNS Resolver settings? I would expect that will cause unbound to restart as well. To be clear, I don't think the DHCP/Static IP requests are the cause of the problem, but I think they seem to be triggering the problem. You pretty much have it isolated down to unbound. Enabling forwarding bypasses unbound.
Bypassing unbound = working.
Not bypassing unbound = not working
Is that correct?It sounds like you haven't actually checked whether the action of connecting the PC was causing unbound to restart. Check the DNS Resolver log under system logs. It could be that unbound is not behaving as expected so look into that. Is unbound actually restarting quickly as expected? Is it hanging and causing pfSense to hang as well? pfSense itself will use unbound if that's the only option available for DNS resolution. There is even a setting to disable that default behavior in the general setup.
I'm not an expert on the topic of unbound but I would suggest looking into that though. Make sure that unbound is running and there aren't multiple instances of it or something crazy like that. I'm not being sarcastic here, I honestly do not know how pfSense would behave if the DNS server it was relying on to do its job was not working.Yes, that's pretty much correct. Since your last post, I decided to keep forwarding enabled in unbound and things were smooth similar to how things were when I used my Asus router. And then I decided to turn on my desktop at 11/6/2019 6:51 AM today and surprisingly the monitoring graph caught the issue:
These were the system logs at that time:
Nov 6 06:53:13 php-fpm 77800 /index.php: Successful login for user 'kevindd992002' from: 192.168.20.21 (Local Database) Nov 6 06:53:18 rc.gateway_alarm 54269 >>> Gateway alarm: WAN_DHCP (Addr:8.8.8.8 Alarm:1 RTT:21.129ms RTTsd:.260ms Loss:21%) Nov 6 06:53:18 check_reload_status updating dyndns WAN_DHCP Nov 6 06:53:18 check_reload_status Restarting ipsec tunnels Nov 6 06:53:18 check_reload_status Restarting OpenVPN tunnels/interfaces Nov 6 06:53:18 check_reload_status Reloading filter Nov 6 06:53:20 php-fpm 380 /rc.dyndns.update: MONITOR: WAN_DHCP is down, omitting from routing group Failover 8.8.8.8|192.168.100.2|WAN_DHCP|21.139ms|0.274ms|21%|down Nov 6 06:53:20 php-fpm 77800 /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. '' Nov 6 06:53:20 php-fpm 77800 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN_DHCP. Nov 6 06:53:20 php-fpm 380 /rc.dyndns.update: 380MONITOR: WAN_DHCP is available now, adding to routing group Failover 8.8.8.8|192.168.100.2|WAN_DHCP|21.141ms|0.276ms|20%|loss Nov 6 06:53:20 php-fpm 73145 /rc.filter_configure_sync: MONITOR: WAN_DHCP is down, omitting from routing group Failover 8.8.8.8|192.168.100.2|WAN_DHCP|21.139ms|0.3ms|21%|down Nov 6 06:53:21 php-fpm 380 /rc.dyndns.update: 380MONITOR: WAN_DHCP is available now, adding to routing group Failover 8.8.8.8|192.168.100.2|WAN_DHCP|21.142ms|0.296ms|20%|loss Nov 6 06:53:22 php-fpm 380 /rc.dyndns.update: phpDynDNS (Condo): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Nov 6 06:53:30 rc.gateway_alarm 83271 >>> Gateway alarm: WAN_DHCP (Addr:8.8.8.8 Alarm:0 RTT:21.140ms RTTsd:.302ms Loss:20%) Nov 6 06:53:30 check_reload_status updating dyndns WAN_DHCP Nov 6 06:53:30 check_reload_status Restarting ipsec tunnels Nov 6 06:53:30 check_reload_status Restarting OpenVPN tunnels/interfaces Nov 6 06:53:30 check_reload_status Reloading filter Nov 6 06:53:31 rc.gateway_alarm 87686 >>> Gateway alarm: WAN_DHCP (Addr:8.8.8.8 Alarm:1 RTT:21.144ms RTTsd:.318ms Loss:21%) Nov 6 06:53:31 check_reload_status updating dyndns WAN_DHCP Nov 6 06:53:31 check_reload_status Restarting ipsec tunnels Nov 6 06:53:31 check_reload_status Restarting OpenVPN tunnels/interfaces Nov 6 06:53:31 check_reload_status Reloading filter Nov 6 06:53:31 php-fpm 73145 /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. '' Nov 6 06:53:31 php-fpm 73145 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN_DHCP. Nov 6 06:53:31 php-fpm 380 /rc.filter_configure_sync: MONITOR: WAN_DHCP is down, omitting from routing group Failover 8.8.8.8|192.168.100.2|WAN_DHCP|21.144ms|0.318ms|21%|down Nov 6 06:53:32 php-fpm 380 /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. '' Nov 6 06:53:32 php-fpm 380 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN_DHCP. Nov 6 06:53:33 php-cgi notify_monitor.php: Message sent to kevindd992002@yahoo.com OK Nov 6 06:53:36 php-fpm 381 /rc.dyndns.update: phpDynDNS (Condo): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Nov 6 06:53:40 php-fpm 77800 /rc.dyndns.update: phpDynDNS (Condo): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Nov 6 06:53:57 php-cgi notify_monitor.php: Message sent to kevindd992002@yahoo.com OK Nov 6 06:54:11 rc.gateway_alarm 30595 >>> Gateway alarm: WAN_DHCP (Addr:8.8.8.8 Alarm:0 RTT:21.204ms RTTsd:.290ms Loss:20%) Nov 6 06:54:11 check_reload_status updating dyndns WAN_DHCP Nov 6 06:54:11 check_reload_status Restarting ipsec tunnels Nov 6 06:54:11 check_reload_status Restarting OpenVPN tunnels/interfaces Nov 6 06:54:11 check_reload_status Reloading filter Nov 6 06:54:12 php-fpm 33202 /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. '' Nov 6 06:54:12 php-fpm 33202 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN_DHCP. Nov 6 06:54:12 php-fpm 380 /rc.dyndns.update: 380MONITOR: WAN_DHCP is available now, adding to routing group Failover 8.8.8.8|192.168.100.2|WAN_DHCP|21.195ms|0.296ms|18%|loss Nov 6 06:54:14 php-fpm 380 /rc.dyndns.update: phpDynDNS (Condo): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Nov 6 06:54:32 php-cgi notify_monitor.php: Message sent to kevindd992002@yahoo.com OK
I don't see anything in the DNS Resolver logs too. I mean, the unbound service restarted as expected but it's just 1 second and that shouldn't be the issue:
Nov 6 06:46:22 unbound 28470:0 notice: Restart of unbound 1.9.1. Nov 6 06:46:22 unbound 28470:0 notice: init module 0: validator Nov 6 06:46:22 unbound 28470:0 notice: init module 1: iterator Nov 6 06:46:23 unbound 28470:0 info: start of service (unbound 1.9.1). Nov 6 06:46:29 unbound 28470:1 info: generate keytag query _ta-4f66. NULL IN Nov 6 06:53:21 filterdns merge_config: configuration reload Nov 6 06:53:21 filterdns Adding Action: pf table: HostsToTunnel host: plex.tv Nov 6 06:53:31 filterdns merge_config: configuration reload Nov 6 06:53:31 filterdns Adding Action: pf table: HostsToTunnel host: plex.tv Nov 6 06:53:33 filterdns merge_config: configuration reload Nov 6 06:53:33 filterdns Adding Action: pf table: HostsToTunnel host: plex.tv Nov 6 06:54:13 filterdns merge_config: configuration reload Nov 6 06:54:13 filterdns Adding Action: pf table: HostsToTunnel host: plex.tv Nov 6 07:43:46 unbound 28470:0 info: service stopped (unbound 1.9.1). Nov 6 07:43:46 unbound 28470:0 info: server stats for thread 0: 740 queries, 216 answers from cache, 524 recursions, 0 prefetch, 0 rejected by ip ratelimiting Nov 6 07:43:46 unbound 28470:0 info: server stats for thread 0: requestlist max 2 avg 0.0877863 exceeded 0 jostled 0 Nov 6 07:43:46 unbound 28470:0 info: average recursion processing time 0.065975 sec Nov 6 07:43:46 unbound 28470:0 info: histogram of recursion processing times Nov 6 07:43:46 unbound 28470:0 info: [25%]=0.0055808 median[50%]=0.0104123 [75%]=0.0318903 Nov 6 07:43:46 unbound 28470:0 info: lower(secs) upper(secs) recursions
I also don't see anything unusual in the DHCP logs:
Nov 6 06:51:31 dhcpd DHCPDISCOVER from <MAC> via igb1 Nov 6 06:51:31 dhcpd DHCPOFFER on 192.168.20.21 to <MAC> via igb1 Nov 6 06:51:34 dhcpd DHCPREQUEST for 192.168.20.21 (192.168.20.1) from <MAC> via igb1 Nov 6 06:51:34 dhcpd DHCPACK on 192.168.20.21 to <MAC> via igb1 Nov 6 07:00:41 dhcpd DHCPREQUEST for 192.168.20.21 from <MAC>via igb1 Nov 6 07:00:41 dhcpd DHCPACK on 192.168.20.21 to <MAC> via igb1 Nov 6 07:00:42 dhcpd DHCPREQUEST for 192.168.20.21 from <MAC> via igb1 Nov 6 07:00:42 dhcpd DHCPACK on 192.168.20.21 to <MAC> via igb1
So at this point, it's a mix of unbound and the desktop causing the problem? I'm scratching my head hard here. I can definitely say that with forwarding enabled, it was stable for 4 straight days. Turning on my laptop did not do anything to the network compared to when it also caused issues when using just unbound.
-
@kevindd992002 said in Intermittent connection issue:
Loss:21%)
That triggered bunch of different things... Since you prob have it set to reset on loss of gateway.
Why are you monitoring 8.8.8.8 and not pfsense gateway?
-
@johnpoz said in Intermittent connection issue:
@kevindd992002 said in Intermittent connection issue:
Loss:21%)
That triggered bunch of different things... Since you prob have it set to reset on loss of gateway.
Why are you monitoring 8.8.8.8 and not pfsense gateway?
Right, but is it just a big coincidence that the loss happened right after I turned on the desktop?
I'm monitoring 8.8.8.8 because pfsense's gateway is 192.168.100.1 (modem IP because of CGNAT). So if Internet is down, that private IP will always be up and this obviously would not be accurate for gateway monitoring, is it?