pfSense WAN connection hangs after about a minute
-
Sorry for the delay in responding. I did another packet capture on em0 (the NIC underlying the PPP connection). This time I started the packet capture with the WAN cable unplugged, plugged it in, and recorded until it died (about 35 seconds later). I don't see any malformed packets (that I recognize as such) but I do see some spurious re-transmissions as well as a small number of RST packets.
I've attached the PCAP (as much as the forum allows, about the first 8k packets out of 22k total). Anything stand out to you?
If not, then I'm down to calling the Telco and seeing if they can offer any advice. Though I expect their response to be essentially "use our provided router or get stuffed". Regarding which, their provided router is just an ancient D-Link DIR-825 with stock software. Nothing exciting there.em0-2020-10-21-1.zip
-
I don't see how that sniff is from before you connected.. It starts with a flood of DNS..
Why would you be sending a syn (tcp) to 53??
Also - clearly your missing start of conversations... This one for example
There is no syn here, you see sending http get, never getting a response.. So retran, then finally says screw it and sends fin.. Get a answer finally, then client side says screw it and sends RST..
But you never see the start of this conversation.
So how exactly did you start this sniff before you wan was even up? Make sure you clear all states if your going to do that again.. But default I believe is to flush all states when connection goes down, have you adjusted that setting?
Game is on - so that is my comments for now.. Will look later after the game, or maybe halftime
-
I haven't adjusted the settings for flushing states. And just to confirm I haven't accidentally borked anything up I have done a reset on the box settings and then re-configured for my local subnet, PPPoE connection, and DNS Resolver.
For the Sniff, my first attempt was to just unplug the WAN cable (LAN still plugged in), start the capture on WAN in promiscuous mode and then plug it back in. That just got me that flood of DNS packets as you saw. I also tried clearing states and then starting a pcap and, as quickly as I could, running "/usr/local/sbin/pfSctl -c 'interface reload wan'" from the shell but that seems to be giving me similar results to what I already posted.
Is there a better way for me to get good data during PPP startup?
-
When you pull the cable on wan.. Clear all your states.. then start you sniff, then plug the cable back in.. We should then see everything, your pppoe connection, your dhcp on your wan, etc. And since all states are cleared, nothing will be allowed through the firewall to even go out the wan without sending a new syn to start a conversation.
We can then see what conversations are failing, what are working.
Also make sure you run it the whole time until your saying nothing is working.. You said it was like 30 some seconds that it worked, but that sniff is only 11 seconds.
Why I don't get is why so much tcp dns traffic... These all seem to be root servers, but your talking to them via tcp.. Is UDP blocked??
This seems nuts... in less than 1ms you sent 870 something dns queries - not one of them I see from quick look got answered over UDP..
That doesn't seem normal... Most of your dns should be UDP.. and only stuff that is LARGE answers should have to do TCP..
-
Some aliases with fqdns being resolved?
Though those all look like typical stuff you might have behind pfSense.
You have Unbound configured in resolving mode? I could just about imagine something at your ISP is blocking you based on those DNS requests. The ISPs router will just be using the DNS servers supplied by the ISP, that would be a difference when using pfSense.
You could try switching Unbound to forwarding mode and checking 'Allow DNS server list to be overridden by DHCP/PPP on WAN' in System > General Setup.Steve
-
@stephenw10 said in pfSense WAN connection hangs after about a minute:
I could just about imagine something at your ISP is blocking you based on those DNS requests.
Yeah I agree something is blocking dns, tries udp - not working so tries tcp..
I didn't notice anything bad in the queries - just a freaking shit ton of them.. 870 queries in 1 ms?? How many clients do you have behind pfsense?
-
I'll follow your suggestions on getting a new PCAP. I'm getting a small switch in place with just my client laptop and pfSense connected to it to minimize extraneous traffic for the PCAP.
As for clients, this is just a home network so 6-8 PCs, a couple of appliances, and a few mobile devices.
I was running DNS to Cloudflare (1.1.1.1 and 1.0.0.1), have the DNS Forwarder disabled, and the DNS Resolver enabled. Apparently I unchecked 'enable forwarding mode' by mistake somewhere in my testing yesterday. I'll fix that and change to using the default IP specified DNS servers for my next test (I've tested it before and it didn't help, but at this point I'm eliminating as many variables as I can). DNS Resolver will have Network Interfaces set to LAN and Localhost, with Outgoing Network Interface set to WAN.
I'll get a new PCAP completed and uploaded as soon as I can.
-
Success! My one laptop test worked. pfSense is up and running. After testing for 10-15 minutes with just one laptop I put it back on the main switch and so far so good.
The only thing I couldn't do was get a good pcap of the WAN link coming up. As suggested, I rebooted pfSense, cleared the States, re-logged into the GUI and then started a promiscuous pcap with full details up to 10,000 packets on the WAN interface with the WAN cable unplugged. I then plugged the cable in and worked with it until it stopped (or in this case, didn't stop).
When I stopped the pcap after a few minutes, I only had three packets listed. I've pasted the three packets in below, but why only three IPv6 packets? I should be seeing a lot more than this shouldn't I? Or does plugging the cable in somehow disrupt the capture?
No. Time Source Destination Protocol Length Info
1 0.000000 :: ff02::16 ICMPv6 100 Multicast Listener Report Message v2Frame 1: 100 bytes on wire (800 bits), 100 bytes captured (800 bits)
Null/Loopback
Internet Protocol Version 6, Src: ::, Dst: ff02::16
Internet Control Message Protocol v6No. Time Source Destination Protocol Length Info
2 0.152920 :: ff02::1:ff1e:22f0 ICMPv6 76 Neighbor Solicitation for fe80::2e0:67ff:fe1e:22f0Frame 2: 76 bytes on wire (608 bits), 76 bytes captured (608 bits)
Null/Loopback
Internet Protocol Version 6, Src: ::, Dst: ff02::1:ff1e:22f0
Internet Control Message Protocol v6No. Time Source Destination Protocol Length Info
3 0.636174 :: ff02::16 ICMPv6 80 Multicast Listener Report Message v2Frame 3: 80 bytes on wire (640 bits), 80 bytes captured (640 bits)
Null/Loopback
Internet Protocol Version 6, Src: ::, Dst: ff02::16
Internet Control Message Protocol v6 -
So after switching back to DNS forwarding mode and using Clouflare for upstream you are no longer being blocked after a few minutes?
Are you using DNS over TLS for that?Steve
-
That's correct except for the Cloudflare part. Right now it's just using the ISP DNS provided by the PPPoE connection. I'll try Cloudflare and DNS over TLS again this evening and see if that affects anything.
-
So it was never that your connection was down, its your resolution of dns was not work.. Which is why somestuff worked and other stuff didn't
-
Not entirely. While DNS was definitely part of the issue, the WAN connection did appear to stop returning packets. For example, pings from a box on the LAN to a public IP like 8.8.8.8 would start timing out; as would pings to any other public IPs. I would initially get DNS resolution and traffic, but after a short period (15-60 seconds) both DNS resolution and ICMP responses would stop.
-
I re-enabled Cloudflare DNS+TLS and it's still stable. I'm beginning to think it was that Enable Forwarding Mode checkbox that caused the issue (will test this later when the rest of the household isn't online). I wonder if all the root traffic was convincing my ISP that I was hosting a DNS server and they blocked my connection. I'll report back if I find out anything further, but I'm happy that the Vault is up and working like it should be.
Thank you both for taking the time to look at my issue and for providing both solid suggestions and feedback. It really helped to have some experienced eyes on the problem.
-
@fulkren said in pfSense WAN connection hangs after about a minute:
I'm beginning to think it was that Enable Forwarding Mode checkbox that caused the issue
Huh?? When you forward you do not talk to roots..
-
Sorry, I should have been more specific. I was postulating that accidentally having Enable Forwarding Mode disabled (unchecked) was causing the issue.
-
I would check with your ISP - I personally would be livid and looking for another ISP if my isp was interfering in my dns traffic like that.
An intelligent man is sometimes forced to be drunk to spend time with his fools
If you get confused: Listen to the Music Play
Please don't Chat/PM me for help, unless mod related
SG-4860 24.11 | Lab VMs 2.8, 24.11 -
Yeah, that should not happen. But that is what I was postulating above.
Steve
-
@johnpoz said in pfSense WAN connection hangs after about a minute:
I would check with your ISP - I personally would be livid and looking for another ISP if my isp was interfering in my dns traffic like that.
I love when you speak from your heart. Usually that's the truth.
In fact ISP use your DNS queries to make a profile of you, it can be use for you (adds) or against you (if you sue ISP). On top of that they reduced costs by participate on Google Global Cache or GGC program for Youtube, Netflix, and so on.
Just block their GGC cache and you will get a faster speed.
