Experiencing and troubleshooting network latency issues?

[amrogers3]

I am experiencing some latency issues and I am not sure what exactly is causing it.
I checked DNS response times and I am getting above 150ms using dig, even up to 671ms.
last few query times:
166ms
149ms
201ms
151ms
199ms

What would be the best way to troubleshoot this issue and what may be causing it?

[AndyRH]

First question is what provides DNS and what is the chain to get to a DNS server with the answer?
2nd question is what is your network latency when pinging google or any other well placed server?
3rd, what type of WAN do you have? Fiber, DSL, cell, etc.

[stephenw10]

Yeah seeing latency like that to several servers looks like a wireless link. But could be a problem on a wired connection.

The MTR package can be useful for diagnosing issues like this.

[amrogers3]

@AndyRH
Thanks Andy, appreciate the help. Apologies for the big photos, I cannot seem to resize.

1. I think you are asking for a dig +trace, it is querying the local interface
   

queuing from Diagnostics --> DNS

3. I have fiber

[amrogers3]

@stephenw10
Just installed MTR, these readings don't seem to be way off. Any other troubleshooting steps you could recommend?
This is over a wired connection.

[stephenw10]

Yup those look OK. But no response from 1.1.1.1 or 9.9.9.9 does not. Can you ping either of those?

Could be something upstream blocking DNS traffic.

[amrogers3]

@stephenw10
Something is going on. I am experiencing degradation of speed and increased latency. Yes, I can ping both 9.9.9.9 and 1.1.1.1.

I was trying to forward all DNS to local using floating rule

[stephenw10]

Hmm, so you have a port forward NAT rule redirecting DNS traffic to pfSense?

The floating WAN rule looks like it's simply rejecting incoming requests. If it was applying in both directions it would be breaking DNS entirely.

Neither of those things would cause latency though. What does MTR show against 1.1.1.1 or 9.9.9.9?

[johnpoz]

@amrogers3 why would you reject on wan?? Your asking to ddos yourself to be honest. If someone/something did a query to your IP for anything - you would now send a reply.. The more people asking you, more traffic you send.

So see here I did a query to your IP

06:07:37.044348 IP 209.x.x.x.27758 > x.x.x.x.53: UDP, length 55
06:07:37.112595 IP x.x.x.x > 209.x.x.x: ICMP x.x.x.x udp port 53 unreachable, length 91

And you sent back via icmp - hey buddy you can't get there.

You would almost never want to do that on a wan side interface.. If you want to specifically block something on your wan, a normal block would be better - it would just ignore it then. But specific blocks on wan would be a bit weird in general, unless your looking to stop specific IPs or netblocks from seeing your open ports. I block a lot of networks on my wan for example - from the known scanners - no desire to have my open ports listed in some db somewhere. And I have specific allows for the ports I do have open based on country.. None of my users of my plex for example would be coming from RU, etc.. So I only allow US to talk to my plex port.

Your pings to 9.9.9.9 are all over the board. With their global pops, I would think you should get a more constant RTT.

$ ping 9.9.9.9 -t                                            
                                                             
Pinging 9.9.9.9 with 32 bytes of data:                       
Reply from 9.9.9.9: bytes=32 time=11ms TTL=58                
Reply from 9.9.9.9: bytes=32 time=10ms TTL=58                
Reply from 9.9.9.9: bytes=32 time=16ms TTL=58                
Reply from 9.9.9.9: bytes=32 time=14ms TTL=58                
Reply from 9.9.9.9: bytes=32 time=11ms TTL=58                
Reply from 9.9.9.9: bytes=32 time=10ms TTL=58                
Reply from 9.9.9.9: bytes=32 time=11ms TTL=58                
Reply from 9.9.9.9: bytes=32 time=10ms TTL=58                
Reply from 9.9.9.9: bytes=32 time=10ms TTL=58                
Reply from 9.9.9.9: bytes=32 time=10ms TTL=58                
Reply from 9.9.9.9: bytes=32 time=11ms TTL=58                
Reply from 9.9.9.9: bytes=32 time=11ms TTL=58                
Reply from 9.9.9.9: bytes=32 time=10ms TTL=58                
Reply from 9.9.9.9: bytes=32 time=11ms TTL=58                
                                                             
Ping statistics for 9.9.9.9:                                 
    Packets: Sent = 14, Received = 14, Lost = 0 (0% loss),   
Approximate round trip times in milli-seconds:               
    Minimum = 10ms, Maximum = 16ms, Average = 11ms           

and you would hope lower than 30ms or so. Depending on your location.

edit: also something doesn't seem right with your mtr to netgate.com - ending up on a wixsite.. Oh its not netgate you did mtr too, its negate ;)

You seeing some high packet loss over twelve99 network - if things haven't changed they are backbone player. So could be a peering issue between your isp network and theirs? And while loss from hops along the way doesn't always mean a problem.. It could point to say congestion on that path, etc. That hop is answering the trace, but might be busy and not answering all them, etc.

[amrogers3]

@stephenw10 said in Experiencing and troubleshooting network latency issues?:

What does MTR show against 1.1.1.1 or 9.9.9.9?

I toggled the floating rule off.
Even more strange, I did a speed test at where the fiber is coming in and at the "Guest" interface on the pfsense router. Significant speed degradation.

My NICs are gigabit.

MTR results

[stephenw10]

You have any traffic shaping in play?

Check for errors/collisions in Status > Interfaces.

[amrogers3]

@johnpoz said in Experiencing and troubleshooting network latency issues?:

Oh its not netgate you did mtr too, its negate ;)

Ping timings are definitely not consistent.

I toggled that floating rule off. I am having some issues with speed decrease between the fiber and the pfsense interface.
I am not showing any increased CPU or memory issues. I am not sure what could be causing this issue.

[amrogers3]

@stephenw10 said in Experiencing and troubleshooting network latency issues?:

You have any traffic shaping in play?

Check for errors/collisions in Status > Interfaces.

0 errors and 0 collisions.
709Mbps to 53Mbps is wild
No traffic shapers in play

[stephenw10]

@amrogers3 said in Experiencing and troubleshooting network latency issues?:

709Mbps to 53Mbps is wild

Yup it is. It looks like a low level mismatch somewhere. Like something linked at 100M somewhere. The client device itself maybe?

[johnpoz]

@amrogers3 said in Experiencing and troubleshooting network latency issues?:

Ping timings are definitely not consistent.

those look pretty freaking solid to be honest.. Some 30ms away and only have like less than 3ms jitter.. that is pretty good.

[amrogers3]

@stephenw10 said in Experiencing and troubleshooting network latency issues?:

@amrogers3 said in Experiencing and troubleshooting network latency issues?:

709Mbps to 53Mbps is wild

Yup it is. It looks like a low level mismatch somewhere. Like something linked at 100M somewhere. The client device itself maybe?

That is interesting. That made me go check. The WAN is showing green and both the LAN (green) and GUEST (white) interfaces are flashing yellow, so that is not good. Both of these interfaces are built onto the mobo. The WAN interface is connected via PCI. This is weird and I can't think why it is happening. SMH. I double checked the board specs and the two ports are 2x Intel 82574L Gigabit Ethernet.
I used the same the same laptop with the same cable and connector to test both connections.

video

I connected my laptop to GUEST interface and it is auto negotiating 1000BaseT but the interface is still flashing yellow.

[amrogers3]

Disregard the above, I confused myself.
The interfaces are negotiating correctly at 1000BaseT. There is still a bit on a drop between the WAN and GUEST but not 709 to 53 bad.

WAN

GUEST

Is 150ms-200ms typical for DNS queries?

[Gertjan]

@amrogers3

We 'no details' the only answer answer possible is : you saw it happening, so that's normal.

I'll presume some details :
192.168.1.1 is your pfSense, and your doing a 'dig ' on the console on pfSense through localhost.
I presume you use the resolver, and not for example the forwarder (dnsmasq).
If you were asking for an answer for a network device that is local, like for example a LAN device that has a known host name in the resolver's cache, then 200 msec is a lot.
But if your resolver is resolving, and the resolvers cache was empty, then the final answer is knowing after a root server gave a TLD dns server, and the TLD server gave a domain name server, and the domain name server gave an final answer. If one of these was slow, or all of them ? then an initial request can take up some time.
Or, you resolver is forwarding over a TLS connection, and that server (like 1.1.1.1) was slow to answer because a temporary overload, it can take some time.
The good news is : ones the answer is known, it will get stored in the local cache and a subsequent request will go way faster, as no more upstream actions are needed.

Another example : about the raw local wire (LAN) and also upstream WAN speed : be sure your switches are really switches, and not some ancient hub type devices, as these can produce collision effects an the LAN segments.

When you 'dig', use a destination address, like

dig @192.168.1.1 google.com

If not, dig might chose the defayult protocol, and that's not IPv4 but IPv6, so it talks to the resolver over "::1" and a recent forum thread showed (for some still unknown reason) that "::1" was slow of didn't work, so the (FreeBSD pfSense) network stack switches to Ipv4 after some time, introducing a delay.

And then there is the final : "did you have all the bandwidth avaible at that very moment" ?
And even if your uplink or downlink wasn't full, your ISP does share your connection with the entire neighborhood, so what was the neighbor doing at that moment ?
And so on.

Most of us solve this problem the non technical way : they throw more money on it, so they obtain a vague promise from the ISP that you have 'more bandwidth'. remember your phone getting 5G = multiple gibabits / sec on your phone ? Did you test that recently ?

edit : I recently encountered a comparable question, and after some digging I found a firewall (not pfSense) with "thin-foiled settings / rules", where DNS was 'UDP to port 53' only.
That as true in the past, and this changed somewhere in the last century, as DNS packets became to big (DNSSEC will be the reason) so it started to use TCP. But that was blocked. Because AI told so.

And let me add (with some aluminum over my head) : ISPs can do 'things' with DNS ..... because everybody want your DNS traffic.

[Eric_Mann]

Those DNS times are way above normal (should be under ~20-30ms). Try dig google.com @1.1.1.1 vs your current resolver to see if it's your DNS server that's slow or a network path issue — also check mtr to the resolver IP for packet loss/routing problems. If cached queries (dig run twice in a row) are also slow, the issue is upstream at your resolver rather than local config.