DNS 8000+ms, troubleshooting help
-
When running two applications, App[A] and App[B], App[B] communicates with App[A] through loopback address 127.0.0.1 on host.
As soon as App[B] starts:
-DNS has 8000+ms latency, internet un-useable
-However, Iperf between all machines on network shows full speed
-Download from internet also show full speed
-Dashbaord shows low utilization cpu 3%, mem 2%, 10-30Mb/s bandwidth of 1Gbs.
The apps require forwarding of 3 ports, which has been done correctly.How do I trouble shoot and fix this issue it is really doing my head in?
-
@srytryagn said in DNS 8000+ms, troubleshooting help:
-DNS has 8000+ms latency, internet un-useable
Where did you come up with this number? So you are doing a dig or nslookup in debug, or some other tool.. And you see what when you query what?
Anything cached should be a couple of ms at most..
;; QUESTION SECTION: ;www.google.com. IN A ;; ANSWER SECTION: www.google.com. 1476 IN A 142.250.190.36 ;; Query time: 2 msec ;; SERVER: 192.168.3.10#53(192.168.3.10)Even if you query something that not cached.. It should only be a few ms
;; QUESTION SECTION: ;www.cnn.com. IN A ;; ANSWER SECTION: www.cnn.com. 3600 IN CNAME cnn-tls.map.fastly.net. cnn-tls.map.fastly.net. 3600 IN A 146.75.79.5 ;; Query time: 62 msec ;; SERVER: 192.168.3.10#53(192.168.3.10)Where are you seeing 8 some seconds to resolve something?
-
Where are you running these apps?
Where is DNS slow? For all devices on LAN?
-
@johnpoz
ping google.com
ping 1.1.1.1
ping 8.8.8.8from each machine.
3500-8000ms latency
Please let me know what can I provide from my end to help narrow down the issue to troubleshoot what exactly is going on and how I can fix it ? It is KILLING ME.
-
All networks, not just the machin or Lan that it is running on.
I can download full speed, and communicate internally at full speed, but internet web browsing is killed on every network.
What can I provide from my end to help narrow down the issue to troubleshoot what exactly is going on and how I can fix it ?
-
@srytryagn said in DNS 8000+ms, troubleshooting help:
3500-8000ms latency
pinging an IP has zero to do with dns..
pinging an google.com and you get what back, 8000 ms would time out on pretty much any client.. Did you do say a dig google.com +trace on pfsense?
23.09.1-RELEASE][admin@sg4860.home.arpa]/root: dig google.com +trace ; <<>> DiG 9.18.16 <<>> google.com +trace ;; global options: +cmd . 48654 IN NS j.root-servers.net. . 48654 IN NS d.root-servers.net. . 48654 IN NS h.root-servers.net. . 48654 IN NS f.root-servers.net. . 48654 IN NS e.root-servers.net. . 48654 IN NS g.root-servers.net. . 48654 IN NS k.root-servers.net. . 48654 IN NS b.root-servers.net. . 48654 IN NS i.root-servers.net. . 48654 IN NS l.root-servers.net. . 48654 IN NS a.root-servers.net. . 48654 IN NS c.root-servers.net. . 48654 IN NS m.root-servers.net. . 48654 IN RRSIG NS 8 0 518400 20240129050000 20240116040000 30903 . Unu/eARIHWXUOk6Wmw255AaU63iDfnTFqgKnWr12JGDyJyLNJZdiF9yq FIjRsx+SaSsQy+ixUaZzCvAS2o6hjD7FmAL7bAka+gFjcvPsXhgOdZk6 K74eEPfBKF8GaHeSteyoJyb34X+bQv1CV4/M/G/bKJZUdk9cGowDJSLJ Uo/73bz09Zf6JglJdziQFzaFlzKApdY28oMLME5oyaWuLnF0tgLH463Z 1IqEjbhTE7PegN5fsXJv44ONOfv7KqxcVsfYEqtfVO67uF1zLcnyBGrJ 2tc0mC6uvxx6rd0cS3vPqkaM/sVmBVhVIhDuYnvPllrOptU3IwMoTYW0 FDzxGg== ;; Received 525 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms com. 172800 IN NS a.gtld-servers.net. com. 172800 IN NS b.gtld-servers.net. com. 172800 IN NS c.gtld-servers.net. com. 172800 IN NS d.gtld-servers.net. com. 172800 IN NS e.gtld-servers.net. com. 172800 IN NS f.gtld-servers.net. com. 172800 IN NS g.gtld-servers.net. com. 172800 IN NS h.gtld-servers.net. com. 172800 IN NS i.gtld-servers.net. com. 172800 IN NS j.gtld-servers.net. com. 172800 IN NS k.gtld-servers.net. com. 172800 IN NS l.gtld-servers.net. com. 172800 IN NS m.gtld-servers.net. com. 86400 IN DS 19718 13 2 8ACBB0CD28F41250A80A491389424D341522D946B0DA0C0291F2D3D7 71D7805A com. 86400 IN RRSIG DS 8 1 86400 20240129170000 20240116160000 30903 . yD+8fjXVVXWOfim9yMP2BlCyOZgIuGyBgih+SVMGb/MRuM239Zww/XWZ r5hPkZHStx3xC9zcgh3/XUuK2DV2Ozdkho9HkENUGzTONZoq3AMfGJUU KEfXLcQvek4QPtWiZUHp1evh8gPERSbva825eZ0xYyZycjsl4yb+9ZzW jgfkGvOrJ60fFuup8QGxxVY9UhRMHHs1ssURm+h8WfUY0gb/hSBQtvZy BcA7i1Ioy+5Ii9egyctlsFpP8b9ewVCEMxWwGhjG3sshvG8iJMNfT9yQ mtbg6QPuHERXHW78bgqrlqWebwM1w6HzTVfyMQyklyzF1qp6sIrjYZPS uXY3QA== ;; Received 1170 bytes from 199.7.91.13#53(d.root-servers.net) in 9 ms google.com. 172800 IN NS ns2.google.com. google.com. 172800 IN NS ns1.google.com. google.com. 172800 IN NS ns3.google.com. google.com. 172800 IN NS ns4.google.com. CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN NSEC3 1 1 0 - CK0Q2D6NI4I7EQH8NA30NS61O48UL8G5 NS SOA RRSIG DNSKEY NSEC3PARAM CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN RRSIG NSEC3 13 2 86400 20240121052619 20240114041619 46171 com. DBANPdeIUsydaUZpyTuxMq58//eu9Q3V2rkoU/PIPksDGoF7/t3VvpB4 l5HatkMxZjL4S4yjH5BupWjxfq8aYw== S84BKCIBC38P58340AKVNFN5KR9O59QC.com. 86400 IN NSEC3 1 1 0 - S84BR9CIB2A20L3ETR1M2415ENPP99L8 NS DS RRSIG S84BKCIBC38P58340AKVNFN5KR9O59QC.com. 86400 IN RRSIG NSEC3 13 2 86400 20240122054713 20240115043713 46171 com. nGSRGJXEktzQR7fx4WSdzn0NFkJ8D1eMUM70VEBB1QMJcwSXrhmVGk6w 2oGbdbQbC2hY/l4tuTxQzIv0nxRTuQ== ;; Received 644 bytes from 192.54.112.30#53(h.gtld-servers.net) in 27 ms google.com. 300 IN A 142.250.191.174 ;; Received 55 bytes from 216.239.34.10#53(ns2.google.com) in 19 ms [23.09.1-RELEASE][admin@sg4860.home.arpa]/root:What does your output of dns lookup in the gui for something look like?
I take it your just resolving which is unbound out of the box, or did you setup tls forwarding? Or normal forwarding?
-
@johnpoz Thanks for pointing that out.
-(Slow)I type google.com into browser on every connected machine, they timeout.
-(Slow)Ping google.com it says 8000ms.
-(Normal)Iperf between machines 960Mb/s
-(Normal)Download linux mint 900Mb/sHave I misidentified what the issue is as DNS and how may I narrow it down correctly to repair it.
-
Yeah that seems like just latency in general. Unless pings to something else returns normally.
What does the pfSense gateway monitoring show?
What do pings from pfSense itself show?
-
DNS with apps off:
DNS with apps on:
Ping apps off:
Ping with apps on:
Ping from a local box with apps on:
-
@johnpoz Default setup just added some DNS alternatives as you can see in above images.
-
@stephenw10 Shown in previous post.
Gateway monitor ? lmk what you wand to see so I may post it. -
It appears that the applications are putting the network underload. Try doing a packet capture on the lan interface to see what is happening on the network.
-
@s_serra CPU 3% mem 1-3%, bandwidth is wayyy below my 1Gb/s capability.
What specifically shall I check ? -
@srytryagn A loop may be occurring.
-
@s_serra You may be right about that and that (a Loop) would make sense. I have run a packet capture on the lan running the apps. I dont know how to diagnose the loop nor how to troubleshoot it or where to begin. Would you please kindly point me in the right direction.
-
Simple network topology FYI.
-
It depends on what the application is doing with the network. Can you tell me what type of application it is and what this application needs from the network? You can set the IP address in each application's packet capture to see what happens in each one.
-
@s_serra It is a blockchain application so RPC connection, in particular a node (p2p traffic hence the port forward) and a farmer which stores data to a local drive and must prove to the network that it is stored on the drive.
It requires 30333, 30433, 30533 TCP+UDP to be forwarded to the box.
The Node app and Farmer app speak each other on same machine on loopback.
I ran a 1000 data point packet capture on that machine running the apps, but have no clue how to diagnose a loop from what is infront of me.
Could you explain how & what you mean by set IP address in each applications packet capture, doing packcapture in pfsense only allows a machine IP not a specific application ?
-
Since both applications are running on the same machine, I assume they have the same local IP address running on different ports. Put the local IP address of the machine where the applications are in the packet capture to see what is happening. Do ports 30333, 30433, 30533 need to be exposed to the Internet?
-
@s_serra I did exactly that, ran packet capture of machine IP address where the apps are running.
Yes 30333, 30433, and 30533 are exposed to the internet, in pure nat point to app machine, with requisite outbound rules applied on wan/NAT.
