DNS 8000+ms, troubleshooting help

srytryagn

@johnpoz File enclosed with No apps and AppA + AppB running.

I have a bunch of other captures, no port fwd, port fed, start AppA, Sync AppA, Running AppA, AppA + AppB.....

Please let me know what the file points to being wrong w/ my setup and how I may repair it, thanks.

johnpoz

@srytryagn could you give some insight to what IPs you changed to what..

Your one with stuff off I see 993 packets in a total of 228 ms.. between 10.50.18.154 and 10.50.245.28, that is a lot of packets between 2 devices in short amount of time.. For what I would assume is network not doing anything.. Is one of those IPs your public, one of one these node devices.. Thought you said they were off?

Then in this other one with on, Its hard to follow because there is 166 different conversations in it all with this 192.168.86.26, what is this 192.168.86.26? in a total sniff of 1.4 seconds?

srytryagn

@johnpoz

In Apps off mode:
10.-50.18.154 = PC on network 2
10.50.245.28 = KVM over IP connected to PC above

In Apps on mode:
192.168.86.26 = PC on network 1 running the apps

johnpoz

@srytryagn said in DNS 8000+ms, troubleshooting help:

10.50.245.28 = KVM over IP connected to PC above

Well is the only traffic really in that sniff.. So that sniff is pretty useless..

So your saying with that 2nd pcap - is when your network is dead?

srytryagn

@johnpoz

labelled -> pcap_AppA_AppB_anon.pcap = Apps on internet DEAD

labelled -> pcap_apps_off_Port_Fwd_anon.pcap = Apps off, internet and everything else working normal.

Should I do a longer pcap with a particular configuration to make it more useful for analysis ?

johnpoz

@srytryagn well your sniff where you said stuff is broke, I see nothing but this 192.168.86.26 talking to bunch of stuff.. I don't see anything else.. So its hard to say well dns was delayed or arps failed or lots of retrans.. There are a few retrans, but nothing out of the ordinary..

Looking at that sniff I don't see anything wrong at all.. But then again there isn't much other traffic.. And the small about there is, its in the middle of something and don't see any problems.. No retrans for example.. If your network had huge delays on it, you would see loads and loads of retrans when something didn't get an answer fast enough, etc..

srytryagn

@johnpoz Yup, I cant sort out what is going wrong, many folks have confirmed being able to run the apps without any issue.

What do you reckon ?

stephenw10

Seems very likely one or both 'apps' are misconfigured and flooding traffic that should never leave the host IMO.

johnpoz

@srytryagn said in DNS 8000+ms, troubleshooting help:

What do you reckon ?

Can't reckon anything from those sniff.. One is just kvm traffic, and the other is an IP talking to lots of other stuff, but nothing that is insane amount of traffic.. no errors seen, no retrans seen that are anything of any significance.. Don't even see any other traffic, no broakcast, no flood of multicast, Then again is 1.4 seconds worth of data.. So wouldn't expect to see anything - unless there was some sort of flood of traffic..

Maybe a longer sniff - while your trying to do stuff that says failing.

srytryagn

@johnpoz Sure will run a longer sniff and will try to connect to some websites that will fail.

Any point in sniffing the Wan side or anything else while I am working on this ?

johnpoz

@srytryagn not really - you say your devices when turned on are what cause the problem.. So unless when your devices phone home they are doing a volumetric ddosing attack against your wan? The traffic has to go through your lan to get anywhere.

Guess that is a possibility, and if all the traffic was just dropped it wouldn't show really on pfsense as anything.. But your internet could become crazy slow..

I guess if your connecting to some sort of swarm or something - and you get bombed from 100s or thousands of devices trying to talk to your wan? And they are bombing you with large UDP packets or something?

Couldn't hurt to see normal taffic flow on your wan, and then sniff after you turn on your stuff and you say stuff fails/slow, etc.

stephenw10

If that is the case then DNS and/or ping response from pfSense itself to something external would also be affected. I think we asked about that but I may have missed the reply.

srytryagn

This post is deleted!

srytryagn

@johnpoz pcap enclosed w/ couple of attemtps at reaching out to websites (they all failed). extralongpcap_anon.zip

Please let me know what mapping IPs/descriptions are helpful to know if at all.

srytryagn

@stephenw10 Can I ping from the console in web gui to test that ?

stephenw10

In the gui use Diag > Ping or Diag > DNS Lookup.

But better to test from the CLI via SSH if you can.

johnpoz

@srytryagn you can also dump the data from the file.. Just really need the headers and such.. but just to split it up to post if you have no where else to host it.. splitcap comes to mind.. https://www.netresec.com/?page=SplitCap

You should also be able to just zip it and split it up that way, 7z can do that really easy.. There is also a truncate option in that wrangler, which can cut out the data of large packets, we just really need to see the headers.

I just did a quick pcap of wan traffic and fired off a speedtest to make sure it was big.. 226MB, after truncated it down to 18MB.. so something like that could maybe shrink your 7MB file down to size to be able to upload.

srytryagn

@johnpoz I deleted post asking about split, it shrunk in wrangler and I posted it above.

johnpoz

@srytryagn ok what is this? 192.168.226.58 trying to talk to stuff and getting told that port is not reachable, etc.

Where exactly did you sniff this - on the lan interface where your bad guys are? On pfsense.

So for example in this session.. I see that a syn,ack was sent to

that 226.58 then you gavve what looks like answers back, but then see that seq=29.. that makes no sense they should be in order.. And clearly that 43.x guy never got answer because he is retrans his syn,ack..

Where exactly was this sniff taken? On pfsense interface?

edit: what exactly are you trying to run - is this something I could try and duplicate?

srytryagn

@johnpoz
Yes on PC on network 2 running both Apps.
Pcap is taken from within pfsense webgui (pointed to problematic Lan).
ICMP is me pining google.com from that same pc.

I really appreciate your willingness to run it on your end and would happily walk you though how to do that. You will need a wallet address (or donation address), and a partition NVME or SSD with about 200Gb free to run the node and farmer.

How do want to proceed ?