pfSense using unreasonable amount of bandwidth while idle
-
Yeah your going to have to watch your log.. Keeping in mind that every time a query is done if it has to open a new tls connection that is going to be quite a bit of overhead, and if doing it to multiple servers and for multiple queries that can add up.
So just took a look at my top "talkers" for dns - and noticed I had left a unbuntu vm running.. Guess what it likes do do a lot!! check for connectivity-check.ubuntu.com, not only for A but also for AAAA, and then guess what it does it also checks for connectivity-check.ubuntu.com.local.lan - because local.lan is my local suffix.
Its making these every few minutes if not seconds - just incase you know there is a captive portal it needs to auth too.. While a couple of packets for dns normally not a big, deal if your using doh or tls that data is amplified X.. Now do that for how many devices, how many queries.. So yeah you could for sure raise the grass level of your traffic going back and forth even when your thinking your network is idle..
In the last 24 hours there have been over 1000 queries for these fqdn.
What else is being looked for? That is why log of your queries will help, or if you were not encrypted you could just sniff the wan traffic and see what was being asked for since your dns query would be in the clear.
Now in my case that .local.lan won't be sent upstream, since I set unbound to not do that by changing the type of zone... But out of the box transparent mode when asked for stuff that is not in your zone, it will ask upstream for that..
So you have figured out that there seems to be a lot of what you would consider idle traffic to your doh servers.. So you need to figure out what is being asked, so you can just live with it, or make some changes to reduce that traffic or eliminate it completely.. My example, I turned off that check in ubuntu - since have zero need of it.. I don't run capture portal, etc.
-
@johnpoz Good points--such things can certainly do a seemingly excessive number of lookups. However, that assumes the traffic is originating from inside the LAN. In this case, I can tell just based on traffic volume that this traffic is originating from the pfSense itself. While it is sending/receiving a massive number of requests, the LAN and OPT ports are seeing almost no traffic at all.
-
Your local traffic would be small udp dns queries of a couple of bits.. You get an amplification when your using doh..
Maybe its some package on pfsense doing queries for say IPs hitting its wan, do you have say IPS running
This is why you need to log the queries.. You will see be it pfsense or something else doing the queries..
look at the size of your exchanges in your few packets you sniffed.. Over a time frame of like what .2 seconds.. Now multipy that by hundreds of queries or 1000's even..
To be honest using doh your doing a dns amplification attack against yourself - to hide that your going to google.com from your ISP.. I don't get it... So you make your dns slower, you now hand over every thing you ever ask for to google and quad nine, etc. on a silver platter.. Because your trying to hide that you ask for xyz.com from your ISP?? Its crazy.. But because company xyz says hey you can trust us - hand all your dns to us we won't do anything bad with that info "we promise" ;)
It's nuts!! I will just resolve myself thank you very much..
-
You get an amplification when your using doh..
That makes some sense, but how would a few b/s become 70 Kb/s upload and 400 Kb/s download? Overhead? Sure, there's some, but that's ridiculous. Meanwhile, in the middle of the day while the pfSense has devices behind it actually in use, this bandwidth consumption decreases considerably. It's only while everything is idle that my problem occurs. Just for laughs, I could shut off DoH for a bit and see, but this doesn't seem to explain the issue. If it were DoH at fault, I would be seeing this "overhead" increase, or at least remain constant while devices behind it are in use. Instead this "overhead" vanishes entirely while its devices are in use.
do you have say IPS running
I have an IDS (SNORT) running on the pfSense. It is passive.
So you make your dns slower, you now hand over every thing you ever ask for to google and quad nine, etc. on a silver platter.. Because your trying to hide that you ask for xyz.com from your ISP?? Its crazy..
Fair point, but that's why the DNS provider needs to be chosen carefully (I hope to switch to OpenDNS soon, but they do not have standardized DoH support yet). I would never use Google, for example, because their business model is based on selling private data. However, local ISPs are also selling private browsing data, so if can make it just a little bit harder for them to farm my internet traffic by sending it to someone I trust, I'll do it (unless, of course, it is causing a malfunction). From a user experience perspective, you can't even see the difference in speed with DoH running.
Tell you what, I will watch it as is for now, but next time I see this problem, I will check my query log, then try disabling DoH. It can't hurt to try (other than the cost of private data).
-
@CyberMinion said in pfSense using unreasonable amount of bandwidth while idle:
so if can make it just a little bit harder for them to farm my internet traffic
Why? I mean really - the genie is already out of the bottle.. Do you really think your hiding from anyone.. You are worried that your isp knows you like to visit xyz domain. You understand they know you are going there anyway by the traffic your sending from IP to IP, if they really want to look.
Do you only use cash, do you only use burner phones. Do you not drive a car because they can track your license plate. What about your cell phone, you know they know exactly where your at all times, be it gps or just what cell tower your talking to. And your text message - do you encrypt them all? Do you not use reward cards.. The amount of info given away by your typical day to day usage.. And people are worried about your isp knowing you did a query for xyz.com is just freaking ridiculous.. Your just making your experience suck more is all your doing ;)
For some reason you think sending all your dns to company X is better than just forcing your isp to actually sniff your traffic for your dns queries because your sending them all over the planet via resolving. It makes no sense. And hey its worth hiding it from my isp so much that I want my dns to suck by slowing it down and forcing more traffic to be used..
I just don't get it..
I have an IDS (SNORT) running on the pfSense. It is passive
Does not mean its not trying to resolve every IP it sees.. It sees ip 1.2.3.4 hit your wan on port X, so it tries to look up via ptr that IP.. So it sends query to 9.9.9.9 via doh.. So via that 2 bits hit your wan on port X, now you have amplified the traffic trying to find out the ptr for 1.2.3.4
Here is the thing - the data that is you has already be monetized.. You sending to company X, because you "trust" them more is just giving another company more info about you, in an attempt to hide it from company Y - that you actually pay to connect you ;) And if they could somehow make your internet better ok sure, but really all it does is slow down your internet, and make troubleshooting issues more difficult..
-
DoH in a localy administered lan, when YOU are the admin is absurd.
DoH is good for journalists behind the Great China Wall, visitors to North Korea and the like.
There who do you trust is often a life or death decision.
When you have a local resolver like unbound I see zero value using anything else.
One can argue that by doing that we put too much strain into root servers. But with todays traffic engineering capabilities and available load balancing methods this is rather trivial.
And the money payed for top level domains fees has sky rocketed too, so a decent root dns infrastructure is expected.As for the traffic pattern, I still find it hard to be like this at an almost constant rate just by resolving. I mean, there should be some caching somewhere (and negative too).
Gut feeling, this is some kind of bug inside the rather new DoH, or some other kind of bug, upstream.I would disable all kinds of upstream queries by port forwarding everything to pf unbound. (and blocking outbound access to port 53 and 853 for ipv6 queries too)
Put unbound to resolve without any forwarder.
The only manual thing that needs to be done on clients is disabling DoH. -
@netblues said in pfSense using unreasonable amount of bandwidth while idle:
DoH in a localy administered lan, when YOU are the admin is absurd.
QFT!!!
The only manual thing that needs to be done on clients is disabling DoH
Agreed, and that is a serious problem! It should never be opt out, it should be opt in.. Its BS plain and simple that make a choice of sending my dns to them without explicit permission from the user. My browser had a local dns working just fine, until you thought it was better to send my dns to you, etc..
I shouldn't have to setup canary domains or make or click don't in the browser.. I should have to on purpose choose to send my dns via doh.
-
Ok, so the issue is back. I'm see the same two queries over an over again:
(I masked the LAN's name...pardon that)It seems to be alternating between bursts of 066.136.238.176 queries, and 066.136.237.192 queries.
I tried disabling DoH, and applying this update to the DNS resolver...no change. These are requests for AAAA records; I previously disabled IPv6 on the pfSense, but due to the aforementioned issue of config changes being lost 12-48 hours after being applied, IPv6 is enabled again. I might try disabling it, to see what that does.
Why? I mean really - the genie is already out of the bottle
I didn't mean to drag us into the "Privacy is a right" vs "Privacy is gone, give up, there's no hope so just let them have it all. You have nothing to hide, right?" argument. You might be surprised the lengths I go to which some would consider unreasonable. Some degree of privacy is still attainable, if you are willing to work for it.
Does not mean its not trying to resolve every IP it sees.. It sees ip 1.2.3.4 hit your wan on port X, so it tries to look up via ptr that IP
I don't think SNORT is resolving IPs. I looked through it's config, and found nothing in that regard. It certainly isn't showing resolved info to me in the event log either. I could be missing something though.
-
You have those in a alias? Somthing with a rogue . or digit causing the firewall to try to resolve an IP as an FQDN?
-
Yeah what exactly is trying to be resolved there.. Its not a PTR, and doesn't even look like a valid IP? 066? But what hidden there in the tld?
is ti adding your local domain as the tld?
-
It's appending the local domain after failing without it.
Check the Resolver logs in pfSense for filterdns entries. That looks exactly like it's a bad alias entry.
Steve
-
So those IPs via ptr are in the dsl.ltrkar.swbell.net domain... I take it thats your isp?
-
@CyberMinion said in pfSense using unreasonable amount of bandwidth while idle:
I don't think SNORT is resolving IPs. I looked through it's config, and found nothing in that regard. It certainly isn't showing resolved info to me in the event log either. I could be missing something though.
Correct, neither Snort nor Suricata do anything with automatic DNS lookups. There is not even the required client code within either package (not in the binary portion and not in the GUI portion).
The IDS/IPS packages only cause a DNS lookup via two methods. The user manually clicks the little "i" icon next to an alert on the ALERTS tab to perform a reverse lookup on the IP. That lookup is actually handed off to the firewall for the DNS task. The other time the packages would use DNS is when the periodic rules update cron task executes and calls
curlwith a URL to download the rules files. That happens at most twice per day. -
rogue . or digit causing the firewall to try to resolve an IP as an FQDN?
Maybe. I was wondering what kind of a lookup that is.
@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:
Yeah what exactly is trying to be resolved there.. Its not a PTR, and doesn't even look like a valid IP? 066? But what hidden there in the tld?
I think maybe it is an IP with its octets inverted. So in this case, 176.238.136.066. (That doesn't have a DNS record)
is ti adding your local domain as the tld?
Yes, the local domain is showing as the TLD...that is what I masked. I've seen this a few times before on my network, and wondered why.
I have an internal DNS resolver (Pi-Hole) which uses pfSense as my upstream resolver. PiHole has not seen any queries for these IPs in the past 30 days, so they are coming from the pfSense itself.@stephen10
Check the Resolver logs in pfSense for filterdns entries. That looks exactly like it's a bad alias entry.
What exactly should I be looking for? All I'm really seeing is that I published above, repeating over and over.
So those IPs via ptr are in the dsl.ltrkar.swbell.net domain... I take it thats your isp?
No, that is not my ISP.
Correct, neither Snort nor Suricata do anything with automatic DNS lookups
Good to know.
The IDS/IPS packages only cause a DNS lookup via two methods. The user manually clicks the little "i" icon next to an alert on the ALERTS tab to perform a reverse lookup on the IP
I haven't done that any time recently
The other time the packages would use DNS is when the periodic rules update cron task executes and calls curl with a URL to download the rules files. That happens at most twice per day.
That occurred to me as a possibility. It currently performs this a 2am, and if I notice my problem, it will be in the morning. Sometime throughout the late morning or early afternoon, it stops.
-
Something like this:
Aug 4 22:04:38 filterdns Adding Action: pf table: test_alias host: 78.89.1000.25 Aug 4 22:04:38 filterdns Adding host 78.89.1000.25 Aug 4 22:04:38 filterdns failed to resolve host 78.89.1000.25 will retry later again. -
I had this question come up from a customer.. Turns out he was VPN'd into the site to watch WAN traffic graphs. Is there the possibility that someone is looking at the WAN remotely?
-
He says not. I thought it could easily be a VPN thought the traffic would be more symmetric if it was an external user pulling external files hairpinned.
Steve
-
Something like this:
I'm not seeing any logs that look like that...would this be under Status/System Logs/System/DNS Resolver?
Is there the possibility that someone is looking at the WAN remotely?
Shouldn't be, unless something is compromised. The exterior NAT router, and the pfsense behind it both have VPN services turned off. All ports are closed on the exterior SOHO NAT router, and UPnP is disabled there. On the pfSense behind it, UPnP is actually enabled (oops!) but in past experiments, I found that the UPnP requests sent upstream by one of my devices only reached the pfSense, where they were honored (at present, no UPnP ports are opened on pfSense). On the edge router, no ports were opened while it had UPnP enabled. Anyway, the point is, pfSense currently has UPnP enabled, but unless there is a way to get the edge router to open ports while its UPnP is disabled, there should be no option to open an unsolicited connection from the outside, even if internal malware was trying to open ports. I will disable UPnP on pfSense soon, but I don't want to change too many things at once while troubleshooting.
P.S. Thanks for sticking with me on this issue! Much appreciated!
-
Yes, if you were hitting that it would be in the resolver log.
-
Yes, if you were hitting that it would be in the resolver log.
Okay, well I don't see that going on right now, but next time I notice the issue, I will check.
