pfSense using unreasonable amount of bandwidth while idle
-
@CyberMinion said in pfSense using unreasonable amount of bandwidth while idle:
I don't think SNORT is resolving IPs. I looked through it's config, and found nothing in that regard. It certainly isn't showing resolved info to me in the event log either. I could be missing something though.
Correct, neither Snort nor Suricata do anything with automatic DNS lookups. There is not even the required client code within either package (not in the binary portion and not in the GUI portion).
The IDS/IPS packages only cause a DNS lookup via two methods. The user manually clicks the little "i" icon next to an alert on the ALERTS tab to perform a reverse lookup on the IP. That lookup is actually handed off to the firewall for the DNS task. The other time the packages would use DNS is when the periodic rules update cron task executes and calls
curlwith a URL to download the rules files. That happens at most twice per day. -
rogue . or digit causing the firewall to try to resolve an IP as an FQDN?
Maybe. I was wondering what kind of a lookup that is.
@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:
Yeah what exactly is trying to be resolved there.. Its not a PTR, and doesn't even look like a valid IP? 066? But what hidden there in the tld?
I think maybe it is an IP with its octets inverted. So in this case, 176.238.136.066. (That doesn't have a DNS record)
is ti adding your local domain as the tld?
Yes, the local domain is showing as the TLD...that is what I masked. I've seen this a few times before on my network, and wondered why.
I have an internal DNS resolver (Pi-Hole) which uses pfSense as my upstream resolver. PiHole has not seen any queries for these IPs in the past 30 days, so they are coming from the pfSense itself.@stephen10
Check the Resolver logs in pfSense for filterdns entries. That looks exactly like it's a bad alias entry.
What exactly should I be looking for? All I'm really seeing is that I published above, repeating over and over.
So those IPs via ptr are in the dsl.ltrkar.swbell.net domain... I take it thats your isp?
No, that is not my ISP.
Correct, neither Snort nor Suricata do anything with automatic DNS lookups
Good to know.
The IDS/IPS packages only cause a DNS lookup via two methods. The user manually clicks the little "i" icon next to an alert on the ALERTS tab to perform a reverse lookup on the IP
I haven't done that any time recently
The other time the packages would use DNS is when the periodic rules update cron task executes and calls curl with a URL to download the rules files. That happens at most twice per day.
That occurred to me as a possibility. It currently performs this a 2am, and if I notice my problem, it will be in the morning. Sometime throughout the late morning or early afternoon, it stops.
-
Something like this:
Aug 4 22:04:38 filterdns Adding Action: pf table: test_alias host: 78.89.1000.25 Aug 4 22:04:38 filterdns Adding host 78.89.1000.25 Aug 4 22:04:38 filterdns failed to resolve host 78.89.1000.25 will retry later again. -
I had this question come up from a customer.. Turns out he was VPN'd into the site to watch WAN traffic graphs. Is there the possibility that someone is looking at the WAN remotely?
-
He says not. I thought it could easily be a VPN thought the traffic would be more symmetric if it was an external user pulling external files hairpinned.
Steve
-
Something like this:
I'm not seeing any logs that look like that...would this be under Status/System Logs/System/DNS Resolver?
Is there the possibility that someone is looking at the WAN remotely?
Shouldn't be, unless something is compromised. The exterior NAT router, and the pfsense behind it both have VPN services turned off. All ports are closed on the exterior SOHO NAT router, and UPnP is disabled there. On the pfSense behind it, UPnP is actually enabled (oops!) but in past experiments, I found that the UPnP requests sent upstream by one of my devices only reached the pfSense, where they were honored (at present, no UPnP ports are opened on pfSense). On the edge router, no ports were opened while it had UPnP enabled. Anyway, the point is, pfSense currently has UPnP enabled, but unless there is a way to get the edge router to open ports while its UPnP is disabled, there should be no option to open an unsolicited connection from the outside, even if internal malware was trying to open ports. I will disable UPnP on pfSense soon, but I don't want to change too many things at once while troubleshooting.
P.S. Thanks for sticking with me on this issue! Much appreciated!
-
Yes, if you were hitting that it would be in the resolver log.
-
Yes, if you were hitting that it would be in the resolver log.
Okay, well I don't see that going on right now, but next time I notice the issue, I will check.
-
@stephenw10 said in pfSense using unreasonable amount of bandwidth while idle:
failed to resolve host
I'm still not seeing any of the lines you mentioned in the log, just a whole lot this going on:
(network hostname removed to protect the guilty)I see several pages of this for each second that passes.
-
@CyberMinion said in pfSense using unreasonable amount of bandwidth while idle:
I see several pages of this for each second that passes.
Hello!
Are you running any python modules in unbound?
John
-
Be it that is causing that much bandwidth or not.. You got something doing A queries for what is suppose to be an IP it looks like..
Figure out what is doing asking for that.. Pfsense out of the box is not going to query for that..
And doing a suffix search, which is just local? If so why are you hiding it?
-
Are you running any python modules in unbound?
None that I am aware of...unless maybe SNORT is running unbound. I can't find any mention of that in the config, but I think it might use Python. EDIT: Oh duh, pfBlocker uses Unbound.
All I know is that it is originating from the pfSense. Here are the packages I have installed. There's always the possibility of a misconfiguration on one or more of them.
And doing a suffix search, which is just local? If so why are you hiding it?
I'm not sure what kind of a lookup it is doing, but I'm hiding that part of it just because it is the internal hostname. Names are hidden to protect the guilty.
Just suppose what I covered is "MyLAN" or "MyNetworkName"I don't need SNORT, so maybe I will try shutting that off for a bit. It shouldn't be doing any lookups of its own, but it's worth a try. If that doesn't work, maybe I'll try shutting down pfBlocker. I don't want to do that, but I can if needed. These are the two main packages I have running here.
Is there a way to get details on resource utilization of each process? That way, I might be able to correlate the network traffic with a specific pid.
-
@CyberMinion said in pfSense using unreasonable amount of bandwidth while idle:
I'm not sure what kind of a lookup it is doing,
Says right there what its doing, its doing a A query.. but 066.159.140.072 is very ODD A query and yeah it would return a NX since there is no .072 TLD, and same for any local domain, public dns not going to know anything about those..
Are you doing any direction of dns to loopback, ie trying to redirect things from using external dns?
For example.. here is pfsense trying to check for updates
Aug 12 13:04:47 unbound 52178:3 info: 127.0.0.1 files01.netgate.com. A IN NOERROR 0.000000 1 53 Aug 12 13:04:47 unbound 52178:3 info: 127.0.0.1 files01.netgate.com. A IN Aug 12 13:04:47 unbound 52178:0 info: 127.0.0.1 files01.netgate.com. AAAA IN NOERROR 0.040565 0 65 Aug 12 13:04:47 unbound 52178:0 info: 127.0.0.1 files01.netgate.com. AAAA IN Aug 12 13:04:47 unbound 52178:3 info: 127.0.0.1 files01.netgate.com. A IN NOERROR 0.036122 0 53 Aug 12 13:04:47 unbound 52178:3 info: 127.0.0.1 files01.netgate.com. A IN Aug 12 13:04:47 unbound 52178:0 info: 127.0.0.1 _https._tcp.firmware.netgate.com. SRV IN NOERROR 0.195872 0 128 Aug 12 13:04:47 unbound 52178:0 info: 127.0.0.1 _https._tcp.firmware.netgate.com. SRV INOff the top of my head, I am not sure how you would find out what process is asking loopback for query?
-
@johnpoz Nothing should be mapped to the loopback address. I have given two internal IPs local domain names in my resolver, and pfBlocker returns a dummy internal IP for "blocked" DNS requests (192.168.5.1). Other than that, it should be operating normally.
-
Where does pfsense point to? It points to loopback (127.0.0.1) out of the box, any process running on pfsense would be using that for dns queries.
-
@johnpoz Ah, I see.
-
To be honest if what you want to make sure pfsense queries are using tls to talk to 1.1.1.1 or quad 9.. You have to do the forwards in the unbound resolver options box.. Because when setup this way if for some reason pfsense own dns client goes to ask 1.1.1.1 or quad9 it will just do a normal dns query and will not use tls.
So from your log you show that loopback is asking for that weird A query.. Out of the box pfsense would not ask for that.. It only checks to see if there is any updates. From like what I showed you, or if it trying to resolve something via some other process running on it, like a package.. Filterdns would ask for stuff if you setup any aliases, etc.
It would look to download bogon, etc..
The trick is trying to figure out where that query is coming from.. Off the top I do no know a way to monitor all internal processes for what they query for..
That sort of tracking has always been tricky no matter the os, be it windows, linux or a bsd..
-
You have to do the forwards in the unbound resolver options box..
I am, but before I enabled DoH, I added these DNS providers to the standard resolver, to use them in the clear (both are much faster than my ISP's DNS). Since then, I've run tests and do see DoH traffic, not regular UDP, so I assume those are no being used. Here is what I have in my custom box right now:
forward-zone: name: "." forward-ssl-upstream: yes forward-addr: 1.1.1.1@853 forward-addr: 9.9.9.9@853 server:include: /var/unbound/pfb_dnsbl.*conf server: log-queries: yes log-replies: yesThe trick is trying to figure out where that query is coming from
Agreed. Other than the OS itself doing the basic stuff, the only other lookups I would expect to see from this box are from pfBlocker, which uses numerous block lists which it updates periodically. This process certainly shouldn't be doing what we are seeing happen, but it is the only other main source of queries I can think of. It is also doing Geo blocking (yes, I know...it probably isn't really that helpful in most circumstances) so it needs to update those IP lists periodically.
-
Yeah something is not right because that is not really a valid query.. A query for an IP should be a PTR query.. And a A record like that would really never resolve because that last part would be considered the tld.. and .072 sure isn't a valid tld..
Now your 066.140.x.x.localdomain could resolve.. if the local NS you were asking had a record like that in it.. but public would never resolve that sort of query.
Maybe look through your xml, you can download a full backup and search for anything that could be put in somewhere that matches that odd query.
edit: BTW your not seeing any other odd queries like that? Just that 1 weird one that repeats over and over?
Are you seeing any firewall logs with that IP in it? Or that IP reversed, where maybe the PTR got messed up? so like 72.140.159.66 or 66.159.140.72 ?
-
My apologies for taking so long to respond. Some other things we demanding my time.
@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:
Now your 066.140.x.x.localdomain could resolve.. if the local NS you were asking had a record like that in it.. but public would never resolve that sort of query.
My local resolver does not have any entries like that.
Maybe look through your xml, you can download a full backup and search for anything that could be put in somewhere that matches that odd query.
I'm not exactly sure what to look for, but I found no references to that address in the XML
edit: BTW your not seeing any other odd queries like that? Just that 1 weird one that repeats over and over?
Pretty much, although I think I have seen one or two other addresses on rare occasions. I ran another pcap, but forgot I had DoH on, so it doesn't offer many insights...just that it was communicating a great deal with both DNS providers. (I can hear your "I told you so"'s already...)
I forgot to grab the DNS log at the time.Are you seeing any firewall logs with that IP in it? Or that IP reversed, where maybe the PTR got messed up? so like 72.140.159.66 or 66.159.140.72 ?
Well, I haven't checked recently. I will do that next time. My log is currently swamped with some IPv6 address pounding away on it with UDP. (odd, since this is behind another firewall...)
By the way, I tried disabling pfblocker while this noise was going on, and it didn't resolve the problem. I didn't restart the device though, since that would temporarily "solve" the issue anyway. When I turned off pfblocker, here is what happened for network activity.
Then it resumed the background noise again. I did think it was a little interesting that the download speed actually dipped as the upload spiked. Maybe that's just due to a network bottleneck though. Also, why so much upload for this?Upon re-enabling pfblocker, the same basic thing happened.
I doubt that is significant, but I figured I'd share it anyway.
Also, FYI here is a "baseline" of sorts, from later in the day while it has 4 computers semi-active on it.
Note that it has quieted down.
