pfSense using unreasonable amount of bandwidth while idle
-
@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:
How is this rule not just completely pointless? You have a ipv4 rule with a IPv6 source of addresses.. Be it your going to open a port or not?
Yup, that would be useless. I just corrected it to IPv6--thanks.
In your traffic graph or iptop, or darkstat -- look to see where the remote non pfsense IPs are in that traffic flow.. Its not insignificant - so should be easy enough to spot.. Clear your counters..
Now look in your state tables.. Pfsense would have zero reason to be sending traffic to that IP listed for example.. You hid the other one.. so have no idea what that is.. But that ip listed that has some traffic going to it - top of your list.. Is not something pfsense would ever talk to on its own from any sort of normal setup. And sure is not your dns your forwarding too.. So what is creating that traffic, its not pfsense. Unless its maybe something downloading some list? It is 443 traffic. And from pfsense top - it would be this 192.168.3.2 generating the traffic!! See the pftop exact same packet counts..
Yes, but keep in mind, that includes the traffic load from the daytime, when PCs behind it are in use. Of course those entries have a lot of traffic...this is not just an idle lab environment. If I reboot the pfsense to reset these entries, the unwanted "noise" will also go away. The only way I can troubleshoot this is with dirty data. For example, the IP I masked was seeing moderate to heavy traffic for about 10 hours, due to user activity within the network. Then it stopped. Then a few hours later, pfsense suddenly got very talkative, but that did not outweigh the user activity from earlier.
The only thing shown not matched up is thos pings from 192.168.1.1 - which pfsense would ping if that is its gateway.. But default is to send 0 size pings, and only 2 a second they would not account for 200k of data.
If it was just noise hitting your wan, there wouldn't be outbound traffic.. Pfsense if no ports open or actual reject rules does not answer noise.. its just dropped..
Exactly, that's why I started this thread in the first place. Only after the pfsense's network goes quiet, does the pfSesne start communicating a great deal with the outside. Based on the utilization of each physical port, I can see that it is not coming from inside the network, so it must be the pfsense itself. Should it be doing this? No, I don't think so. Can I figure out why it might be? No, or I wouldn't be asking here. It is always possible that something is misconfigured, but I don't have any ideas what. Since my WAN ports are closed and I'm seeing 2-way traffic, I must surmise that pfSense is initiating this activity, and based on what we have discovered since then, it looks like the activity is being initiated with my upstream DNS.
As a side note, I noticed most of this traffic was going between my secondary DNS provider, which I figured meant that the primary couldn't give an answer, so the pfSense tried the secondary. On a whim, I replaced my secondary (formerly IBM's Quad-9) to CloudFlare's secondary (1.0.0.1). Although this noise doesn't always start every night, it has been almost every night. Last night though, it didn't. At this point it doesn't conclusively mean anything, but I'm going to check back on it in a while, and see if the noise returns. It would be surprising if this actually "fixed" the problem, but I'll give it some time and see.
-
@Cool_Corona said in pfSense using unreasonable amount of bandwidth while idle:
Using SSL/TLS encrypted DNS via 9.9.9.9??
Disable it and use the local resolver and delete all DNS entries under general -> setup.
Reboot.
Let us know.
DoH is currently disabled, but I did not clear the entries. I will try that soon, but as I just posted above, I also just cut Quad-9 out of the provider list. I don't want to try too many things at once. I'm also trying to get a clean pcap of the noise for johnpoz.
Thanks for the tip!
-
So pfBlocker isn't supposed to be doing DNS lookups, as stated earlier. However, it occurs to me that the DNSBL entries can be filtered against to Alexa top sites...could this mechanic result in any DNS queries?
-
So did turning of pfblocker stop the problem.. been a few days..
-
@johnpoz
I was still having this issue after flipping off the pfBlocker switch, but I did not reboot. I have rebooted the pfsense with pfBlocker disabled, and will monitor it to see how things go. -
Yes, after a reboot with pfblocker off, I am still seeing this DNS rubbish going on.
-
Well can tell you for sure that pfsense out of the box is not going to query "185.190.088.000" as a A query..
You have something asking pfsense for that.. Or something running on pfsense doing dns query to loopback asking for that.
Even if pfsense was asked to resolve an IP, say in the logs or something - it would do a PTR for it..
Example... I enabled logging in unbound. So now I get all queries asked for logged..
server: log-queries: yesI then go into say the firewall log and ask pfsense to tell me hey who is this IP hitting my server.
by clicking the little "i" it then tries to resolve that.. Via a PTR query.. Not a A query for some name with that the IP as the name.. Also why and the F would be asking for what is clearly a bogon..
What other things do you have installed on pfsense? Or what do you have doing queries to pfsense that your not seeing the query come in? A dns redirect setup would look like loopback is asking for the record.
The only thing pfsense itself would be asking for is hey I need to get the package list from netgate, hey lets check if there is a update, or hey I have a widget on the home page and need to pull the RSS feed, etc.
Can tell you for sure if there was something in pfsense out of the box causing this traffic, or even something common... The forums would be blowing up with hey why is there 300K of traffic all the time...
-
@johnpoz
Well, I don't have RSS feeds, or other such junk going on in the box. and the lookups are showing the source being the loopback. Here are the packages I have installed:
I haven't (side)loaded any other software outside of the package manager, or installed any of my own scrips. I formerly had Suricata installed also, but it wasn't working due to RUST, so I uninstalled it. Although the package manager shows it gone, I still have a few traces of it, indicating an incomplete uninstall.
I also previously installed then uninstalled Acme. To my knowledge, that was a clean uninstallCan tell you for sure if there was something in pfsense out of the box causing this traffic, or even something common... The forums would be blowing up with hey why is there 300K of traffic all the time...
Yeah, instead we have one single, ridiculously long thread...ugh.
So what now? Would it make any difference at all if I take a full backup, do a factory reset, then restore from the backup? I have a feeling that would bring the problem right back. I don't want to reconfigure from scratch, though.
-
well iftop or nmap sure wouldn't be doing it, they don't even run anything unless you run them.
I guess snort could be be doing it? In theory - it could try and look for IPs that hit your wan..
Thought you said you removed pfblocker? I would remove them completely, both pfblocker and snort.. Just to be 110% sure its not them..
But what I can tell you for sure is there nothing in pfsense out of the box that would try and look those such fqdn..
What I would prob do, is do a clean install of pfsense with min sort of setup, an out of the box sort of setup.. And do you get such queries now?
I can tell you for sure my setup isn't doing anything of the sort.. I have logging of all queries turned on in unbound, and see nothing even close to those nonsense lookups.
-
Snort and Suricata do zero DNS queries unless the user manually clicks the designated icon on the ALERTS tab. Neither package does anything on its own with the exception of using
curlto perform the daily rules update check. During that processcurlwill look up the IP for the rules download URLs. However, those lookups are a miniscule amount of traffic. -
I hear yeah - but what else could be doing queries for such nonsense.. Seeing queries like this..
Lots and lots of them, and both A and AAA for just what looks to IPs, but not PTR queries but actual A and AAAA queries for what looks to be an IP.
I would say its some stupid client, but he states he is not doing any dns redirection, and log of the queries in unbound show they are from loopback.
-
There is no code within the PHP package that would do that (save for the
curlreference I mentioned), and I am quite familiar with the Snort binary code and know of nothing in there that would account for such activity.The last time I recall a discussion similar to this with mysterious DNS lookups it turned out to be a logging application of some type that was attempting to convert IP addresses to hostnames.
You know there is a setting on the pfSense firewall logging tab that turns this feature on, but I'm not sure if that is a permanent thing or just active while you have the firewall log tab open. It resolves IP to hostnames (or attempts to).
-
That is a good hint for sure.. You wouldn't be able to recall enough to find this thread where this was happening before? That might be a clue for sure.
I don't recall seeing such a thread.. But so many, and many are just the same over and over they all tend to blend together.
So you think it was something or could have something to do with logging in pfsense, or some package?? Prob the best clue we have had sofar in trying to track this down.
-
@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:
That is a good hint for sure.. You wouldn't be able to recall enough to find this thread where this was happening before? That might be a clue for sure.
I don't recall seeing such a thread.. But so many, and many are just the same over and over they all tend to blend together.
So you think it was something or could have something to do with logging in pfsense, or some package?? Prob the best clue we have had sofar in trying to track this down.
I think it was a year ago or maybe a little more, but I don't recall more than that. My years are catching up with me. My memory has "read errors" sometimes now ...
.Oh, and that Firewall Log thing is actually an icon you click on the page, not a parameter.
-
Looking in all the settings for logging - and I don't see anything that should or could do that.. Unless something really odd going on if sending logs to remote syslog???
@CyberMinion Do you have logs being sent to remote syslog?
Also what about looking in your cron jobs for something? You can install the cron package to get look see to what is in there.
-
Thought you said you removed pfblocker?
I disabled it, but did not uninstall it. Think it makes a difference? I also tried this with Snort at one point btw, and yes, no change.
Looking in all the settings for logging - and I don't see anything that should or could do that.. Unless something really odd going on if sending logs to remote syslog???
I will look around as well
@CyberMinion Do you have logs being sent to remote syslog?
No, local logging only.
Also what about looking in your cron jobs for something? You can install the cron package to get look see to what is in there.
Not a bad idea...The only cron jobs I'm aware of are updating pfblocker's rules periodically. Do you have a suggestion of which package to try for this purpose?
-
Am I correct in thinking the queries you see are the odd-IP-as-FQDN followed by odd-IP-as-FQDN.yourdomain every time?
Is there anything that could be running on pfSense that actually queries like that?
My usual assumption when I see that sort of query pattern is that it's a Windows client.Also, reviewing this, the majority of this traffic appeared to be inbound on WAN. Just the nature of DoT?
Steve
-
@CyberMinion said in pfSense using unreasonable amount of bandwidth while idle:
Do you have a suggestion of which package to try for this purpose?
Yeah the cron package ;)
And yeah I would agree, whatever is doing those queries is messed up... So yeah it would scream some shit software on some windows machine if you ask me as well.
But he insists he is not doing dns redirection, and a windows client on the network would show its actual IP as source vs loopback. The only way to get loopback as the source, is something running on pfsense doing the query, or redirection.. Can you post up your portfowards tab?
Other thing could be proxy running, but he says not running that, etc.
And yeah the queries are adding his domain in a search suffix query, after the query.. After the first query.. Really just at a loss to what would/could be doing it.
I personally would do a clean slate..
And yeah using doh/dot especially when forwarding to multiple ns amounts to a low level dos on yourself when you have something basically looking for nonsense..
None of that shit that is being asked for is ever going to resolve.. Its just asking for nonsense, with extra helping of overhead on top of it, and lets not just ask 1 ns, lets ask them all.. Once the first nx comes back, you shouldn't be asking the next guy ever.. His sniffs are really a constant stream of asking for junk..
edit: I just counted up 1 second of dns queries in on of his sniffs, doing 30 queries a second for utter garbage.. If I filter on just queries in the 78 second long sniff, there are 1740 dns queries or 22 queries a second for the duration of the sniff.. all for garbage that will never resolve.
Forget that its adding his suffix, none of this shit would ever resolve
So what is doing them is the question.. I don't know how to look to see what is doing the query as far as process.. That would be the smoking gun, is to know what is actually asking for this nonsense. unbound is just doing what it was asked to do..
Guess you could try running through
sockstat -4 -p 53doing that trying to catch what process is talking to loopback on 53.. Other than unbound.
-
Someone may have already done this, and I missed it, but ARIN says the netblock 61.125.0.0/18 belongs to NTT-ME Corporation, which is located in Japan. The specific company brand listed is Xephion, which appears to be an ISP.
The 61.124.0.0/16 block of addresses is listed as belonging to Fujitsu, LTD. Specifically it seems to be registered for their InfoWeb service, which also appears to be a Japanese ISP.
-
@stephenw10 said in pfSense using unreasonable amount of bandwidth while idle:
Am I correct in thinking the queries you see are the odd-IP-as-FQDN followed by odd-IP-as-FQDN.yourdomain every time?
It tries to lookup the IP first, then adds the network name as if it was a FQDN, as descrided, and tries getting an A record, then an AAA record.
Is there anything that could be running on pfSense that actually queries like that?
My usual assumption when I see that sort of query pattern is that it's a Windows client.None that I am aware of. It is vaguely possible that a Windows client is triggering the problem, but once this "noise" is going on, no internal clients are needed to keep it going. I've tried pulling the LAN cables out of the box, but the problem continues. Is there any way a client could be triggering this problem, then disconnect and have the problem persist?
Also, reviewing this, the majority of this traffic appeared to be inbound on WAN. Just the nature of DoT?
Steve
Assuming you mean DoH, I did try disabling that, but this problem persisted. It is flowing to and from the WAN port only with nominal if any traffic on the internal ports. DoH adds a little "bulk" to the DNS, but not THIS much...
