pfSense using unreasonable amount of bandwidth while idle
-
There is no code within the PHP package that would do that (save for the
curlreference I mentioned), and I am quite familiar with the Snort binary code and know of nothing in there that would account for such activity.The last time I recall a discussion similar to this with mysterious DNS lookups it turned out to be a logging application of some type that was attempting to convert IP addresses to hostnames.
You know there is a setting on the pfSense firewall logging tab that turns this feature on, but I'm not sure if that is a permanent thing or just active while you have the firewall log tab open. It resolves IP to hostnames (or attempts to).
-
That is a good hint for sure.. You wouldn't be able to recall enough to find this thread where this was happening before? That might be a clue for sure.
I don't recall seeing such a thread.. But so many, and many are just the same over and over they all tend to blend together.
So you think it was something or could have something to do with logging in pfsense, or some package?? Prob the best clue we have had sofar in trying to track this down.
-
@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:
That is a good hint for sure.. You wouldn't be able to recall enough to find this thread where this was happening before? That might be a clue for sure.
I don't recall seeing such a thread.. But so many, and many are just the same over and over they all tend to blend together.
So you think it was something or could have something to do with logging in pfsense, or some package?? Prob the best clue we have had sofar in trying to track this down.
I think it was a year ago or maybe a little more, but I don't recall more than that. My years are catching up with me. My memory has "read errors" sometimes now ...
.Oh, and that Firewall Log thing is actually an icon you click on the page, not a parameter.
-
Looking in all the settings for logging - and I don't see anything that should or could do that.. Unless something really odd going on if sending logs to remote syslog???
@CyberMinion Do you have logs being sent to remote syslog?
Also what about looking in your cron jobs for something? You can install the cron package to get look see to what is in there.
-
Thought you said you removed pfblocker?
I disabled it, but did not uninstall it. Think it makes a difference? I also tried this with Snort at one point btw, and yes, no change.
Looking in all the settings for logging - and I don't see anything that should or could do that.. Unless something really odd going on if sending logs to remote syslog???
I will look around as well
@CyberMinion Do you have logs being sent to remote syslog?
No, local logging only.
Also what about looking in your cron jobs for something? You can install the cron package to get look see to what is in there.
Not a bad idea...The only cron jobs I'm aware of are updating pfblocker's rules periodically. Do you have a suggestion of which package to try for this purpose?
-
Am I correct in thinking the queries you see are the odd-IP-as-FQDN followed by odd-IP-as-FQDN.yourdomain every time?
Is there anything that could be running on pfSense that actually queries like that?
My usual assumption when I see that sort of query pattern is that it's a Windows client.Also, reviewing this, the majority of this traffic appeared to be inbound on WAN. Just the nature of DoT?
Steve
-
@CyberMinion said in pfSense using unreasonable amount of bandwidth while idle:
Do you have a suggestion of which package to try for this purpose?
Yeah the cron package ;)
And yeah I would agree, whatever is doing those queries is messed up... So yeah it would scream some shit software on some windows machine if you ask me as well.
But he insists he is not doing dns redirection, and a windows client on the network would show its actual IP as source vs loopback. The only way to get loopback as the source, is something running on pfsense doing the query, or redirection.. Can you post up your portfowards tab?
Other thing could be proxy running, but he says not running that, etc.
And yeah the queries are adding his domain in a search suffix query, after the query.. After the first query.. Really just at a loss to what would/could be doing it.
I personally would do a clean slate..
And yeah using doh/dot especially when forwarding to multiple ns amounts to a low level dos on yourself when you have something basically looking for nonsense..
None of that shit that is being asked for is ever going to resolve.. Its just asking for nonsense, with extra helping of overhead on top of it, and lets not just ask 1 ns, lets ask them all.. Once the first nx comes back, you shouldn't be asking the next guy ever.. His sniffs are really a constant stream of asking for junk..
edit: I just counted up 1 second of dns queries in on of his sniffs, doing 30 queries a second for utter garbage.. If I filter on just queries in the 78 second long sniff, there are 1740 dns queries or 22 queries a second for the duration of the sniff.. all for garbage that will never resolve.
Forget that its adding his suffix, none of this shit would ever resolve
So what is doing them is the question.. I don't know how to look to see what is doing the query as far as process.. That would be the smoking gun, is to know what is actually asking for this nonsense. unbound is just doing what it was asked to do..
Guess you could try running through
sockstat -4 -p 53doing that trying to catch what process is talking to loopback on 53.. Other than unbound.
-
Someone may have already done this, and I missed it, but ARIN says the netblock 61.125.0.0/18 belongs to NTT-ME Corporation, which is located in Japan. The specific company brand listed is Xephion, which appears to be an ISP.
The 61.124.0.0/16 block of addresses is listed as belonging to Fujitsu, LTD. Specifically it seems to be registered for their InfoWeb service, which also appears to be a Japanese ISP.
-
@stephenw10 said in pfSense using unreasonable amount of bandwidth while idle:
Am I correct in thinking the queries you see are the odd-IP-as-FQDN followed by odd-IP-as-FQDN.yourdomain every time?
It tries to lookup the IP first, then adds the network name as if it was a FQDN, as descrided, and tries getting an A record, then an AAA record.
Is there anything that could be running on pfSense that actually queries like that?
My usual assumption when I see that sort of query pattern is that it's a Windows client.None that I am aware of. It is vaguely possible that a Windows client is triggering the problem, but once this "noise" is going on, no internal clients are needed to keep it going. I've tried pulling the LAN cables out of the box, but the problem continues. Is there any way a client could be triggering this problem, then disconnect and have the problem persist?
Also, reviewing this, the majority of this traffic appeared to be inbound on WAN. Just the nature of DoT?
Steve
Assuming you mean DoH, I did try disabling that, but this problem persisted. It is flowing to and from the WAN port only with nominal if any traffic on the internal ports. DoH adds a little "bulk" to the DNS, but not THIS much...
-
@bmeeks said in pfSense using unreasonable amount of bandwidth while idle:
Someone may have already done this, and I missed it, but ARIN says the netblock 61.125.0.0/18 belongs to NTT-ME Corporation, which is located in Japan. The specific company brand listed is Xephion, which appears to be an ISP.
The 61.124.0.0/16 block of addresses is listed as belonging to Fujitsu, LTD. Specifically it seems to be registered for their InfoWeb service, which also appears to be a Japanese ISP.
Yeah, a number of these seem to be owned by ISPs around the world. Another one it was doing was owned by a Chinese ISP...I blocked that one manually on the firewall, just because.
@johnpoz
Right...what a difficult to guess package name. Here's what I've got for jobs:
It's interesting to me that there are suricata and acme jobs here still.
-
This post is deleted! -
@CyberMinion said in pfSense using unreasonable amount of bandwidth while idle:
@bmeeks said in pfSense using unreasonable amount of bandwidth while idle:
Someone may have already done this, and I missed it, but ARIN says the netblock 61.125.0.0/18 belongs to NTT-ME Corporation, which is located in Japan. The specific company brand listed is Xephion, which appears to be an ISP.
The 61.124.0.0/16 block of addresses is listed as belonging to Fujitsu, LTD. Specifically it seems to be registered for their InfoWeb service, which also appears to be a Japanese ISP.
Yeah, a number of these seem to be owned by ISPs around the world. Another one it was doing was owned by a Chinese ISP...I blocked that one manually on the firewall, just because.
@johnpoz
Right...what a difficult to guess package name. Here's what I've got for jobs:
It's interesting to me that there are suricata and acme jobs here still.
The Suricata and Snort cron tasks should be removed when you delete the package from the firewall using the SYSTEM > PACKAGE MANAGER tab. However, they will remain in place and active if you simply disable Snort or Suricata on the interfaces. But I can guarantee you that neither of those packages is responsible for any of those DNS lookups. Since I wrote them, I know what they can and cannot do ...
-
That was just a very small sample of the IPs.. They are from all over.
I would say its something stupid trying to look up IPs that hit the wan, but its not that for sure.. He is not seeing 22 hits a second on his wan in noise ;)
-
@CyberMinion said in pfSense using unreasonable amount of bandwidth while idle:
DoH adds a little "bulk" to the DNS, but not THIS much...
No it adds a LOT of bulk! But no this is with doh off, and still seeing quite a bit of traffic..
Problem with doh other than the amount of added overhead, is when had that enabled could not actually see what was being queried for..
There for sure is nothing in pfsense out of the box that would be looking for this sort of garbage.. Its not trying to do PTRs on those Ips.. Its doing a normal A and AAAA query for what looks to be an IP.. Doing a normal A query for xxx.xxx.xxx.xxx is not going to return anything almost ever..
There should be nothing that looks for such FQDNs be it the suffix is added or not.. Trying to find a A or AAAA for what looks like an IP is nonsense.. In my little snippet of queries you would be looking for stuff with tld of .136, .207, .243 etc.. those are not TLDs that would ever be valid, ever..
So whatever it is asking for this is broken, borked, something.. Bad coding trying to parse thru a list of IPs maybe? Something clearly F'd up for sure.. Those are never going to be valid queries.. Now possible sure if you added a valid domain as the search suffix, you might find a A record like xxx.xxx.xxx.xxx.domain.tld - but search A for just xxx.xxx.xxx.xxx where that last .xxx would be the TLD is almost never going to be valid. You might be able to run a local NS where .xxx as a number is valid??? But why would anyone do such a thing, and sure never going to be viable on the public
There are no TLDs that are numbers
https://www.iana.org/domains/root/dbSo asking for something like A or AAAA for AAA.BBB.CCC.DDD where .ddd is the tld would never find anything.
Now in nslookup you ask for say 192.168.9.253, but it changes that for you to a PTR, see below the actual query/question that is asked "100.9.168.192.in-addr.arpa"
$ nslookup Default Server: pi-hole.local.lan Address: 192.168.3.10 > set debug > 192.168.9.100 Server: pi-hole.local.lan Address: 192.168.3.10 ------------ Got answer: HEADER: opcode = QUERY, id = 2, rcode = NOERROR header flags: response, want recursion, recursion avail. questions = 1, answers = 1, authority records = 0, additional = 0 QUESTIONS: 100.9.168.192.in-addr.arpa, type = PTR, class = IN ANSWERS: -> 100.9.168.192.in-addr.arpa name = i5-win.local.lan ttl = 3212 (53 mins 32 secs) ------------ Name: i5-win.local.lan Address: 192.168.9.100It doesn't do a A or AAAA query for 192.168.9.100
So whatever is asking for these is bad code? Or something being fed info to do a dns queries the wrong sort of info..
-
@bmeeks said in pfSense using unreasonable amount of bandwidth while idle:
The Suricata and Snort cron tasks should be removed when you delete the package from the firewall using the SYSTEM > PACKAGE MANAGER tab. However, they will remain in place and active if you simply disable Snort or Suricata on the interfaces. But I can guarantee you that neither of those packages is responsible for any of those DNS lookups.
I removed Suricata and acme through the package manager, but SNORT is still installed and active. If it won't cause any confusion with troubleshooting, maybe I'll just remove those jobs manually. Like I said...the Suricata uninstall messed up somewhere, for me.
Since I wrote them, I know what they can and cannot do ...
Nice! Well thank you for writing them, and for the input
-
@CyberMinion said in pfSense using unreasonable amount of bandwidth while idle:
I removed Suricata and acme through the package manager, but SNORT is still installed and active. If it won't cause any confusion with troubleshooting, maybe I'll just remove those jobs manually. Like I said...the Suricata uninstall messed up somewhere, for me.
It won't hurt anything to manually remove them. If the uninstall crashed, that might explain them hanging around. The uninstall routine within the Suricata GUI code specifically calls a pfSense system function to remove its cron tasks. But if the uninstall crashed or was not allowed to complete (by leaving the page prematurely, for instance), then the piece of code that makes that system call would not have executed.
-
You know looking deeper into these queries... Something is way messed up... Lets say there was some package or script doing this.. It is borked out of its mind.. These sorts of queries can not resolve - really ever! Doing them is trash!!
If your sure its not a client asking for them via a redirect.. I would really start over, no matter how much of a PITA it is.. The amount of queries and the type of them is something you need to get rid of, even if was bandwidth you could live with.
They are not hits to your wan, and something looking to resolve said IP. So whatever it is parsing some "list" it got from somewhere. Or just randomly asking for shit?
Very ODD to be sure.. But it is very difficult to track down a specific process using the own internal OS dns features to query for something.
You could go through every process.. doing a ps -ax do you see something that is loaded with some odd conf file or text file?
-
@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:
So whatever it is asking for this is broken, borked, something.. Bad coding trying to parse thru a list of IPs maybe? Something clearly F'd up for sure.. Those are never going to be valid queries.. Now possible sure if you added a valid domain as the search suffix, you might find a A record like xxx.xxx.xxx.xxx.domain.tld - but search A for just xxx.xxx.xxx.xxx where that last .xxx would be the TLD is almost never going to be valid.
Yep, something is borked for sure. The only code going through lists of IPs (as far as I know) should be the OS itself, and pfblocker.
You might be able to run a local NS where .xxx as a number is valid??? But why would anyone do such a thing, and sure never going to be viable on the public
Maybe you could (don't know) but I certainly am not doing that.
So whatever is asking for these is bad code? Or something being fed info to do a dns queries the wrong sort of info..
No idea. I have wondered if there is a glitch in my pfSense OS itself...as mentioned before, I'm also having an issue where I apply a config change, it saves, and for somewhere between 6 hours and 3 days, it uses that change. Then suddenly...the change is gone. This happeens for small changes all the time...I have gotten in the habit of making a change, a backup, and a note about what I changed...then I check back a few days later, and put it back--sometimes several times. This also happened when I installed Snort the first time...I installed it set it up, and let it run. According to logs, it ran for 3 days, then it just...vanished. The logs still contained entries from it, and I still had a menu item for it, but the package itself was gone. I have no idea how that happened...I do have auto config backups running, which can restore the settings with just a few clicks, but I certainly wasn't doing those clicks...I wasn't even touching the box until a week later, when I went to see what Snort had detected so far (and found it missing).
-
Well that sure sounds like you need to start over to me..
-
@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:
You could go through every process.. doing a ps -ax do you see something that is loaded with some odd conf file or text file?
Here is what I have going right now, but this lookup nonsense isn't going on right now. Nothing stands out to me here. I will check again when it is.
PID TT STAT TIME COMMAND 0 - DLs 0:01.06 [kernel] 1 - ILs 0:00.10 /sbin/init -- 2 - DL 0:00.00 [crypto] 3 - DL 0:00.00 [crypto returns 0] 4 - DL 0:00.00 [crypto returns 1] 5 - DL 0:00.00 [cam] 6 - DL 2:27.66 [e6000sw tick kproc] 7 - DL 0:00.00 [sctp_iterator] 8 - DL 2:45.51 [pf purge] 9 - DL 0:49.76 [rand_harvestq] 10 - DL 0:00.00 [audit] 11 - RNL 9836:51.39 [idle] 12 - WL 68:35.58 [intr] 13 - DL 0:00.00 [ng_queue] 14 - DL 0:28.79 [geom] 15 - DL 0:05.44 [usb] 16 - DL 0:00.00 [task: mx25l flash] 17 - DL 1:56.38 [mmcsd0: mmc/sd card] 18 - DL 0:00.00 [mmcsd0boot0: mmc/sd] 19 - DL 0:00.00 [mmcsd0boot1: mmc/sd] 20 - DL 0:00.15 [soaiod1] 21 - DL 0:00.15 [soaiod2] 22 - DL 0:00.15 [soaiod3] 23 - DL 0:00.15 [soaiod4] 24 - DL 0:44.90 [pagedaemon] 25 - DL 0:00.00 [vmdaemon] 26 - DNL 0:00.01 [pagezero] 27 - DL 0:04.48 [bufspacedaemon] 28 - DL 0:19.28 [bufdaemon] 29 - DL 0:04.71 [vnlru] 30 - DL 0:30.19 [syncer] 70 - DL 0:01.24 [md0] 363 - Ss 0:19.18 php-fpm: master process (/usr/local/lib/php-fpm.conf) 379 - INs 0:00.10 /usr/local/sbin/check_reload_status 380 - IN 0:00.00 check_reload_status: Monitoring daemon of check_reloa 400 - Is 0:00.33 /sbin/devd -q -f /etc/pfSense-devd.conf 8054 - S 0:04.74 php-fpm: pool nginx (php-fpm) 9837 - Is 0:00.01 dhclient: mvneta0.4090 [priv] (dhclient) 13957 - SCs 0:00.55 dhclient: mvneta0.4090 (dhclient) 23809 - R 0:00.02 ps -ax 25049 - Ss 186:14.74 /usr/local/sbin/unbound -c /var/unbound/unbound.conf 26128 - S 0:45.80 /usr/local/sbin/lighttpd_pfb -f /var/unbound/pfb_dnsb 32713 - Ss 0:27.54 /usr/local/sbin/filterlog -i pflog0 -p /var/run/filte 33817 - Ss 0:09.52 /usr/local/sbin/radvd -p /var/run/radvd.pid -C /var/e 34101 - Is 0:22.94 /usr/local/bin/dpinger -S -r 0 -i WAN_DHCP -B 192.168 40730 - S 0:00.73 /usr/libexec/getty al.Pc ttyv0 41549 - INC 0:00.00 sleep 60 42128 - Is 0:00.03 /bin/sh /usr/local/sbin/sshguard -i /var/run/sshguard 42553 - S 0:00.01 /bin/cat 42795 - SC 0:00.02 /usr/local/libexec/sshg-parser 42987 - IC 0:00.58 /usr/local/libexec/sshg-blocker 43320 - I 0:00.00 /bin/sh /usr/local/sbin/sshguard -i /var/run/sshguard 43618 - I 0:00.01 /bin/sh /usr/local/libexec/sshg-fw-pf 45305 - IN 0:20.69 /bin/sh /var/db/rrd/updaterrd.sh 45799 - Ss 0:41.91 /usr/local/sbin/miniupnpd -f /var/etc/miniupnpd.conf 54131 - Ss 0:13.31 /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroo 56847 - Is 0:00.17 /usr/local/sbin/dhcpleases -l /var/dhcpd/var/db/dhcpd 61201 - S 0:15.37 php-fpm: pool nginx (php-fpm) 78271 - S 0:09.77 php-fpm: pool nginx (php-fpm) 78427 - S 0:45.02 /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfbl 82925 - IN 0:00.00 /bin/sh /etc/rc.update_pkg_metadata 83006 - INC 0:00.01 sleep 46656 86292 - Ss 28:56.28 /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log 87116 - Is 0:00.01 nginx: master process /usr/local/sbin/nginx -c /var/e 87180 - S 11:42.70 nginx: worker process (nginx) 87731 - Is 0:02.54 /usr/sbin/cron -s 87748 - Is 0:00.00 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid / 88398 - Ss 1:04.33 /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var 88411 - Is 0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts 88796 - I 0:00.03 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.e 88909 - Is 0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_u 89308 - I 0:00.37 minicron: helper /usr/local/bin/ping_hosts.sh (minic 89639 - I 0:00.00 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.u 40954 u0 Is 0:00.04 login [pam] (login) 44424 u0 I 0:00.02 -sh (sh) 56005 u0 S+ 0:00.02 /bin/sh /etc/rc.initialI still find it quite interesting that when the network has clients and is busy, the problem goes away. But when the network goes idle, it starts up again. That in itself seems significant somehow...
