pfSense using unreasonable amount of bandwidth while idle
-
@CyberMinion said in pfSense using unreasonable amount of bandwidth while idle:
I removed Suricata and acme through the package manager, but SNORT is still installed and active. If it won't cause any confusion with troubleshooting, maybe I'll just remove those jobs manually. Like I said...the Suricata uninstall messed up somewhere, for me.
It won't hurt anything to manually remove them. If the uninstall crashed, that might explain them hanging around. The uninstall routine within the Suricata GUI code specifically calls a pfSense system function to remove its cron tasks. But if the uninstall crashed or was not allowed to complete (by leaving the page prematurely, for instance), then the piece of code that makes that system call would not have executed.
-
You know looking deeper into these queries... Something is way messed up... Lets say there was some package or script doing this.. It is borked out of its mind.. These sorts of queries can not resolve - really ever! Doing them is trash!!
If your sure its not a client asking for them via a redirect.. I would really start over, no matter how much of a PITA it is.. The amount of queries and the type of them is something you need to get rid of, even if was bandwidth you could live with.
They are not hits to your wan, and something looking to resolve said IP. So whatever it is parsing some "list" it got from somewhere. Or just randomly asking for shit?
Very ODD to be sure.. But it is very difficult to track down a specific process using the own internal OS dns features to query for something.
You could go through every process.. doing a ps -ax do you see something that is loaded with some odd conf file or text file?
-
@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:
So whatever it is asking for this is broken, borked, something.. Bad coding trying to parse thru a list of IPs maybe? Something clearly F'd up for sure.. Those are never going to be valid queries.. Now possible sure if you added a valid domain as the search suffix, you might find a A record like xxx.xxx.xxx.xxx.domain.tld - but search A for just xxx.xxx.xxx.xxx where that last .xxx would be the TLD is almost never going to be valid.
Yep, something is borked for sure. The only code going through lists of IPs (as far as I know) should be the OS itself, and pfblocker.
You might be able to run a local NS where .xxx as a number is valid??? But why would anyone do such a thing, and sure never going to be viable on the public
Maybe you could (don't know) but I certainly am not doing that.
So whatever is asking for these is bad code? Or something being fed info to do a dns queries the wrong sort of info..
No idea. I have wondered if there is a glitch in my pfSense OS itself...as mentioned before, I'm also having an issue where I apply a config change, it saves, and for somewhere between 6 hours and 3 days, it uses that change. Then suddenly...the change is gone. This happeens for small changes all the time...I have gotten in the habit of making a change, a backup, and a note about what I changed...then I check back a few days later, and put it back--sometimes several times. This also happened when I installed Snort the first time...I installed it set it up, and let it run. According to logs, it ran for 3 days, then it just...vanished. The logs still contained entries from it, and I still had a menu item for it, but the package itself was gone. I have no idea how that happened...I do have auto config backups running, which can restore the settings with just a few clicks, but I certainly wasn't doing those clicks...I wasn't even touching the box until a week later, when I went to see what Snort had detected so far (and found it missing).
-
Well that sure sounds like you need to start over to me..
-
@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:
You could go through every process.. doing a ps -ax do you see something that is loaded with some odd conf file or text file?
Here is what I have going right now, but this lookup nonsense isn't going on right now. Nothing stands out to me here. I will check again when it is.
PID TT STAT TIME COMMAND 0 - DLs 0:01.06 [kernel] 1 - ILs 0:00.10 /sbin/init -- 2 - DL 0:00.00 [crypto] 3 - DL 0:00.00 [crypto returns 0] 4 - DL 0:00.00 [crypto returns 1] 5 - DL 0:00.00 [cam] 6 - DL 2:27.66 [e6000sw tick kproc] 7 - DL 0:00.00 [sctp_iterator] 8 - DL 2:45.51 [pf purge] 9 - DL 0:49.76 [rand_harvestq] 10 - DL 0:00.00 [audit] 11 - RNL 9836:51.39 [idle] 12 - WL 68:35.58 [intr] 13 - DL 0:00.00 [ng_queue] 14 - DL 0:28.79 [geom] 15 - DL 0:05.44 [usb] 16 - DL 0:00.00 [task: mx25l flash] 17 - DL 1:56.38 [mmcsd0: mmc/sd card] 18 - DL 0:00.00 [mmcsd0boot0: mmc/sd] 19 - DL 0:00.00 [mmcsd0boot1: mmc/sd] 20 - DL 0:00.15 [soaiod1] 21 - DL 0:00.15 [soaiod2] 22 - DL 0:00.15 [soaiod3] 23 - DL 0:00.15 [soaiod4] 24 - DL 0:44.90 [pagedaemon] 25 - DL 0:00.00 [vmdaemon] 26 - DNL 0:00.01 [pagezero] 27 - DL 0:04.48 [bufspacedaemon] 28 - DL 0:19.28 [bufdaemon] 29 - DL 0:04.71 [vnlru] 30 - DL 0:30.19 [syncer] 70 - DL 0:01.24 [md0] 363 - Ss 0:19.18 php-fpm: master process (/usr/local/lib/php-fpm.conf) 379 - INs 0:00.10 /usr/local/sbin/check_reload_status 380 - IN 0:00.00 check_reload_status: Monitoring daemon of check_reloa 400 - Is 0:00.33 /sbin/devd -q -f /etc/pfSense-devd.conf 8054 - S 0:04.74 php-fpm: pool nginx (php-fpm) 9837 - Is 0:00.01 dhclient: mvneta0.4090 [priv] (dhclient) 13957 - SCs 0:00.55 dhclient: mvneta0.4090 (dhclient) 23809 - R 0:00.02 ps -ax 25049 - Ss 186:14.74 /usr/local/sbin/unbound -c /var/unbound/unbound.conf 26128 - S 0:45.80 /usr/local/sbin/lighttpd_pfb -f /var/unbound/pfb_dnsb 32713 - Ss 0:27.54 /usr/local/sbin/filterlog -i pflog0 -p /var/run/filte 33817 - Ss 0:09.52 /usr/local/sbin/radvd -p /var/run/radvd.pid -C /var/e 34101 - Is 0:22.94 /usr/local/bin/dpinger -S -r 0 -i WAN_DHCP -B 192.168 40730 - S 0:00.73 /usr/libexec/getty al.Pc ttyv0 41549 - INC 0:00.00 sleep 60 42128 - Is 0:00.03 /bin/sh /usr/local/sbin/sshguard -i /var/run/sshguard 42553 - S 0:00.01 /bin/cat 42795 - SC 0:00.02 /usr/local/libexec/sshg-parser 42987 - IC 0:00.58 /usr/local/libexec/sshg-blocker 43320 - I 0:00.00 /bin/sh /usr/local/sbin/sshguard -i /var/run/sshguard 43618 - I 0:00.01 /bin/sh /usr/local/libexec/sshg-fw-pf 45305 - IN 0:20.69 /bin/sh /var/db/rrd/updaterrd.sh 45799 - Ss 0:41.91 /usr/local/sbin/miniupnpd -f /var/etc/miniupnpd.conf 54131 - Ss 0:13.31 /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroo 56847 - Is 0:00.17 /usr/local/sbin/dhcpleases -l /var/dhcpd/var/db/dhcpd 61201 - S 0:15.37 php-fpm: pool nginx (php-fpm) 78271 - S 0:09.77 php-fpm: pool nginx (php-fpm) 78427 - S 0:45.02 /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfbl 82925 - IN 0:00.00 /bin/sh /etc/rc.update_pkg_metadata 83006 - INC 0:00.01 sleep 46656 86292 - Ss 28:56.28 /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log 87116 - Is 0:00.01 nginx: master process /usr/local/sbin/nginx -c /var/e 87180 - S 11:42.70 nginx: worker process (nginx) 87731 - Is 0:02.54 /usr/sbin/cron -s 87748 - Is 0:00.00 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid / 88398 - Ss 1:04.33 /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var 88411 - Is 0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts 88796 - I 0:00.03 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.e 88909 - Is 0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_u 89308 - I 0:00.37 minicron: helper /usr/local/bin/ping_hosts.sh (minic 89639 - I 0:00.00 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.u 40954 u0 Is 0:00.04 login [pam] (login) 44424 u0 I 0:00.02 -sh (sh) 56005 u0 S+ 0:00.02 /bin/sh /etc/rc.initialI still find it quite interesting that when the network has clients and is busy, the problem goes away. But when the network goes idle, it starts up again. That in itself seems significant somehow...
-
Do you have any aliases setup? That might have some listing of what you think is IPs? and the filterdns mech is trying to resolve them because it thinks they are fqdn? filterdns out of the box would run every 5 minutes though.
This is clearly one of the weirdest things I have seen around here in a very long time, if not the most weirdest thing.. Its crazy..
-
@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:
Do you have any aliases setup? That might have some listing of what you think is IPs? and the filterdns mech is trying to resolve them because it thinks they are fqdn? filterdns out of the box would run every 5 minutes though.
Yes, I have a bunch a aliases in my FW config (they need to be updated, actually)
The "Ports" and "URLs" tabs are empty. Where does filterdns live? Do you mean the component of pfBlocker, or is this something else?This is clearly one of the weirdest things I have seen around here in a very long time, if not the most weirdest thing.. Its crazy..
Well, at least it's not some stupid thing I should have figured out myself...I wouldn't be surprised if I still did something wrong in config to cause this, though. I have a backup package from about 9 months ago. Not much has been changed in config since then, so I'm thinking of trying restoring to that, just in case that wipes out this problem. That's probably wishful thinking, but its a low-risk change.
-
Well filterdns runs every 5 minutes out of the box
[2.4.5-RELEASE][admin@sg4860.local.lan]/root: ps -ax | grep filterdns 61721 - Is 0:10.79 /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 300 -c /var/etc/filterdns.conf -d 1 17951 0 S+ 0:00.00 grep filterdnsYou could delete all of those.. Or when its doing it, you could kill off the filterdns process and see if it stops.
I guess its possible that something wrong where you think you have a alias setup as fqdn and not IP with an IP in it, that you might resolve that.. Let me do a little test, brb
What do you have being loaded by URL? I created a alias and then watched my unbound log, and not seeing any odd queries for it as a A or AAAA query. But guess its possible if your loading something via a url where something could get parsed wrong?
Post up what lists your using via url so I can take a look see, in your sniff I did see lists being loaded - but didn't notice any IPs that jumped out at me in them for what was being queried.
-
@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:
You could delete all of those.. Or when its doing it, you could kill off the filterdns process and see if it stops.
Interesting...I'll try that.
What do you have being loaded by URL? I created a alias and then watched my unbound log, and not seeing any odd queries for it as a A or AAAA query. But guess its possible if your loading something via a url where something could get parsed wrong?
I have nothing by way of aliases
But I'm using pfBlocker to pull some lists.Post up what lists your using via url so I can take a look see, in your sniff I did see lists being loaded - but didn't notice any IPs that jumped out at me in them for what was being queried.
Excluding the DNSBL lists, I am using the ones listed here: https://cloud.archnet.us/index.php/s/R2jXJzkJXBoXxkT
(I couldn't include them in this post. Apparently, I'm not allowed to have more than 1 URL in my posts? Or maybe it didn't like one of those URLs specifically? I don't know...the spam filter on this forum is aggressive.)Do you want my DNSBL lists too? I have many more of those, but I can gather them all if needed.
-
well this one
https://www.spamhaus.org/drop/asndrop.txtis a bunch of AS numbers - those would have to be resolved to IP ranges or CIDR blocks..
Why are you loading all these lists? You don't even have any port forwards setup? I just don't get the reason for trying to filter shit when you don't even have anything exposed?
-
@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:
well this one
https://www.spamhaus.org/drop/asndrop.txtis a bunch of AS numbers - those would have to be resolved to IP ranges or CIDR blocks..
Oh..that's interesting. It used to have the IP blocks themselves on that list. I'll remove that one. I should check through all of my lists...it has been almost two years since those were added.
Why are you loading all these lists? You don't even have any port forwards setup? I just don't get the reason for trying to filter shit when you don't even have anything exposed?
I don't have any ports forwarded right now although I did in the past. That is partially because I wasn't sure what was wrong on this network. If I regain confidence that nothing is vulnerable or compromised, I will probably open ports again. To be clear, I want to have open ports, but I'd prefer to make sure everything is shipshape before doing that again. It looks like this is probably a benign issue (although I still need to fix it), but I want to get it working properly first, just the same.
-
Vs trying to block the planet, if you are going to open up ports - its better to just limit what can talk to your ports. Example - I have plex open for friends and family. They are all in the US.. So I only allow US Ips to talk to that.. I don't load every possible bad guy list and block them - that is just insane to try and do that..
When I had someone outside the US, I added that country..
Sure in a perfect setup would only allow their specific IPs.. But then again they are users - asking them for their IP is like asking a monkey to explain theory of relativity.. And many of them use phones and travel, etc.
I would really if your not going to just nuke this thing from orbit and start over - which is what I would do.. Is turn off all this list shit..
You use bad guy lists, when that is the smaller list. Ie you want to serve up stuff to the globe - that is when you filter bad guy lists. Since that is smaller list ;)
-
@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:
Vs trying to block the planet, if you are going to open up ports - its better to just limit what can talk to your ports.
You use bad guy lists, when that is the smaller list. Ie you want to serve up stuff to the globe - that is when you filter bad guy lists. Since that is smaller list ;)At the risk of continuing down the age-old "block the planet" vs whitelist dispute, the problem is, I plan to have incoming connections from six continents. Whitelist management is going to be a much bigger headache than some low-maintenance blacklists.
I would really if your not going to just nuke this thing from orbit and start over - which is what I would do.. Is turn off all this list shit..
Yeah, I'm leaning that way. I reverted this unit to a backup from December 2019. After fixing a few small things, it seems to be humming along without issues so far. It does still have the aforementioned fragments of the Suricata package, but hopefully that's irrelevant. After about 4 days, I have yet to notice these garbage DNS queries, although I may have missed some.
-
@CyberMinion said in pfSense using unreasonable amount of bandwidth while idle:
I have yet to notice these garbage DNS queries, although I may have missed some.
You should have ZERO of those - there is NOTHING that should create dns queries like that.. They can not resolve - they are garbage..
-
I hate to bump this to the top, but I just wanted to let you know that I have not seen any of this bad behavior from the firewall since restoring from backup. Whatever it was, it seems to be fixed. I just wish I knew the root cause.
Thank you for your help, everyone!
-
Mmm, weird. Thanks for following up though. Yeah it would be good to know what it was but that's probably not possible at this point.
Steve
