pfSense using unreasonable amount of bandwidth while idle
-
Well that sure sounds like you need to start over to me..
-
@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:
You could go through every process.. doing a ps -ax do you see something that is loaded with some odd conf file or text file?
Here is what I have going right now, but this lookup nonsense isn't going on right now. Nothing stands out to me here. I will check again when it is.
PID TT STAT TIME COMMAND 0 - DLs 0:01.06 [kernel] 1 - ILs 0:00.10 /sbin/init -- 2 - DL 0:00.00 [crypto] 3 - DL 0:00.00 [crypto returns 0] 4 - DL 0:00.00 [crypto returns 1] 5 - DL 0:00.00 [cam] 6 - DL 2:27.66 [e6000sw tick kproc] 7 - DL 0:00.00 [sctp_iterator] 8 - DL 2:45.51 [pf purge] 9 - DL 0:49.76 [rand_harvestq] 10 - DL 0:00.00 [audit] 11 - RNL 9836:51.39 [idle] 12 - WL 68:35.58 [intr] 13 - DL 0:00.00 [ng_queue] 14 - DL 0:28.79 [geom] 15 - DL 0:05.44 [usb] 16 - DL 0:00.00 [task: mx25l flash] 17 - DL 1:56.38 [mmcsd0: mmc/sd card] 18 - DL 0:00.00 [mmcsd0boot0: mmc/sd] 19 - DL 0:00.00 [mmcsd0boot1: mmc/sd] 20 - DL 0:00.15 [soaiod1] 21 - DL 0:00.15 [soaiod2] 22 - DL 0:00.15 [soaiod3] 23 - DL 0:00.15 [soaiod4] 24 - DL 0:44.90 [pagedaemon] 25 - DL 0:00.00 [vmdaemon] 26 - DNL 0:00.01 [pagezero] 27 - DL 0:04.48 [bufspacedaemon] 28 - DL 0:19.28 [bufdaemon] 29 - DL 0:04.71 [vnlru] 30 - DL 0:30.19 [syncer] 70 - DL 0:01.24 [md0] 363 - Ss 0:19.18 php-fpm: master process (/usr/local/lib/php-fpm.conf) 379 - INs 0:00.10 /usr/local/sbin/check_reload_status 380 - IN 0:00.00 check_reload_status: Monitoring daemon of check_reloa 400 - Is 0:00.33 /sbin/devd -q -f /etc/pfSense-devd.conf 8054 - S 0:04.74 php-fpm: pool nginx (php-fpm) 9837 - Is 0:00.01 dhclient: mvneta0.4090 [priv] (dhclient) 13957 - SCs 0:00.55 dhclient: mvneta0.4090 (dhclient) 23809 - R 0:00.02 ps -ax 25049 - Ss 186:14.74 /usr/local/sbin/unbound -c /var/unbound/unbound.conf 26128 - S 0:45.80 /usr/local/sbin/lighttpd_pfb -f /var/unbound/pfb_dnsb 32713 - Ss 0:27.54 /usr/local/sbin/filterlog -i pflog0 -p /var/run/filte 33817 - Ss 0:09.52 /usr/local/sbin/radvd -p /var/run/radvd.pid -C /var/e 34101 - Is 0:22.94 /usr/local/bin/dpinger -S -r 0 -i WAN_DHCP -B 192.168 40730 - S 0:00.73 /usr/libexec/getty al.Pc ttyv0 41549 - INC 0:00.00 sleep 60 42128 - Is 0:00.03 /bin/sh /usr/local/sbin/sshguard -i /var/run/sshguard 42553 - S 0:00.01 /bin/cat 42795 - SC 0:00.02 /usr/local/libexec/sshg-parser 42987 - IC 0:00.58 /usr/local/libexec/sshg-blocker 43320 - I 0:00.00 /bin/sh /usr/local/sbin/sshguard -i /var/run/sshguard 43618 - I 0:00.01 /bin/sh /usr/local/libexec/sshg-fw-pf 45305 - IN 0:20.69 /bin/sh /var/db/rrd/updaterrd.sh 45799 - Ss 0:41.91 /usr/local/sbin/miniupnpd -f /var/etc/miniupnpd.conf 54131 - Ss 0:13.31 /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroo 56847 - Is 0:00.17 /usr/local/sbin/dhcpleases -l /var/dhcpd/var/db/dhcpd 61201 - S 0:15.37 php-fpm: pool nginx (php-fpm) 78271 - S 0:09.77 php-fpm: pool nginx (php-fpm) 78427 - S 0:45.02 /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfbl 82925 - IN 0:00.00 /bin/sh /etc/rc.update_pkg_metadata 83006 - INC 0:00.01 sleep 46656 86292 - Ss 28:56.28 /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log 87116 - Is 0:00.01 nginx: master process /usr/local/sbin/nginx -c /var/e 87180 - S 11:42.70 nginx: worker process (nginx) 87731 - Is 0:02.54 /usr/sbin/cron -s 87748 - Is 0:00.00 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid / 88398 - Ss 1:04.33 /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var 88411 - Is 0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts 88796 - I 0:00.03 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.e 88909 - Is 0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_u 89308 - I 0:00.37 minicron: helper /usr/local/bin/ping_hosts.sh (minic 89639 - I 0:00.00 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.u 40954 u0 Is 0:00.04 login [pam] (login) 44424 u0 I 0:00.02 -sh (sh) 56005 u0 S+ 0:00.02 /bin/sh /etc/rc.initialI still find it quite interesting that when the network has clients and is busy, the problem goes away. But when the network goes idle, it starts up again. That in itself seems significant somehow...
-
Do you have any aliases setup? That might have some listing of what you think is IPs? and the filterdns mech is trying to resolve them because it thinks they are fqdn? filterdns out of the box would run every 5 minutes though.
This is clearly one of the weirdest things I have seen around here in a very long time, if not the most weirdest thing.. Its crazy..
-
@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:
Do you have any aliases setup? That might have some listing of what you think is IPs? and the filterdns mech is trying to resolve them because it thinks they are fqdn? filterdns out of the box would run every 5 minutes though.
Yes, I have a bunch a aliases in my FW config (they need to be updated, actually)
The "Ports" and "URLs" tabs are empty. Where does filterdns live? Do you mean the component of pfBlocker, or is this something else?This is clearly one of the weirdest things I have seen around here in a very long time, if not the most weirdest thing.. Its crazy..
Well, at least it's not some stupid thing I should have figured out myself...I wouldn't be surprised if I still did something wrong in config to cause this, though. I have a backup package from about 9 months ago. Not much has been changed in config since then, so I'm thinking of trying restoring to that, just in case that wipes out this problem. That's probably wishful thinking, but its a low-risk change.
-
Well filterdns runs every 5 minutes out of the box
[2.4.5-RELEASE][admin@sg4860.local.lan]/root: ps -ax | grep filterdns 61721 - Is 0:10.79 /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 300 -c /var/etc/filterdns.conf -d 1 17951 0 S+ 0:00.00 grep filterdnsYou could delete all of those.. Or when its doing it, you could kill off the filterdns process and see if it stops.
I guess its possible that something wrong where you think you have a alias setup as fqdn and not IP with an IP in it, that you might resolve that.. Let me do a little test, brb
What do you have being loaded by URL? I created a alias and then watched my unbound log, and not seeing any odd queries for it as a A or AAAA query. But guess its possible if your loading something via a url where something could get parsed wrong?
Post up what lists your using via url so I can take a look see, in your sniff I did see lists being loaded - but didn't notice any IPs that jumped out at me in them for what was being queried.
-
@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:
You could delete all of those.. Or when its doing it, you could kill off the filterdns process and see if it stops.
Interesting...I'll try that.
What do you have being loaded by URL? I created a alias and then watched my unbound log, and not seeing any odd queries for it as a A or AAAA query. But guess its possible if your loading something via a url where something could get parsed wrong?
I have nothing by way of aliases
But I'm using pfBlocker to pull some lists.Post up what lists your using via url so I can take a look see, in your sniff I did see lists being loaded - but didn't notice any IPs that jumped out at me in them for what was being queried.
Excluding the DNSBL lists, I am using the ones listed here: https://cloud.archnet.us/index.php/s/R2jXJzkJXBoXxkT
(I couldn't include them in this post. Apparently, I'm not allowed to have more than 1 URL in my posts? Or maybe it didn't like one of those URLs specifically? I don't know...the spam filter on this forum is aggressive.)Do you want my DNSBL lists too? I have many more of those, but I can gather them all if needed.
-
well this one
https://www.spamhaus.org/drop/asndrop.txtis a bunch of AS numbers - those would have to be resolved to IP ranges or CIDR blocks..
Why are you loading all these lists? You don't even have any port forwards setup? I just don't get the reason for trying to filter shit when you don't even have anything exposed?
-
@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:
well this one
https://www.spamhaus.org/drop/asndrop.txtis a bunch of AS numbers - those would have to be resolved to IP ranges or CIDR blocks..
Oh..that's interesting. It used to have the IP blocks themselves on that list. I'll remove that one. I should check through all of my lists...it has been almost two years since those were added.
Why are you loading all these lists? You don't even have any port forwards setup? I just don't get the reason for trying to filter shit when you don't even have anything exposed?
I don't have any ports forwarded right now although I did in the past. That is partially because I wasn't sure what was wrong on this network. If I regain confidence that nothing is vulnerable or compromised, I will probably open ports again. To be clear, I want to have open ports, but I'd prefer to make sure everything is shipshape before doing that again. It looks like this is probably a benign issue (although I still need to fix it), but I want to get it working properly first, just the same.
-
Vs trying to block the planet, if you are going to open up ports - its better to just limit what can talk to your ports. Example - I have plex open for friends and family. They are all in the US.. So I only allow US Ips to talk to that.. I don't load every possible bad guy list and block them - that is just insane to try and do that..
When I had someone outside the US, I added that country..
Sure in a perfect setup would only allow their specific IPs.. But then again they are users - asking them for their IP is like asking a monkey to explain theory of relativity.. And many of them use phones and travel, etc.
I would really if your not going to just nuke this thing from orbit and start over - which is what I would do.. Is turn off all this list shit..
You use bad guy lists, when that is the smaller list. Ie you want to serve up stuff to the globe - that is when you filter bad guy lists. Since that is smaller list ;)
-
@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:
Vs trying to block the planet, if you are going to open up ports - its better to just limit what can talk to your ports.
You use bad guy lists, when that is the smaller list. Ie you want to serve up stuff to the globe - that is when you filter bad guy lists. Since that is smaller list ;)At the risk of continuing down the age-old "block the planet" vs whitelist dispute, the problem is, I plan to have incoming connections from six continents. Whitelist management is going to be a much bigger headache than some low-maintenance blacklists.
I would really if your not going to just nuke this thing from orbit and start over - which is what I would do.. Is turn off all this list shit..
Yeah, I'm leaning that way. I reverted this unit to a backup from December 2019. After fixing a few small things, it seems to be humming along without issues so far. It does still have the aforementioned fragments of the Suricata package, but hopefully that's irrelevant. After about 4 days, I have yet to notice these garbage DNS queries, although I may have missed some.
-
@CyberMinion said in pfSense using unreasonable amount of bandwidth while idle:
I have yet to notice these garbage DNS queries, although I may have missed some.
You should have ZERO of those - there is NOTHING that should create dns queries like that.. They can not resolve - they are garbage..
-
I hate to bump this to the top, but I just wanted to let you know that I have not seen any of this bad behavior from the firewall since restoring from backup. Whatever it was, it seems to be fixed. I just wish I knew the root cause.
Thank you for your help, everyone!
-
Mmm, weird. Thanks for following up though. Yeah it would be good to know what it was but that's probably not possible at this point.
Steve
