pfSense using unreasonable amount of bandwidth while idle
-
@johnpoz Ah, I see.
-
To be honest if what you want to make sure pfsense queries are using tls to talk to 1.1.1.1 or quad 9.. You have to do the forwards in the unbound resolver options box.. Because when setup this way if for some reason pfsense own dns client goes to ask 1.1.1.1 or quad9 it will just do a normal dns query and will not use tls.
So from your log you show that loopback is asking for that weird A query.. Out of the box pfsense would not ask for that.. It only checks to see if there is any updates. From like what I showed you, or if it trying to resolve something via some other process running on it, like a package.. Filterdns would ask for stuff if you setup any aliases, etc.
It would look to download bogon, etc..
The trick is trying to figure out where that query is coming from.. Off the top I do no know a way to monitor all internal processes for what they query for..
That sort of tracking has always been tricky no matter the os, be it windows, linux or a bsd..
-
You have to do the forwards in the unbound resolver options box..
I am, but before I enabled DoH, I added these DNS providers to the standard resolver, to use them in the clear (both are much faster than my ISP's DNS). Since then, I've run tests and do see DoH traffic, not regular UDP, so I assume those are no being used. Here is what I have in my custom box right now:
forward-zone: name: "." forward-ssl-upstream: yes forward-addr: 1.1.1.1@853 forward-addr: 9.9.9.9@853 server:include: /var/unbound/pfb_dnsbl.*conf server: log-queries: yes log-replies: yesThe trick is trying to figure out where that query is coming from
Agreed. Other than the OS itself doing the basic stuff, the only other lookups I would expect to see from this box are from pfBlocker, which uses numerous block lists which it updates periodically. This process certainly shouldn't be doing what we are seeing happen, but it is the only other main source of queries I can think of. It is also doing Geo blocking (yes, I know...it probably isn't really that helpful in most circumstances) so it needs to update those IP lists periodically.
-
Yeah something is not right because that is not really a valid query.. A query for an IP should be a PTR query.. And a A record like that would really never resolve because that last part would be considered the tld.. and .072 sure isn't a valid tld..
Now your 066.140.x.x.localdomain could resolve.. if the local NS you were asking had a record like that in it.. but public would never resolve that sort of query.
Maybe look through your xml, you can download a full backup and search for anything that could be put in somewhere that matches that odd query.
edit: BTW your not seeing any other odd queries like that? Just that 1 weird one that repeats over and over?
Are you seeing any firewall logs with that IP in it? Or that IP reversed, where maybe the PTR got messed up? so like 72.140.159.66 or 66.159.140.72 ?
-
My apologies for taking so long to respond. Some other things we demanding my time.
@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:
Now your 066.140.x.x.localdomain could resolve.. if the local NS you were asking had a record like that in it.. but public would never resolve that sort of query.
My local resolver does not have any entries like that.
Maybe look through your xml, you can download a full backup and search for anything that could be put in somewhere that matches that odd query.
I'm not exactly sure what to look for, but I found no references to that address in the XML
edit: BTW your not seeing any other odd queries like that? Just that 1 weird one that repeats over and over?
Pretty much, although I think I have seen one or two other addresses on rare occasions. I ran another pcap, but forgot I had DoH on, so it doesn't offer many insights...just that it was communicating a great deal with both DNS providers. (I can hear your "I told you so"'s already...)
I forgot to grab the DNS log at the time.Are you seeing any firewall logs with that IP in it? Or that IP reversed, where maybe the PTR got messed up? so like 72.140.159.66 or 66.159.140.72 ?
Well, I haven't checked recently. I will do that next time. My log is currently swamped with some IPv6 address pounding away on it with UDP. (odd, since this is behind another firewall...)
By the way, I tried disabling pfblocker while this noise was going on, and it didn't resolve the problem. I didn't restart the device though, since that would temporarily "solve" the issue anyway. When I turned off pfblocker, here is what happened for network activity.
Then it resumed the background noise again. I did think it was a little interesting that the download speed actually dipped as the upload spiked. Maybe that's just due to a network bottleneck though. Also, why so much upload for this?Upon re-enabling pfblocker, the same basic thing happened.
I doubt that is significant, but I figured I'd share it anyway.
Also, FYI here is a "baseline" of sorts, from later in the day while it has 4 computers semi-active on it.
Note that it has quieted down. -
So link local looking for something via LLMR (5355)..
Dude sorry but the couple of packets you list in you sniff does not account for 200K of traffic..
You got something going on that your not catching in the sniff or you not showing ups the details of capture.. Are you seeing 1000's of queries and your just showing us the 1?
How about you turn off that doh shit and see if your issue goes away? But for doh to generate that amount of traffic there would have to be way more queries than your showing.
-
@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:
So link local looking for something via LLMR (5355)..
I found nothing in that regard
Dude sorry but the couple of packets you list in you sniff does not account for 200K of traffic..
Agreed, I don't seem to be seeing enough packets to explain the bandwidth consumption.
You got something going on that your not catching in the sniff or you not showing ups the details of capture.. Are you seeing 1000's of queries and your just showing us the 1?
No, I just don't seem to be seeing very many packets in my pcaps. The DNS resolver was generating 1000 log entries about every 6 seconds though.
How about you turn off that doh shit and see if your issue goes away? But for doh to generate that amount of traffic there would have to be way more queries than your showing.
I did that a month or so ago, and the problem was still occurring. I turned it off again though, for argument's sake. Someone upstream of me will be very happy...
By the way, last night it was looping a request over and over again for 185.190.088.000.MyNetworkName It was mostly requesting the AAAA record, which is slightly interesting since IPv6 traffic is blocked on the pfSense.
-
@CyberMinion said in pfSense using unreasonable amount of bandwidth while idle:
It was mostly requesting the AAAA record
That has nothing to do with if IPv6 is enabled or not..
What was asking for that 185.190.088.000.MyNetworkName - your saying pfsense was asking itself for that.. or some client on your network was asking for it?
Do you have any sort of haproxy setup with a backend check. Or the old loadbalancer setup?
-
Hi @CyberMinion - Does using Diagnostics -> pfTop reveal anything interesting? For instance, try using View = "speed" and Sort By = "Bytes"
Hope this helps.
-
@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:
What was asking for that 185.190.088.000.MyNetworkName - your saying pfsense was asking itself for that.. or some client on your network was asking for it?
It was being requested by the loopback address on the pfsense - 127.0.0.1
Do you have any sort of haproxy setup with a backend check. Or the old loadbalancer setup?
No, nothing so fancy.
@tman222
Good idea, but I'm not seeing anything which particularly stands out to me. I see a bunch of idle connections to by backup DNS provider, and while I don't see any connections to my primary DNS IP (1.1.1.1)listed, I am seeing connections to two other CloudFlare IPs, so maybe the requests got load balanced off to other IPs.
By the way, is it normal to have over 130 processes running on this this FreeDSD/pfSense build? I honestly don't know, but it seems a little high to me.
When I run ps I only see this:
PID TT STAT TIME COMMAND 5340 u0 I+ 0:00.02 /bin/sh /etc/rc.initial 56616 u0- IN 0:35.73 /bin/sh /var/db/rrd/updaterrd.sh 87977 u0 Is 0:00.04 login [pam] (login) 95417 u0 I 0:00.02 -sh (sh)but in monitoring, I see this:
Note also that there was a PC actively doing stuff behind this firewall until a little after 3am. It automatically powered down, leaving just two idle service computers running. Almost immediately, the "user utilization" increased markedly.
-
That IP isn't even valid - its a bogon
185.190.88.0/22
-
@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:
That IP isn't even valid - its a bogon
185.190.88.0/22
Good point. Bogons are set to be blocked by both the pfSense native rules, and pfBlocker. Interesting.
-
A few dns queries even using tls is not going to generate that sort of traffic..
Do a sniff on wan in promiscuous mode capturing all the traffic, not just 100 packets..
What is in front of pfsense? This 192.168.1.232 is not an IP your ISP would give you..
Also on you traffic graph, set the filter to all - so we can see where the traffic is coming from.. No just your IP. Or use the iftop package.. Where is this that amount of traffic coming from or going to - makes no sense that a few queries over tls could generate anywhere near that amount of constant traffic.
-
@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:
A few dns queries even using tls is not going to generate that sort of traffic..
Do a sniff on wan in promiscuous mode capturing all the traffic, not just 100 packets..
Done. After filtering out the significant amount of DNS noise, I do still have a lot left over. For example, the upsteam NAT router is apparently pinging me a fair amount. I confirmed that wasn't anything being done manually by anyone on my end, although I can't imagine ICMP pongs justifying this load.
I'm not sure what I should be looking for here, though. Nothing else jumps out at me--any particular suggestions?
What is in front of pfsense? This 192.168.1.232 is not an IP your ISP would give you..
Correct, I thought I'd explained this, but in skimming back, I don't see it. The pfSense is behind a NAT router (192.169.1.1), which is mostly outside my control. It has some devices on it I don't control, so I wanted a barrier between me and them. The pfSesne firewall is on DHCP from that, currently with 192.168.1.232.
Also on you traffic graph, set the filter to all - so we can see where the traffic is coming from.. No just your IP.
Oh, right. the network is in use some now, so it's harder to get a clean graph...pardon the brief spikes. The "background noise" is still there though, clearly, which is somewhat unusual. Typically once the network starts getting more use, this noise goes away.
Or use the iftop package.. Where is this that amount of traffic coming from or going to - makes no sense that a few queries over tls could generate anywhere near that amount of constant traffic.
The only devices on this list which remain online through the night (when this noise typically starts) are 192.168.4.70 and 192.168.3.2 (SOHO NAT router). I masked one IP, since it is somewhat identifying. It is a trusted remote server.Any thoughts on the processes/hardware utilization I mentioned earlier? Or is that a dead-end?
-
Well if you on some network in front of pfsense that noise could really be any sort of broadcast noise on the network..
How about you do a sniff when you are seeing all this traffic in promiscuous mode for say a minute or so while you seeing just that background noise.. And then send me link to where I can download the file in PM.. Or if you PM I can send you my email and you can mail it to me so I can take a look.
Pings are not going to be it - it would have to be a shiton of pints to equal that amount of bandwidth.. I mean a shiton!!!
Or look at the sniff and see where all the bandwidth is, by looking at the conversations. etc..
Pfsense really shouldn't be sending anything back though... do you have ports open on your wan? post up your wan rules.
Your largest amount of traffic there you have hidden.. But you have traffic to that 74.125.172.57, which is a google IP on 443.. that isn't their dns IP..
I show these pointing to that IP.
r3.sn-ab5szn7s.googlevideo.com r3.sn-ab5szn7s.gvt1.com r3.sn-ab5szn7s.a1.googlevideo.com r3.sn-ab5szn7s.c.youtube.com r3.sn-ab5szn7s.c.2mdn.net r3.sn-ab5szn7s.c.drive.google.com r3.sn-ab5szn7s.c.mail.google.com r3.sn-ab5szn7s.c.android.clients.google.com r3.sn-ab5szn7s.c.doc-0-0-sj.sj.googleusercontent.com -
@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:
Well if you on some network in front of pfsense that noise could really be any sort of broadcast noise on the network..
How about you do a sniff when you are seeing all this traffic in promiscuous mode for say a minute or so while you seeing just that background noise.. And then send me link to where I can download the file in PM.. Or if you PM I can send you my email and you can mail it to me so I can take a look.
Will do, thanks! I have a dirty one now, but I'll wait until tomorrow morning to capture a cleaner copy.
Pings are not going to be it - it would have to be a shiton of pints to equal that amount of bandwidth.. I mean a shiton!!!
No kidding! Not even the Ping of Death used that much bandwidth.
Or look at the sniff and see where all the bandwidth is, by looking at the conversations. etc..
I've been trying, but I'm not that good at deciphering pcaps.
Pfsense really shouldn't be sending anything back though... do you have ports open on your wan? post up your wan rules.
None open, although UPnP (yuck) is enabled right now. I've disabled it twice, but due to a different problem, it has come back twice. I could disable it again, but that might make troubleshooting more difficult. Any time I check, it does not have any ports open. Remember too that this is behind another NAT, which has no open ports on it.
Your largest amount of traffic there you have hidden.. But you have traffic to that 74.125.172.57, which is a google IP on 443.. that isn't their dns IP..
I show these pointing to that IP.
r3.sn-ab5szn7s.googlevideo.com r3.sn-ab5szn7s.gvt1.com r3.sn-ab5szn7s.a1.googlevideo.com r3.sn-ab5szn7s.c.youtube.com r3.sn-ab5szn7s.c.2mdn.net r3.sn-ab5szn7s.c.drive.google.com r3.sn-ab5szn7s.c.mail.google.com r3.sn-ab5szn7s.c.android.clients.google.com r3.sn-ab5szn7s.c.doc-0-0-sj.sj.googleusercontent.comI have an android OS running behind the internal NAT router, so I assume this is just it doing creepy Google things.
-
If you have no ports open - then all of those rules are POINTLESS!!!
Your also behind a nat... So how do you think something from Asia is going to even get to you? All of those rules have zero hits...
That is not all the rules? For why if you have no port forwards... Default is deny, anything that is no a reply would be dropped hitting your wan.. Unsolicited no matter from where is dropped.. Those rules don't stop something behind pfsense from talking to something..
And again your behind a nat - do you have stuff forwarded to your 192.168 address.. How are you doing that you said your not in control of that router in front of you..
My point about that IP, is none of those are anything pfsense would generate traffic too.. It would be some client watching youtube or checking their mail for traffic to go there.
Maybe google drive?? A sync or watching a video would for sure account for such traffic..
-
Yeah, I was just reading and thinking that....
If you don't have a pass rule on WAN for something then those inbound pfBlocker rules are just loading up the firewall unnecessarily. All that is blocked by default anyway.
-
@stephen10
@johnpozIf you have no ports open - then all of those rules are POINTLESS!!!
I figured I'd get yelled at for that. Yes, they are pointless right now. In the past, a port was open on the exterior NAT, and on the pfsense. That port has since been closed, but may be opened again in the future. For ease of use, I left these rules in place.
Believe it or not, I do know how NAT works.
And again your behind a nat - do you have stuff forwarded to your 192.168 address.. How are you doing that you said your not in control of that router in front of you..
I'm not in control of it, but ports have been forwarded to me by someone else on occasion, for specific purposes.
My point about that IP, is none of those are anything pfsense would generate traffic too.. It would be some client watching youtube or checking their mail for traffic to go there.
Maybe google drive?? A sync or watching a video would for sure account for such traffic..
When this problem starts (overnight) the only things which should be happening are on a single computer:
- AV - Since it is largely ignored, it has automated AV on it which may do updates ovnight
- 0patch - since it is a Win7 device, 0patch is running on it to "micro patch" some vulnerabilities. This may do updates overnight.
- A custom script, which monitors a web server. Once in a great while, it will interact briefly with that server, but generally it just checks on it periodically and logs what it sees.
- NextCloud Sync Client (Files are very rarely changed, so this incurs a nominal amount of load)
On average, this computer uses less that 1Kb/s of bandwidth, but will occasionally increase to as much a 5Kb/s, and even spike to around 2Mb/s for a few seconds, a couple times a week. As shown earlier, my problem involves a nominal amount of traffic moving through the LAN and OPT ports, while a sizable amount is moving through the WAN port.
-
How is this rule not just completely pointless? You have a ipv4 rule with a IPv6 source of addresses.. Be it your going to open a port or not?
In your traffic graph or iptop, or darkstat -- look to see where the remote non pfsense IPs are in that traffic flow.. Its not insignificant - so should be easy enough to spot.. Clear your counters..
Now look in your state tables.. Pfsense would have zero reason to be sending traffic to that IP listed for example.. You hid the other one.. so have no idea what that is.. But that ip listed that has some traffic going to it - top of your list.. Is not something pfsense would ever talk to on its own from any sort of normal setup. And sure is not your dns your forwarding too.. So what is creating that traffic, its not pfsense. Unless its maybe something downloading some list? It is 443 traffic. And from pfsense top - it would be this 192.168.3.2 generating the traffic!! See the pftop exact same packet counts..
The only thing shown not matched up is thos pings from 192.168.1.1 - which pfsense would ping if that is its gateway.. But default is to send 0 size pings, and only 2 a second they would not account for 200k of data.
If it was just noise hitting your wan, there wouldn't be outbound traffic.. Pfsense if no ports open or actual reject rules does not answer noise.. its just dropped..
