pfSense using unreasonable amount of bandwidth while idle

johnpoz

That IP isn't even valid - its a bogon

185.190.88.0/22

CyberMinion

@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:

That IP isn't even valid - its a bogon

185.190.88.0/22

Good point. Bogons are set to be blocked by both the pfSense native rules, and pfBlocker. Interesting.

johnpoz

A few dns queries even using tls is not going to generate that sort of traffic..

Do a sniff on wan in promiscuous mode capturing all the traffic, not just 100 packets..

What is in front of pfsense? This 192.168.1.232 is not an IP your ISP would give you..

Also on you traffic graph, set the filter to all - so we can see where the traffic is coming from.. No just your IP. Or use the iftop package.. Where is this that amount of traffic coming from or going to - makes no sense that a few queries over tls could generate anywhere near that amount of constant traffic.

CyberMinion

@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:

A few dns queries even using tls is not going to generate that sort of traffic..

Do a sniff on wan in promiscuous mode capturing all the traffic, not just 100 packets..

Done. After filtering out the significant amount of DNS noise, I do still have a lot left over. For example, the upsteam NAT router is apparently pinging me a fair amount. I confirmed that wasn't anything being done manually by anyone on my end, although I can't imagine ICMP pongs justifying this load.
I'm not sure what I should be looking for here, though. Nothing else jumps out at me--any particular suggestions?

What is in front of pfsense? This 192.168.1.232 is not an IP your ISP would give you..

Correct, I thought I'd explained this, but in skimming back, I don't see it. The pfSense is behind a NAT router (192.169.1.1), which is mostly outside my control. It has some devices on it I don't control, so I wanted a barrier between me and them. The pfSesne firewall is on DHCP from that, currently with 192.168.1.232.

Also on you traffic graph, set the filter to all - so we can see where the traffic is coming from.. No just your IP.

Oh, right. the network is in use some now, so it's harder to get a clean graph...pardon the brief spikes. The "background noise" is still there though, clearly, which is somewhat unusual. Typically once the network starts getting more use, this noise goes away.

Or use the iftop package.. Where is this that amount of traffic coming from or going to - makes no sense that a few queries over tls could generate anywhere near that amount of constant traffic.

The only devices on this list which remain online through the night (when this noise typically starts) are 192.168.4.70 and 192.168.3.2 (SOHO NAT router). I masked one IP, since it is somewhat identifying. It is a trusted remote server.

Any thoughts on the processes/hardware utilization I mentioned earlier? Or is that a dead-end?

johnpoz

Well if you on some network in front of pfsense that noise could really be any sort of broadcast noise on the network..

How about you do a sniff when you are seeing all this traffic in promiscuous mode for say a minute or so while you seeing just that background noise.. And then send me link to where I can download the file in PM.. Or if you PM I can send you my email and you can mail it to me so I can take a look.

Pings are not going to be it - it would have to be a shiton of pints to equal that amount of bandwidth.. I mean a shiton!!!

Or look at the sniff and see where all the bandwidth is, by looking at the conversations. etc..

Pfsense really shouldn't be sending anything back though... do you have ports open on your wan? post up your wan rules.

Your largest amount of traffic there you have hidden.. But you have traffic to that 74.125.172.57, which is a google IP on 443.. that isn't their dns IP..

I show these pointing to that IP.

    r3.sn-ab5szn7s.googlevideo.com
    r3.sn-ab5szn7s.gvt1.com
    r3.sn-ab5szn7s.a1.googlevideo.com
    r3.sn-ab5szn7s.c.youtube.com
    r3.sn-ab5szn7s.c.2mdn.net
    r3.sn-ab5szn7s.c.drive.google.com
    r3.sn-ab5szn7s.c.mail.google.com
    r3.sn-ab5szn7s.c.android.clients.google.com
    r3.sn-ab5szn7s.c.doc-0-0-sj.sj.googleusercontent.com

CyberMinion

@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:

Well if you on some network in front of pfsense that noise could really be any sort of broadcast noise on the network..

How about you do a sniff when you are seeing all this traffic in promiscuous mode for say a minute or so while you seeing just that background noise.. And then send me link to where I can download the file in PM.. Or if you PM I can send you my email and you can mail it to me so I can take a look.

Will do, thanks! I have a dirty one now, but I'll wait until tomorrow morning to capture a cleaner copy.

Pings are not going to be it - it would have to be a shiton of pints to equal that amount of bandwidth.. I mean a shiton!!!

No kidding! Not even the Ping of Death used that much bandwidth.

Or look at the sniff and see where all the bandwidth is, by looking at the conversations. etc..

I've been trying, but I'm not that good at deciphering pcaps.

Pfsense really shouldn't be sending anything back though... do you have ports open on your wan? post up your wan rules.

None open, although UPnP (yuck) is enabled right now. I've disabled it twice, but due to a different problem, it has come back twice. I could disable it again, but that might make troubleshooting more difficult. Any time I check, it does not have any ports open. Remember too that this is behind another NAT, which has no open ports on it.

Your largest amount of traffic there you have hidden.. But you have traffic to that 74.125.172.57, which is a google IP on 443.. that isn't their dns IP..

I show these pointing to that IP.
    r3.sn-ab5szn7s.googlevideo.com
    r3.sn-ab5szn7s.gvt1.com
    r3.sn-ab5szn7s.a1.googlevideo.com
    r3.sn-ab5szn7s.c.youtube.com
    r3.sn-ab5szn7s.c.2mdn.net
    r3.sn-ab5szn7s.c.drive.google.com
    r3.sn-ab5szn7s.c.mail.google.com
    r3.sn-ab5szn7s.c.android.clients.google.com
    r3.sn-ab5szn7s.c.doc-0-0-sj.sj.googleusercontent.com

I have an android OS running behind the internal NAT router, so I assume this is just it doing creepy Google things.

johnpoz

If you have no ports open - then all of those rules are POINTLESS!!!

Your also behind a nat... So how do you think something from Asia is going to even get to you? All of those rules have zero hits...

That is not all the rules? For why if you have no port forwards... Default is deny, anything that is no a reply would be dropped hitting your wan.. Unsolicited no matter from where is dropped.. Those rules don't stop something behind pfsense from talking to something..

And again your behind a nat - do you have stuff forwarded to your 192.168 address.. How are you doing that you said your not in control of that router in front of you..

My point about that IP, is none of those are anything pfsense would generate traffic too.. It would be some client watching youtube or checking their mail for traffic to go there.

Maybe google drive?? A sync or watching a video would for sure account for such traffic..

stephenw10

Yeah, I was just reading and thinking that....

If you don't have a pass rule on WAN for something then those inbound pfBlocker rules are just loading up the firewall unnecessarily. All that is blocked by default anyway.

CyberMinion

@stephen10
@johnpoz

If you have no ports open - then all of those rules are POINTLESS!!!

I figured I'd get yelled at for that. Yes, they are pointless right now. In the past, a port was open on the exterior NAT, and on the pfsense. That port has since been closed, but may be opened again in the future. For ease of use, I left these rules in place.

Believe it or not, I do know how NAT works.

And again your behind a nat - do you have stuff forwarded to your 192.168 address.. How are you doing that you said your not in control of that router in front of you..

I'm not in control of it, but ports have been forwarded to me by someone else on occasion, for specific purposes.

My point about that IP, is none of those are anything pfsense would generate traffic too.. It would be some client watching youtube or checking their mail for traffic to go there.

Maybe google drive?? A sync or watching a video would for sure account for such traffic..

When this problem starts (overnight) the only things which should be happening are on a single computer:

AV - Since it is largely ignored, it has automated AV on it which may do updates ovnight
0patch - since it is a Win7 device, 0patch is running on it to "micro patch" some vulnerabilities. This may do updates overnight.
A custom script, which monitors a web server. Once in a great while, it will interact briefly with that server, but generally it just checks on it periodically and logs what it sees.
NextCloud Sync Client (Files are very rarely changed, so this incurs a nominal amount of load)

On average, this computer uses less that 1Kb/s of bandwidth, but will occasionally increase to as much a 5Kb/s, and even spike to around 2Mb/s for a few seconds, a couple times a week. As shown earlier, my problem involves a nominal amount of traffic moving through the LAN and OPT ports, while a sizable amount is moving through the WAN port.

johnpoz

How is this rule not just completely pointless? You have a ipv4 rule with a IPv6 source of addresses.. Be it your going to open a port or not?

In your traffic graph or iptop, or darkstat -- look to see where the remote non pfsense IPs are in that traffic flow.. Its not insignificant - so should be easy enough to spot.. Clear your counters..

Now look in your state tables.. Pfsense would have zero reason to be sending traffic to that IP listed for example.. You hid the other one.. so have no idea what that is.. But that ip listed that has some traffic going to it - top of your list.. Is not something pfsense would ever talk to on its own from any sort of normal setup. And sure is not your dns your forwarding too.. So what is creating that traffic, its not pfsense. Unless its maybe something downloading some list? It is 443 traffic. And from pfsense top - it would be this 192.168.3.2 generating the traffic!! See the pftop exact same packet counts..

The only thing shown not matched up is thos pings from 192.168.1.1 - which pfsense would ping if that is its gateway.. But default is to send 0 size pings, and only 2 a second they would not account for 200k of data.

If it was just noise hitting your wan, there wouldn't be outbound traffic.. Pfsense if no ports open or actual reject rules does not answer noise.. its just dropped..

Cool_Corona

Using SSL/TLS encrypted DNS via 9.9.9.9??

Disable it and use the local resolver and delete all DNS entries under general -> setup.

Reboot.

Let us know.

CyberMinion

@johnpoz said in pfSense using unreasonable amount of bandwidth while idle:

How is this rule not just completely pointless? You have a ipv4 rule with a IPv6 source of addresses.. Be it your going to open a port or not?

Yup, that would be useless. I just corrected it to IPv6--thanks.

In your traffic graph or iptop, or darkstat -- look to see where the remote non pfsense IPs are in that traffic flow.. Its not insignificant - so should be easy enough to spot.. Clear your counters..

Now look in your state tables.. Pfsense would have zero reason to be sending traffic to that IP listed for example.. You hid the other one.. so have no idea what that is.. But that ip listed that has some traffic going to it - top of your list.. Is not something pfsense would ever talk to on its own from any sort of normal setup. And sure is not your dns your forwarding too.. So what is creating that traffic, its not pfsense. Unless its maybe something downloading some list? It is 443 traffic. And from pfsense top - it would be this 192.168.3.2 generating the traffic!! See the pftop exact same packet counts..

Yes, but keep in mind, that includes the traffic load from the daytime, when PCs behind it are in use. Of course those entries have a lot of traffic...this is not just an idle lab environment. If I reboot the pfsense to reset these entries, the unwanted "noise" will also go away. The only way I can troubleshoot this is with dirty data. For example, the IP I masked was seeing moderate to heavy traffic for about 10 hours, due to user activity within the network. Then it stopped. Then a few hours later, pfsense suddenly got very talkative, but that did not outweigh the user activity from earlier.

The only thing shown not matched up is thos pings from 192.168.1.1 - which pfsense would ping if that is its gateway.. But default is to send 0 size pings, and only 2 a second they would not account for 200k of data.

If it was just noise hitting your wan, there wouldn't be outbound traffic.. Pfsense if no ports open or actual reject rules does not answer noise.. its just dropped..

Exactly, that's why I started this thread in the first place. Only after the pfsense's network goes quiet, does the pfSesne start communicating a great deal with the outside. Based on the utilization of each physical port, I can see that it is not coming from inside the network, so it must be the pfsense itself. Should it be doing this? No, I don't think so. Can I figure out why it might be? No, or I wouldn't be asking here. It is always possible that something is misconfigured, but I don't have any ideas what. Since my WAN ports are closed and I'm seeing 2-way traffic, I must surmise that pfSense is initiating this activity, and based on what we have discovered since then, it looks like the activity is being initiated with my upstream DNS.

As a side note, I noticed most of this traffic was going between my secondary DNS provider, which I figured meant that the primary couldn't give an answer, so the pfSense tried the secondary. On a whim, I replaced my secondary (formerly IBM's Quad-9) to CloudFlare's secondary (1.0.0.1). Although this noise doesn't always start every night, it has been almost every night. Last night though, it didn't. At this point it doesn't conclusively mean anything, but I'm going to check back on it in a while, and see if the noise returns. It would be surprising if this actually "fixed" the problem, but I'll give it some time and see.

CyberMinion

@Cool_Corona said in pfSense using unreasonable amount of bandwidth while idle:

Using SSL/TLS encrypted DNS via 9.9.9.9??

Disable it and use the local resolver and delete all DNS entries under general -> setup.

Reboot.

Let us know.

DoH is currently disabled, but I did not clear the entries. I will try that soon, but as I just posted above, I also just cut Quad-9 out of the provider list. I don't want to try too many things at once. I'm also trying to get a clean pcap of the noise for johnpoz.

Thanks for the tip!

CyberMinion

So pfBlocker isn't supposed to be doing DNS lookups, as stated earlier. However, it occurs to me that the DNSBL entries can be filtered against to Alexa top sites...could this mechanic result in any DNS queries?

johnpoz

So did turning of pfblocker stop the problem.. been a few days..

CyberMinion

@johnpoz
I was still having this issue after flipping off the pfBlocker switch, but I did not reboot. I have rebooted the pfsense with pfBlocker disabled, and will monitor it to see how things go.

CyberMinion

Yes, after a reboot with pfblocker off, I am still seeing this DNS rubbish going on.

johnpoz

Well can tell you for sure that pfsense out of the box is not going to query "185.190.088.000" as a A query..

You have something asking pfsense for that.. Or something running on pfsense doing dns query to loopback asking for that.

Even if pfsense was asked to resolve an IP, say in the logs or something - it would do a PTR for it..

Example... I enabled logging in unbound. So now I get all queries asked for logged..

server:
log-queries: yes

I then go into say the firewall log and ask pfsense to tell me hey who is this IP hitting my server.

by clicking the little "i" it then tries to resolve that.. Via a PTR query.. Not a A query for some name with that the IP as the name.. Also why and the F would be asking for what is clearly a bogon..

What other things do you have installed on pfsense? Or what do you have doing queries to pfsense that your not seeing the query come in? A dns redirect setup would look like loopback is asking for the record.

The only thing pfsense itself would be asking for is hey I need to get the package list from netgate, hey lets check if there is a update, or hey I have a widget on the home page and need to pull the RSS feed, etc.

Can tell you for sure if there was something in pfsense out of the box causing this traffic, or even something common... The forums would be blowing up with hey why is there 300K of traffic all the time...

CyberMinion

@johnpoz
Well, I don't have RSS feeds, or other such junk going on in the box. and the lookups are showing the source being the loopback. Here are the packages I have installed:

I haven't (side)loaded any other software outside of the package manager, or installed any of my own scrips. I formerly had Suricata installed also, but it wasn't working due to RUST, so I uninstalled it. Although the package manager shows it gone, I still have a few traces of it, indicating an incomplete uninstall.

I also previously installed then uninstalled Acme. To my knowledge, that was a clean uninstall

Can tell you for sure if there was something in pfsense out of the box causing this traffic, or even something common... The forums would be blowing up with hey why is there 300K of traffic all the time...

Yeah, instead we have one single, ridiculously long thread...ugh.

So what now? Would it make any difference at all if I take a full backup, do a factory reset, then restore from the backup? I have a feeling that would bring the problem right back. I don't want to reconfigure from scratch, though.

johnpoz

well iftop or nmap sure wouldn't be doing it, they don't even run anything unless you run them.

I guess snort could be be doing it? In theory - it could try and look for IPs that hit your wan..

Thought you said you removed pfblocker? I would remove them completely, both pfblocker and snort.. Just to be 110% sure its not them..

But what I can tell you for sure is there nothing in pfsense out of the box that would try and look those such fqdn..

What I would prob do, is do a clean install of pfsense with min sort of setup, an out of the box sort of setup.. And do you get such queries now?

I can tell you for sure my setup isn't doing anything of the sort.. I have logging of all queries turned on in unbound, and see nothing even close to those nonsense lookups.