• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

pfSense using unreasonable amount of bandwidth while idle

Scheduled Pinned Locked Moved General pfSense Questions
106 Posts 9 Posters 18.7k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • C
    CyberMinion
    last edited by Jul 26, 2020, 7:41 PM

    Hello,
    I am using an sg-1100, to protect a small number of devices from an untrused LAN. The pfSense (v2.4.5-RELEASE-p1) on this device is running the basic services, plus DNS resolver, pfBlocker and SNORT. I've been having a few strange problems, (like config changes almost always being lost after I apply them) but another I am now noticing is that the sg-1100 is using a significant amount of bandwidth while it should be idle.

    While running with one idle PC (LAN port), one idle Raspberry Pi (LAN port), and one idle WiFi router with no connections (OPT port) on this box, I see both the LAN port:
    LAN normal.jpg
    And the OPT port:OPT normal.jpg

    Consuming a reasonable amount of traffic. However, the entire time, the WAN port is moving a lot of data, non-stop:
    WAN stable.jpg

    As I watched it for about an hour, I noticed two brief disruptions to this norm:
    WAN brief break.jpg WAN brief break then spike.jpg

    But other than that, it remained constant. I first began investigating this because the LAN and OPT link lights have been staying mostly solid, but the WAN port's link light is always flashing rapidly.

    Does anyone have suggestions what is going on? Is there some justification for this? Or has it been compromised? It's more ingress than egress traffic, which is slightly less concerning that the opposite ratio, but it still doesn't seem right.

    To be clear, no one other than me uses this device, or connects into it from the LAN/OPT sides for any reason whatsoever.

    Thanks!

    N 1 Reply Last reply Jul 26, 2020, 8:06 PM Reply Quote 0
    • N
      netblues @CyberMinion
      last edited by Jul 26, 2020, 8:06 PM

      @CyberMinion Change graph to remote and see...

      1 Reply Last reply Reply Quote 0
      • S
        stephenw10 Netgate Administrator
        last edited by Jul 27, 2020, 11:42 AM

        What is that host: 192.168.1.232? The WAN IP?

        Check the state tables for what's open there.

        Run a packet capture, see what that traffic actually is.

        Steve

        1 Reply Last reply Reply Quote 0
        • C
          CyberMinion
          last edited by CyberMinion Jul 27, 2020, 6:57 PM Jul 27, 2020, 6:48 PM

          @netblues

          Change graph to remote and see...

          That is showing "All" but remote does not change the readout much in this case. This is behind a NAT, so 192.168.1.x is in this case the WAN subnet.

          @stephenw10

          What is that host: 192.168.1.232? The WAN IP?

          Yes, that is the WAN IP of this pfSense firewall.

          Check the state tables for what's open there.

          Right now, a whole lot, but that is to be expected. That's a good suggestion, and I will check on that next time this issues arises.

          Run a packet capture, see what that traffic actually is.

          Also an excellent idea, but I'm not set up for that. I'll need to investigate whether I can do this right from pfSense, or need to hook up a tap. EDIT: Just found diag_packet_capture.php and will try to use that next time this happens.

          Now that I've started poking at it, the issue has vanished, but I'm sure it will return. It did this to me once before as well. When I notice the issue and first start poking at it a little bit, it keeps going, but if I do much or start a bandwidth-intensive process on the LAN, this unexplained load suddenly vanishes. Not sure if this is on the up-and-up (just prioritizing user activity over whatever this is), or an IOC.

          Probably unrelated, but I think this most recent occurrence may have started around 3am (my time), which is when I think I set pfBlocker to run. There's no good reason for pfBlocker to be both downloading and uploading many GB of data for the next 12 hours after triggering, though. I checked on it shortly after 3am today, but nothing strange was going on. All is quiet for now.r

          1 Reply Last reply Reply Quote 0
          • S
            stephenw10 Netgate Administrator
            last edited by Jul 27, 2020, 9:17 PM

            Indeed pfBlocker updates are much smaller and much faster.

            More likely you have something open on WAN that is being sent traffic and probably shouldn't be, DNS or NTP maybe?

            You have a VPN on there at all? Could be something remote connecting to it and downloading. You might expect the traffic to be more symmetric if it was though.

            Steve

            1 Reply Last reply Reply Quote 0
            • B
              bmeeks
              last edited by bmeeks Jul 27, 2020, 10:12 PM Jul 27, 2020, 10:09 PM

              Maybe you have a Skynet node in your LAN and it is plotting a world-wide takeover with other IoT devices across the planet by scheming to exterminate all the carbon units infesting Earth ... 😁.

              Just kidding with the Terminator movie reference. Also saw the 1979 Star Trek original movie again during the COVID lockdown, so the "carbon units" reference was fresh in my mind.

              Exactly what types of devices are on this "untrusted" LAN? Are they IoT type devices? Lots of those chit-chat with the mothership for various reasons. Some innocuous, and some -- "who knows?". That's not a ton of traffic. The resolution is bits/second in those graph screen grabs. So while not silent, the interfaces are not exactly maxed out either.

              As has already been suggested, doing a WAN packet capture using the tools under the DIAGNOSTICS menu might be helpful.

              1 Reply Last reply Reply Quote 0
              • C
                CyberMinion
                last edited by Jul 28, 2020, 2:29 AM

                This post is deleted!
                1 Reply Last reply Reply Quote 0
                • C
                  CyberMinion
                  last edited by Jul 28, 2020, 3:32 AM

                  Bloody heck...it is so hard to post messages on this forum. It's always being blocked by one rule or another.
                  @stephenw10

                  More likely you have something open on WAN that is being sent traffic and probably shouldn't be, DNS or NTP maybe

                  I have two internal DNS resolvers, one on the pfSense, and one on its LAN (don't ask...). The one on the LAN uses the pfSense as its upstream DNS server, and that uses CloudFlare (DoH). Neither one is (or should be) publicly exposed. There shouldn't be any NTP stuff going on.

                  You have a VPN on there at all? Could be something remote connecting to it and downloading. You might expect the traffic to be more symmetric if it was though.

                  No, VPN service on the pfSense is disabled, and unless something on the "trusted" LAN is compromised, there shouldn't typically be anything inside making or receiving VPN connections.

                  @bmeeks

                  Maybe you have a Skynet node in your LAN

                  Haha! You never know, these days! 😁

                  Exactly what types of devices are on this "untrusted" LAN? Are they IoT type devices?

                  No, there are no IoT devices on the "WAN", but a few old, creaky devices I have no control over. That's why the pfSense is between that stuff and me. There are a few PCs and an antique smartphone that I know of over there. However, that traffic should never be hitting my WAN port, much less flowing through it. If it does get sent my way for some reason, the firewall should block that.

                  The resolution is bits/second in those graph screen grabs. So while not silent, the interfaces are not exactly maxed out either.

                  Correct, but 300Kb/s is still a lot of data for the pfSense to be downloading non-stop, while the other two ports are idle. Where is that data going? And where is the uploaded data coming from?

                  doing a WAN packet capture using the tools under the DIAGNOSTICS menu might be helpful.

                  Agreed, I wasn't aware that feature was baked in, but I certainly hope to get a capture next time. Oddly enough, all is still quiet in this regard. Maybe I poked at it too much.

                  1 Reply Last reply Reply Quote 0
                  • S
                    stephenw10 Netgate Administrator
                    last edited by Jul 28, 2020, 10:40 AM

                    I agree, there's nothing in pfSense itself I would expect to behave like that. A pcap would tell all.

                    C 1 Reply Last reply Aug 3, 2020, 4:14 AM Reply Quote 0
                    • C
                      CyberMinion @stephenw10
                      last edited by Aug 3, 2020, 4:14 AM

                      @stephenw10 I was preoccupied for a bit there, but I just noticed this going on again, and got a couple pcaps.

                      Here's the weird thing...virtually all of this traffic is between my DoH providers and the pfSense. It is mostly my backup DNS provider in this case (IBM) but I see a little bit of my primary provider (Cloudflare) in there. I'm not sure If I should go publishing the pcaps on here publicly, but do you want me to DM them to you?

                      N 1 Reply Last reply Aug 3, 2020, 5:38 AM Reply Quote 0
                      • N
                        netblues @CyberMinion
                        last edited by Aug 3, 2020, 5:38 AM

                        @CyberMinion Are you still getting almost constant rates from public dns??? Have you run the pcaps through wireshark?

                        1 Reply Last reply Reply Quote 0
                        • S
                          stephenw10 Netgate Administrator
                          last edited by Aug 3, 2020, 1:58 PM

                          I assume it's all outbound connections? It should be....

                          Hard to imagine what it could possibly be sending legitimately for that long.

                          Steve

                          1 Reply Last reply Reply Quote 0
                          • J
                            johnpoz LAYER 8 Global Moderator
                            last edited by johnpoz Aug 3, 2020, 2:46 PM Aug 3, 2020, 2:42 PM

                            Well you got something trying to find some fqdn, that doesnt' resolve and the client keeps asking, or has a ttl of 30 seconds, etc. etc.

                            Problem with doh, is its hard to see exactly is being queried for.. But you know not being able to troubleshoot what is going on is well worth you know my isp knowing I went to www.google.com for gosh sake ;)

                            setup unbound to log all queries, and maybe you can see what is being queried, even pfsense if you have it pointing to loopback..

                            In the unbound option box

                            server:
                            log-queries: yes
                            log-replies: yes
                            

                            An intelligent man is sometimes forced to be drunk to spend time with his fools
                            If you get confused: Listen to the Music Play
                            Please don't Chat/PM me for help, unless mod related
                            SG-4860 24.11 | Lab VMs 2.7.2, 24.11

                            1 Reply Last reply Reply Quote 0
                            • C
                              CyberMinion
                              last edited by CyberMinion Aug 3, 2020, 5:29 PM Aug 3, 2020, 5:28 PM

                              @netblues

                              Are you still getting almost constant rates from public dns???

                              Sorry, can you clarify?

                              Have you run the pcaps through wireshark?

                              Yes, I looked at them in Wireshark.

                              @stephenw10

                              I assume it's all outbound connections? It should be....

                              No, I see see bi-directional traffic
                              cd200067-a323-4870-8822-9afb02719af3-image.png

                              @johnpoz

                              In the unbound option box...

                              Sorry, being clueless here...where do I find this "unbound option box"? I have "Custom options" in my DNS resolver management page, but that's about all I'm seeing.

                              1 Reply Last reply Reply Quote 0
                              • J
                                johnpoz LAYER 8 Global Moderator
                                last edited by Aug 3, 2020, 5:31 PM

                                Yes that is the options box.

                                An intelligent man is sometimes forced to be drunk to spend time with his fools
                                If you get confused: Listen to the Music Play
                                Please don't Chat/PM me for help, unless mod related
                                SG-4860 24.11 | Lab VMs 2.7.2, 24.11

                                C 1 Reply Last reply Aug 3, 2020, 6:09 PM Reply Quote 0
                                • C
                                  CyberMinion @johnpoz
                                  last edited by Aug 3, 2020, 6:09 PM

                                  @johnpoz
                                  Okay, I've appended that to my options. All is quiet for now since I am active on this network right now; I'll keep an eye on it for the next incident.

                                  1 Reply Last reply Reply Quote 0
                                  • J
                                    johnpoz LAYER 8 Global Moderator
                                    last edited by johnpoz Aug 3, 2020, 6:36 PM Aug 3, 2020, 6:29 PM

                                    Yeah your going to have to watch your log.. Keeping in mind that every time a query is done if it has to open a new tls connection that is going to be quite a bit of overhead, and if doing it to multiple servers and for multiple queries that can add up.

                                    So just took a look at my top "talkers" for dns - and noticed I had left a unbuntu vm running.. Guess what it likes do do a lot!! check for connectivity-check.ubuntu.com, not only for A but also for AAAA, and then guess what it does it also checks for connectivity-check.ubuntu.com.local.lan - because local.lan is my local suffix.

                                    Its making these every few minutes if not seconds - just incase you know there is a captive portal it needs to auth too.. While a couple of packets for dns normally not a big, deal if your using doh or tls that data is amplified X.. Now do that for how many devices, how many queries.. So yeah you could for sure raise the grass level of your traffic going back and forth even when your thinking your network is idle..

                                    In the last 24 hours there have been over 1000 queries for these fqdn.

                                    last24hours.png

                                    What else is being looked for? That is why log of your queries will help, or if you were not encrypted you could just sniff the wan traffic and see what was being asked for since your dns query would be in the clear.

                                    Now in my case that .local.lan won't be sent upstream, since I set unbound to not do that by changing the type of zone... But out of the box transparent mode when asked for stuff that is not in your zone, it will ask upstream for that..

                                    So you have figured out that there seems to be a lot of what you would consider idle traffic to your doh servers.. So you need to figure out what is being asked, so you can just live with it, or make some changes to reduce that traffic or eliminate it completely.. My example, I turned off that check in ubuntu - since have zero need of it.. I don't run capture portal, etc.

                                    An intelligent man is sometimes forced to be drunk to spend time with his fools
                                    If you get confused: Listen to the Music Play
                                    Please don't Chat/PM me for help, unless mod related
                                    SG-4860 24.11 | Lab VMs 2.7.2, 24.11

                                    C 1 Reply Last reply Aug 3, 2020, 6:45 PM Reply Quote 0
                                    • C
                                      CyberMinion @johnpoz
                                      last edited by Aug 3, 2020, 6:45 PM

                                      @johnpoz Good points--such things can certainly do a seemingly excessive number of lookups. However, that assumes the traffic is originating from inside the LAN. In this case, I can tell just based on traffic volume that this traffic is originating from the pfSense itself. While it is sending/receiving a massive number of requests, the LAN and OPT ports are seeing almost no traffic at all.

                                      1 Reply Last reply Reply Quote 0
                                      • J
                                        johnpoz LAYER 8 Global Moderator
                                        last edited by johnpoz Aug 3, 2020, 7:38 PM Aug 3, 2020, 7:20 PM

                                        Your local traffic would be small udp dns queries of a couple of bits.. You get an amplification when your using doh..

                                        Maybe its some package on pfsense doing queries for say IPs hitting its wan, do you have say IPS running

                                        This is why you need to log the queries.. You will see be it pfsense or something else doing the queries..

                                        look at the size of your exchanges in your few packets you sniffed.. Over a time frame of like what .2 seconds.. Now multipy that by hundreds of queries or 1000's even..

                                        To be honest using doh your doing a dns amplification attack against yourself - to hide that your going to google.com from your ISP.. I don't get it... So you make your dns slower, you now hand over every thing you ever ask for to google and quad nine, etc. on a silver platter.. Because your trying to hide that you ask for xyz.com from your ISP?? Its crazy.. But because company xyz says hey you can trust us - hand all your dns to us we won't do anything bad with that info "we promise" ;)

                                        It's nuts!! I will just resolve myself thank you very much..

                                        An intelligent man is sometimes forced to be drunk to spend time with his fools
                                        If you get confused: Listen to the Music Play
                                        Please don't Chat/PM me for help, unless mod related
                                        SG-4860 24.11 | Lab VMs 2.7.2, 24.11

                                        C 1 Reply Last reply Aug 3, 2020, 8:49 PM Reply Quote 0
                                        • C
                                          CyberMinion @johnpoz
                                          last edited by Aug 3, 2020, 8:49 PM

                                          @johnpoz

                                          You get an amplification when your using doh..

                                          That makes some sense, but how would a few b/s become 70 Kb/s upload and 400 Kb/s download? Overhead? Sure, there's some, but that's ridiculous. Meanwhile, in the middle of the day while the pfSense has devices behind it actually in use, this bandwidth consumption decreases considerably. It's only while everything is idle that my problem occurs. Just for laughs, I could shut off DoH for a bit and see, but this doesn't seem to explain the issue. If it were DoH at fault, I would be seeing this "overhead" increase, or at least remain constant while devices behind it are in use. Instead this "overhead" vanishes entirely while its devices are in use.

                                          do you have say IPS running

                                          I have an IDS (SNORT) running on the pfSense. It is passive.

                                          So you make your dns slower, you now hand over every thing you ever ask for to google and quad nine, etc. on a silver platter.. Because your trying to hide that you ask for xyz.com from your ISP?? Its crazy..

                                          Fair point, but that's why the DNS provider needs to be chosen carefully (I hope to switch to OpenDNS soon, but they do not have standardized DoH support yet). I would never use Google, for example, because their business model is based on selling private data. However, local ISPs are also selling private browsing data, so if can make it just a little bit harder for them to farm my internet traffic by sending it to someone I trust, I'll do it (unless, of course, it is causing a malfunction). From a user experience perspective, you can't even see the difference in speed with DoH running.

                                          Tell you what, I will watch it as is for now, but next time I see this problem, I will check my query log, then try disabling DoH. It can't hurt to try (other than the cost of private data).

                                          1 Reply Last reply Reply Quote 0
                                          20 out of 106
                                          • First post
                                            20/106
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                                            This community forum collects and processes your personal information.
                                            consent.not_received