Navigation

    Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    PfBlockerNG SoNewConn Issues

    pfBlockerNG
    3
    13
    860
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      armouredking last edited by

      Hi all,

      Since the latest patch to pfSense I've been getting these errors:

      
      Jan 5 12:07:51	kernel		sonewconn: pcb 0xfffff80158f7a740: Listen queue overflow: 1537 already in queue awaiting acceptance (228 occurrences)
      
      

      They occur once a minute, and start at some interval x after a reboot depending on how high I set sonewconn. I usually can tell when it starts happening because pfSense DNS resolution dies; it takes minutes to load a website. I got all the way up to 15k and it would take like 3 days before they would start occurring but I couldn't get rid of them except by turning off pfBlockerNG.

      It took me a long time to isolate this issue to pfBlockerNG as there is nothing in Google search that points to it, and doing netstat on that pcb shows no entries. I had to go back to the basics and just start turning off plugins one at a time to see what isolated it, and it's pfBlockerNG.

      System Build:
      Version 2.4.2-RELEASE-p1 (amd64)
      CPU Type Intel(R) Atom(TM) CPU C2758 @ 2.40GHz
      System Super Micro C2758

      pfBlockerNG net 2.1.2_2

      I'm using the 4 onboard NICs as well as an addon 520-DA1 with vLANs and configured per wiki for ix and igb ethernet interfaces. Other plugins in use: ACME, HAProxy, Cron, and pfBlockerNG; no others.

      1 Reply Last reply Reply Quote 0
      • RonpfS
        RonpfS last edited by

        maybe take a look at https://forum.pfsense.org/index.php?topic=137103.msg754536#msg754536

        1 Reply Last reply Reply Quote 0
        • A
          armouredking last edited by

          I'm aware of that glitch but the gateway itself is still accessible on my end, especially if I use IP addresses. I don't actually get a timeout, just long loading times over DNS when a name is requested I'm guessing because the listen queues are filled. Turning off pfBlockerNG immediately solves the issue. I am not convinced this is the same issue as the 502 gateway bug mentioned in the other thread..

          1 Reply Last reply Reply Quote 0
          • RonpfS
            RonpfS last edited by

            Did you look at pfblockerng logs ? System, Resolver logs? etc

            What's the size of you DNSBL db in regards to your memory?
            I have about 1M DNSBL entries with 8 GB  of memory. When I was running on a 2.5GB system, I had to limit to about 400K.

            What is your Resolver configuration?

            1 Reply Last reply Reply Quote 0
            • A
              armouredking last edited by

              I have 32GB of memory, that is not an issue ( though I do have 2.5M ish entries in DNSBL ).

              The other logs do not show anything with this particular error. Certainly no smoking guns. The sonewconn error does take up the whole of that particular log file though since it basically prints once a minute.

              1 Reply Last reply Reply Quote 0
              • RonpfS
                RonpfS last edited by

                Do you have the Dashboard open all the time?
                Or the pfblockerNG alerts tab with auto-refresh ?

                1 Reply Last reply Reply Quote 0
                • A
                  armouredking last edited by

                  I don't have it open all the time; I hardly look at pfSense. Well, except for times like now where it gives me grief, but lately the screens I've been swapping back and forth are the system tunables, reboot, and logs. The alerts tab does have auto refresh and autoresolve ticked, but does that matter if you're not on the page?

                  Scrolling again through all the logs, the only oddity I can find ( most of them are empty with the last entries from the reboot date or the daily / dhcp /dns tasks ) is in the DNSBL log specifically there are multiple repeated entries and nothing else:

                  
                  DNSBL Reject HTTPS,Jan 05 20:34:51,events.gfe.nvidia.com
                  DNSBL Reject HTTPS,Jan 05 20:34:52,events.gfe.nvidia.com
                  DNSBL Reject HTTPS,Jan 05 20:34:53,events.gfe.nvidia.com
                  DNSBL Reject HTTPS,Jan 05 20:34:56,events.gfe.nvidia.com
                  DNSBL Reject HTTPS,Jan 05 20:34:57,events.gfe.nvidia.com
                  DNSBL Reject HTTPS,Jan 05 20:34:58,events.gfe.nvidia.com
                  DNSBL Reject HTTPS,Jan 05 20:34:59,mobile.pipe.aria.microsoft.com
                  DNSBL Reject HTTPS,Jan 05 20:34:59,mobile.pipe.aria.microsoft.com
                  DNSBL Reject HTTPS,Jan 05 20:34:59,mobile.pipe.aria.microsoft.com
                  DNSBL Reject HTTPS,Jan 05 20:34:59,mobile.pipe.aria.microsoft.com
                  DNSBL Reject HTTPS,Jan 05 20:34:59,mobile.pipe.aria.microsoft.com
                  DNSBL Reject HTTPS,Jan 05 20:34:59,mobile.pipe.aria.microsoft.com
                  DNSBL Reject HTTPS,Jan 05 20:35:01,events.gfe.nvidia.com
                  DNSBL Reject HTTPS,Jan 05 20:35:02,events.gfe.nvidia.com
                  DNSBL Reject HTTPS,Jan 05 20:35:03,mobile.pipe.aria.microsoft.com
                  DNSBL Reject HTTPS,Jan 05 20:35:03,mobile.pipe.aria.microsoft.com
                  DNSBL Reject HTTPS,Jan 05 20:35:03,mobile.pipe.aria.microsoft.com
                  DNSBL Reject HTTPS,Jan 05 20:35:03,mobile.pipe.aria.microsoft.com
                  DNSBL Reject HTTPS,Jan 05 20:35:03,mobile.pipe.aria.microsoft.com
                  DNSBL Reject HTTPS,Jan 05 20:35:03,mobile.pipe.aria.microsoft.com
                  DNSBL Reject HTTPS,Jan 05 20:35:03,events.gfe.nvidia.com
                  DNSBL Reject HTTPS,Jan 05 20:35:06,events.gfe.nvidia.com
                  
                  

                  This continues ad nauseum for days as far as I can tell. Is it possible these repeaters / phone homes are causing the issue? There is nothing of note in the error or pfblockerng logs, they just state the last reload / refresh was successful.

                  1 Reply Last reply Reply Quote 0
                  • BBcan177
                    BBcan177 Moderator last edited by

                    That Domain is from MS Outlook mobile app:

                    https://social.technet.microsoft.com/Forums/ie/en-US/c29a50e7-9433-4fa1-b2f3-24ee93299810/urls-needed-for-office-2016-online-help?forum=Office2016setupdeploy

                    Not sure if it should be blocked or not. But if it's hitting DNSBL so frequently. You could add an unbound host override and point it to 127.0.0.1. This will bypass DNSBL completely.

                    1 Reply Last reply Reply Quote 0
                    • A
                      armouredking last edited by

                      I don't have any mobile apps for Office but I do have an Office install. Nvidia is hitting it just as frequently so I added it as well and removed my sonewconn system tunable edits. It usually takes hours for the issue to make itself known though so I won't know for a bit if this fixes it.

                      1 Reply Last reply Reply Quote 0
                      • A
                        armouredking last edited by

                        So this made it about yay long before the error came back. Interestingly enough the DNSBL just terminates on the 7th; ie, service is still running but no further log entries are made. Also, this log file must be gigantic because trying to view it from the webgui slows the browser to a crawl if it doesn't just outright crash it.

                        
                        DNSBL Reject HTTPS,Jan 07 05:45:03,events.gfe.nvidia.com
                        DNSBL Reject HTTPS,Jan 07 05:45:03,mobile.pipe.aria.microsoft.com
                        DNSBL Reject HTTPS,Jan 07 05:45:03,mobile.pipe.aria.microsoft.com
                        DNSBL Reject HTTPS,Jan 07 05:45:03,mobile.pipe.aria.microsoft.com
                        DNSBL Reject HTTPS,Jan 07 05:45:04,mobile.pipe.aria.microsoft.com
                        DNSBL Reject HTTPS,Jan 07 05:45:04,mobile.pipe.aria.microsoft.com
                        DNSBL Reject HTTPS,Jan 07 05:45:04,mobile.pipe.aria.microsoft.com
                        
                        

                        The other thing is, these entries came back? I have the unbound host overrides still configured:

                        Domain Overrides
                        Domain Lookup Server IP Address Description Actions
                        mobile.pipe.aria.microsoft.com 127.0.0.1 Override pfBlockerNG
                        events.gfe.nvidia.com 127.0.0.1 Override pfBlockerNG

                        Again, turning off pfBlockerNG immediately fixes the issue.

                        Edit:

                        Something weird happened while I'm troubleshooting. For now I'm trying to ascertain if it's a set of DNSBL entries that are causing this. Turning pfBlockerNG back on using only EasyList throws this:

                        
                        Jan 9 19:55:56	php-fpm	90038	[pfBlockerNG] Starting cron process.
                        Jan 9 19:55:56	php-fpm	90038	/pkg_edit.php: The command '/sbin/ifconfig 'ix0.520' delete '10.10.10.1'' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address'
                        
                        

                        pfBlockerNG is setup with firewall rules on vLANs 1, 520, and 540. The listening interface is set to vLAN 520.

                        1 Reply Last reply Reply Quote 0
                        • BBcan177
                          BBcan177 Moderator last edited by

                          Set the host override to "0.0.0.0" instead of "127.0.0.1"…

                          1 Reply Last reply Reply Quote 0
                          • A
                            armouredking last edited by

                            Did that, had no affect. 0.0.0.0 or 127.0.0.1.

                            Well, did figure something out at least. For anyone else who runs into this issue it has to be a new block add in my block lists. They worked fine in the past but something got updated in a list somewhere and now it's causing this issue. Can't rule out the nvidia or MS repeat events being the culprit but adding them to Unbound doesn't have any affect. I stripped pfBlockerNG all the way down to just EasyList content and have not had any issues for the past week. Since it takes like 1-3 days before it starts happening though it's going to be a real bugbear to troubleshoot which particular list(s) are the culprits.

                            My DNSBL feeds are basically all from Firebog (wally3k.github.io) for reference.

                            1 Reply Last reply Reply Quote 0
                            • BBcan177
                              BBcan177 Moderator last edited by

                              I sent you a PM…

                              1 Reply Last reply Reply Quote 0
                              • First post
                                Last post

                              Products

                              • Platform Overview
                              • TNSR
                              • pfSense
                              • Appliances

                              Services

                              • Training
                              • Professional Services

                              Support

                              • Subscription Plans
                              • Contact Support
                              • Product Lifecycle
                              • Documentation

                              News

                              • Media Coverage
                              • Press
                              • Events

                              Resources

                              • Blog
                              • FAQ
                              • Find a Partner
                              • Resource Library
                              • Security Information

                              Company

                              • About Us
                              • Careers
                              • Partners
                              • Contact Us
                              • Legal
                              Our Mission

                              We provide leading-edge network security at a fair price - regardless of organizational size or network sophistication. We believe that an open-source security model offers disruptive pricing along with the agility required to quickly address emerging threats.

                              Subscribe to our Newsletter

                              Product information, software announcements, and special offers. See our newsletter archive to sign up for future newsletters and to read past announcements.

                              © 2021 Rubicon Communications, LLC | Privacy Policy