Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    URL Table not working correctly

    Scheduled Pinned Locked Moved General pfSense Questions
    48 Posts 6 Posters 10.1k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • J
      johns
      last edited by

      I setup 4 URL table's (IP's)…three of them work correctly and the number of IP's it pulls from my text files corresponds and is correct.  However one of them, a large one, does not.

      It has 225k entries (see screen shot).  Two strange things happen...1) when viewing in pfsense it only shows 113,890 (see screen shot).  2) When viewing the IP's it shows IP's that are NOT in my original text file/URL Table.  Seen screenshot....it's showing sequential IP's, none of which are in my text file.  What's going on?

      I can understand if I'm hitting a limit with the 225k entries (can I fix this?), but why are the wrong IP's showing?  Is there a way to fix both of these issues?

      Thanks in advance,

      John
      Screenshot.png
      Screenshot.png_thumb
      Screenshot-1.png
      Screenshot-1.png_thumb

      1 Reply Last reply Reply Quote 0
      • D
        doktornotor Banned
        last edited by

        You are doing something very likely insane and will hit BSD kernel limits and issues with this.

        https://redmine.pfsense.org/issues/4876
        https://lists.freebsd.org/pipermail/freebsd-pf/2011-May/006139.html

        (Would appreciate if someone more savvy about the FreeBSD kernel responded here or on that bug.)

        1 Reply Last reply Reply Quote 0
        • F
          firewalluser
          last edited by

          If you have to use swap, you lose the responsiveness of PF managing the states which is perhaps why this is happening as a guess without looking at the code.

          However if using a SSD drive instead of a spin disk, using swap may be less of an issue considering the speed advantages an SSD drive brings to the table over spin disks, but ultimately you have to consider the time & code involved in getting a packet down the bus into the cpu to be processed and then the state held in memory as against then having to run more OS code to handle swapping memory to disk and back again.

          Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

          Asch Conformity, mainly the blind leading the blind.

          1 Reply Last reply Reply Quote 0
          • D
            doktornotor Banned
            last edited by

            Eeeeeeeh? How on earth is swap relevant here???

            1 Reply Last reply Reply Quote 0
            • F
              firewalluser
              last edited by

              Discs are notoriously slow, someone may have put a hard/soft limit in the code to stop too much data being loaded into memory which could then force the OS to use swap more.

              Without knowing this history of the code like when it was originally written which could be a factor, the best option is to examine the code in question, something I'm not able to do at the moment as I'm in the middle of rewriting a Linux network driver as the latest version being shipped in distro's dont work and I want to know why.

              But as another example, it could also be caused by using the wrong data type in code, for example someone might have used a short instead of a long  although that was more common in 16bit code and because of the way code was written back then, it wasnt common to see techniques used which made it easier to port code when wider cpu's came out, ie 8bit to 16bit, 16-32bit, 32-64bit cpu's.

              https://en.wikipedia.org/wiki/C_data_types
              Short signed integer type. Capable of containing at least the [−32767,+32767] range
              Long signed integer type. Capable of containing at least the [−2147483647,+2147483647] range

              The limit doesnt in this case appear to be tied to a data type limit as these stand out quite easily when problems occur, but without examining the code in question my previous post could only be considered as a guess at that stage, but I still suspect swap might be the reason to have a limit.

              Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

              Asch Conformity, mainly the blind leading the blind.

              1 Reply Last reply Reply Quote 0
              • D
                doktornotor Banned
                last edited by

                Dude, are you smoking something strong? Stop flooding this topic with total SHIT. Reported to mods. Get on topic or leave this alone. This is a real issue and has nothing to do with swaps and HDDs.

                1 Reply Last reply Reply Quote 0
                • F
                  firewalluser
                  last edited by

                  @doktornotor:

                  You are doing something very likely insane and will hit BSD kernel limits and issues with this.

                  https://redmine.pfsense.org/issues/4876
                  https://lists.freebsd.org/pipermail/freebsd-pf/2011-May/006139.html
                  (Would appreciate if someone more savvy about the FreeBSD kernel responded here or on that bug.)

                  You asked for some input. Make your mind up?  ::)

                  Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

                  Asch Conformity, mainly the blind leading the blind.

                  1 Reply Last reply Reply Quote 0
                  • D
                    doktornotor Banned
                    last edited by

                    Yeah, I wanted input from people who are familiar with the FreeBSD kernel code in order to get relevant hints.

                    1 Reply Last reply Reply Quote 0
                    • F
                      firewalluser
                      last edited by

                      @doktornotor:

                      Yeah, I wanted input from people who are familiar with the FreeBSD kernel code in order to get relevant hints.

                      So you know everyone now, that must make you the NSA.  ;D

                      Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

                      Asch Conformity, mainly the blind leading the blind.

                      1 Reply Last reply Reply Quote 0
                      • D
                        doktornotor Banned
                        last edited by

                        @firewalluser:

                        So you know everyone now, that must make you the NSA.  ;D

                        Example of relevant hint: https://lists.freebsd.org/pipermail/freebsd-pf/2011-May/006139.html

                        (Except that this one and others mentioned on that thread do NOT work.)

                        Example of totally irrelevant "hint":

                        • I didn't read the code
                        • Swap is slow
                        • Get an SSD
                        • signed long is 2^xyz

                        :( >:( >:(

                        P.S. That box has 8GiB of RAM and is NOT swapping. This is NOT related to running out of physical RAM!!!

                        :( >:( >:(

                        1 Reply Last reply Reply Quote 0
                        • D
                          doktornotor Banned
                          last edited by

                          BTW, @OP:

                          You can see what's in the tables when you go to Diagnostics - Tables and select the one you need from the dropdown.  (The javascript fancy popup is not usable anyway for 200K+ or what entries… cannot search in that at all.)

                          1 Reply Last reply Reply Quote 0
                          • F
                            firewalluser
                            last edited by

                            @firewalluser:

                            Discs are notoriously slow, someone may have put a hard/soft limit in the code to stop too much data being loaded into memory which could then force the OS to use swap more.

                            I saw that thread and one of the thoughts that crossed my mind was the coder has come up with something/algo to limit how much can get loaded into memory to maintain performance. Bear in mind alot of old code was written before the HW developments like loads of RAM & SSD drives we have today, so designs & code compensated or restricted for poor HW performance.

                            Until the code is examined and the reasons for the behaviour is established as it might even be by design, everything is speculation on everyone's part here and in the freebsd thread, but they would be the areas I would look into and for some of the reasons why.

                            Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

                            Asch Conformity, mainly the blind leading the blind.

                            1 Reply Last reply Reply Quote 0
                            • D
                              doktornotor Banned
                              last edited by

                              1 Reply Last reply Reply Quote 0
                              • KOMK
                                KOM
                                last edited by

                                Heh, "right now"…

                                1 Reply Last reply Reply Quote 0
                                • J
                                  johns
                                  last edited by

                                  I'm not having any issues with swap space, however I've also not enabled any firewall rules to use the new URL Table…I had when I first implemented this, but only for a few minutes while testing how things were working after enabling.  I didn't see any HW or OS issues....just the two anomalies I've mentioned, so rules were enabled for 5-10 mins, without performance issues.

                                  Any help/feedback on my actual issues/original questions?

                                  So for some background, the reason I started using this was because I was originally using a Network aliases, but when I starting adding more IP's, I received an error stating something about reaching a limit of 1000 and to change something in php.ini, I was about to do that, but that's when I learned about the URL Tables.  However in my original IP aliases list I only have about 320 IP's (not 1000).  Since I was making some changes I decided to use my large block list I've been gathering, thus the reason for the 200k entries....but originally it was only 320 or so on a Network Alias.

                                  So if I can't have 250k entries, how do I solve this, is there a way to have large numbers of entries without breaking anything (but again, the OS and HW performance were fine for the 5-10 mins)?  Why is it showing numbers that I don't even have in my URL table?

                                  Thanks,

                                  1 Reply Last reply Reply Quote 0
                                  • D
                                    doktornotor Banned
                                    last edited by

                                    @johns:

                                    Any help/feedback on my actual issues/original questions?

                                    See this: https://forum.pfsense.org/index.php?topic=98698.msg549855#msg549855

                                    1 Reply Last reply Reply Quote 0
                                    • J
                                      johns
                                      last edited by

                                      That link takes me to your comment that I've pasted below…how does that help?  You want me to try Diagnostics -Tables to see if it loads?

                                      BTW, @OP:

                                      You can see what's in the tables when you go to Diagnostics - Tables and select the one you need from the dropdown.  (The javascript fancy popup is not usable anyway for 200K+ or what entries... cannot search in that at all.)

                                      1 Reply Last reply Reply Quote 0
                                      • D
                                        doktornotor Banned
                                        last edited by

                                        No. I want you to compare what you see there with what you SHOULD see there, i.e. whether or not it matches the file downloaded. The popup is something that just cannot be worked with on 250K IPs, I'd figure it's very obvious?

                                        1 Reply Last reply Reply Quote 0
                                        • J
                                          johns
                                          last edited by

                                          @doktornotor:

                                          No. I want you to compare what you see there with what you SHOULD see there, i.e. whether or not it matches the file downloaded. The popup is something that just cannot be worked with on 250K IPs, I'd figure it's very obvious?

                                          It shows the same thing as in the first screen shot, IP's that are not in my text file.  I've included a screen shot of it.

                                          ![8-28-2015 12-15-21 pm.png](/public/imported_attachments/1/8-28-2015 12-15-21 pm.png)
                                          ![8-28-2015 12-15-21 pm.png_thumb](/public/imported_attachments/1/8-28-2015 12-15-21 pm.png_thumb)

                                          1 Reply Last reply Reply Quote 0
                                          • D
                                            doktornotor Banned
                                            last edited by

                                            Afraid that unless you make your blocklist available here, this won't get anywhere.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.