Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    IP-Blocklist

    Scheduled Pinned Locked Moved pfSense Packages
    496 Posts 86 Posters 500.9k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • T
      tommyboy180
      last edited by

      @ToxIcon:

      Pfsense 2.0-RC3

      tommyboy180  ip-blocklist will not update the list form iblocklist is it because they made some changes to the update url
      like this address no longer seem to work http://iblocklist.aodcommunity.com/files/wujnnzxrgppvpdujdetm/bt_ads.gz 
      but if enter into a into a browser it downloads.
      their new update format looks like this
      http://list.iblocklist.com/?list=bt_ads&fileformat=p2p&archiveformat=gz
      but when added to  ip-blocklist it also does not download
      their are no errors in the log for ip-blocklist.
       
      Thanks tommyboy180 for all the great work and great packages

      http://iblocklist.taltos.org/files/wujnnzxrgppvpdujdetm/bt_ads.gz is the url for that file. To find the real url just download the file like normal in your browser and go back into your downloads and look at the url in your browser history.

      I'm not sure how often the url scheme changes on that site so checking the url or saving it to a website you own may help with stability.

      Thanks for your support! If you know php then perhaps we could work together on the project!

      -Tom Schaefer
      SuperMicro 1U 2X Intel pro/1000 Dual Core Intel 2.2 Ghz - 2 Gig RAM

      Please support pfBlocker | File Browser | Strikeback

      1 Reply Last reply Reply Quote 0
      • W
        wm408
        last edited by

        Tommyboy,

        I have yet to upgrade to 2.0.x, as I will probably wait for final release (however I did test it and it looks great).  This means I am still running 1.2.3.

        I have recently worked with your latest, 0.3.2.  As we've talked before in this thread, I do use the nanobsd (embedded) version and I do have a SATA 160GB drive mounted separately of the OS in my lil Soekris box.  With the debugging/output that I think is now available (versus previous versions), I was able to modify all the scripts that were getting an error, not being able to write to the embedded file system, ie: read-only.  Now they're all writing to my SATA drive and the IPblocklist package has been running for 72 hours without a crash.  This is a lot better than before where the service would crash, or it would act silly, or whatever.

        The files that needed to be written were files like: ipfw.ipfw, whitelist, and wlists.txt  (i think thats all of them).  (again once IPblocklist could write somewhere, it worked fine… maybe for embedded systems you could write it all to tmp?  And then just have it reload them to tmp if the system gets reset?, not sure.), or where in the scripts it needs to write these files, invoke the /etc/rc.conf_mount_rw and then back /etc/rc.conf_mount_ro when done?  (just a thought)

        I am using a custom list that I created which basically just blocks 98% of all countries that I load from an internal webserver.

        Maybe by now this whole thread includes this kind of information, or you probably already know.

        I just wanted to say thanks.   :)

        1 Reply Last reply Reply Quote 0
        • T
          tommyboy180
          last edited by

          @wm408:

          Tommyboy,

          I have yet to upgrade to 2.0.x, as I will probably wait for final release (however I did test it and it looks great).  This means I am still running 1.2.3.

          I have recently worked with your latest, 0.3.2.  As we've talked before in this thread, I do use the nanobsd (embedded) version and I do have a SATA 160GB drive mounted separately of the OS in my lil Soekris box.  With the debugging/output that I think is now available (versus previous versions), I was able to modify all the scripts that were getting an error, not being able to write to the embedded file system, ie: read-only.  Now they're all writing to my SATA drive and the IPblocklist package has been running for 72 hours without a crash.  This is a lot better than before where the service would crash, or it would act silly, or whatever.

          The files that needed to be written were files like: ipfw.ipfw, whitelist, and wlists.txt  (i think thats all of them).  (again once IPblocklist could write somewhere, it worked fine… maybe for embedded systems you could write it all to tmp?  And then just have it reload them to tmp if the system gets reset?, not sure.), or where in the scripts it needs to write these files, invoke the /etc/rc.conf_mount_rw and then back /etc/rc.conf_mount_ro when done?  (just a thought)

          I am using a custom list that I created which basically just blocks 98% of all countries that I load from an internal webserver.

          Maybe by now this whole thread includes this kind of information, or you probably already know.

          I just wanted to say thanks.   :)

          Excellent. Would you be interested in helping write code and take this package to the next level? I'm getting more busy every day with normal day-to-day things and my school work. I'm actually in an Oracle class right now away from home. If you want to help out and dedicate a couple hours a week to this project that would super awesome.

          Have you considered using Countryblock to block just countries? You should find it more stable for embedded installs. There are a couple hosting companies that are using countryblock and they have found it to be quite stable.

          When I am able to find the time I will release a new version that will detect if embedded and put the files that must be written to and read from in /tmp.

          -Tom Schaefer
          SuperMicro 1U 2X Intel pro/1000 Dual Core Intel 2.2 Ghz - 2 Gig RAM

          Please support pfBlocker | File Browser | Strikeback

          1 Reply Last reply Reply Quote 0
          • W
            wm408
            last edited by

            Tommyboy,

            Thanks for the coding offer.  I just did a few, little hacks in your scripts in response to debugging output that was provided.  I am taking Linux admin and some programming classes, but I am not uber like you.  ;D

            Let me know what you have in mind and maybe I can help, (pm ?)

            As far as writing to tmp, that'd be a cool option I think.  Maybe we could rig a cron job or something to help the IPblocklist reload after a power cycle.

            And maybe the country block would have been a better choice.  (doh).

            1 Reply Last reply Reply Quote 0
            • T
              tommyboy180
              last edited by

              @wm408:

              Tommyboy,

              Thanks for the coding offer.  I just did a few, little hacks in your scripts in response to debugging output that was provided.  I am taking Linux admin and some programming classes, but I am not uber like you.   ;D

              Let me know what you have in mind and maybe I can help, (pm ?)

              As far as writing to tmp, that'd be a cool option I think.  Maybe we could rig a cron job or something to help the IPblocklist reload after a power cycle.

              And maybe the country block would have been a better choice.  (doh).

              The latest version of ipblocklist reloads after a filter reload. It's actually really nice because you don't have to setup email notifications or a cron job anymore. It's stable now.
              I do set the file system to read/write if on embedded but I guess the files need the same permission so /tmp is an excellent place.

              I will be able to get to this update within the next couple of weeks. Thank you so much and when you feel comfortable with PHP send me a PM. I don't want to leave the package DEV scene but it just takes me a little longer to get updates out than users want.

              -Tom Schaefer
              SuperMicro 1U 2X Intel pro/1000 Dual Core Intel 2.2 Ghz - 2 Gig RAM

              Please support pfBlocker | File Browser | Strikeback

              1 Reply Last reply Reply Quote 0
              • W
                wm408
                last edited by

                Ahhh OK I see.  After a filter reload sounds better.

                Thanks!

                @tommyboy180:

                @wm408:

                Tommyboy,

                Thanks for the coding offer.  I just did a few, little hacks in your scripts in response to debugging output that was provided.  I am taking Linux admin and some programming classes, but I am not uber like you.   ;D

                Let me know what you have in mind and maybe I can help, (pm ?)

                As far as writing to tmp, that'd be a cool option I think.  Maybe we could rig a cron job or something to help the IPblocklist reload after a power cycle.

                And maybe the country block would have been a better choice.  (doh).

                The latest version of ipblocklist reloads after a filter reload. It's actually really nice because you don't have to setup email notifications or a cron job anymore. It's stable now.
                I do set the file system to read/write if on embedded but I guess the files need the same permission so /tmp is an excellent place.

                I will be able to get to this update within the next couple of weeks. Thank you so much and when you feel comfortable with PHP send me a PM. I don't want to leave the package DEV scene but it just takes me a little longer to get updates out than users want.

                1 Reply Last reply Reply Quote 0
                • T
                  tommyboy180
                  last edited by

                  New IP-Blocklist poll at http://forum.pfsense.org/index.php/topic,39649.0.html

                  Everyone please vote on the issue. This directly affects the next release of the package. Thank you.

                  -Tom Schaefer
                  SuperMicro 1U 2X Intel pro/1000 Dual Core Intel 2.2 Ghz - 2 Gig RAM

                  Please support pfBlocker | File Browser | Strikeback

                  1 Reply Last reply Reply Quote 0
                  • W
                    Woger
                    last edited by

                    Hi,
                    I am new to pfsense but not completely new to freebsd. I have pfsense 2.0RC3 latest svn running with ip-blocklist installed. IP-blocklist seems to work fine. The problem I have is in the list. When I install sh-drop.tgz it installs about 430 IP's. If I install bt_hijjacked.gz only I get about 370 IP's. When I install both, I get just 70 IP's It seems that the script crashes as soon as it sees double IP ranges. I did read earlier about this and that doubles ones are completely removed, but it seems that the script stops completely.

                    I also get these errors:

                    [2.0-RC3][admin@pfsense.localdomain]/usr/local/www/packages/ipblocklist(39): ./convert-execute.sh
                    1 table deleted.
                    1 table deleted.
                    Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 63, <infile>line 469.
                    Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 36, <infile>line 469.
                    Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 36, <infile>line 469.
                    Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 63, <infile>line 469.
                    Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 36, <infile>line 469.
                    Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 36, <infile>line 469.
                    Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 63, <infile>line 469.
                    Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 36, <infile>line 469.
                    Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 36, <infile>line 469.
                    Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 63, <infile>line 469.
                    Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 36, <infile>line 469.
                    Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 36, <infile>line 469.
                    rm: /usr/local/www/packages/ipblocklist/lists/ipfw.ipfwTEMP: No such file or directory
                    rm: Wlists/whitelistTEMP: No such file or directory
                    rm: /tmp/rules.debug.tmp: No such file or directory
                    rm: /tmp/rules.debug.tmp: No such file or directory
                    0
                    1
                    2
                    3
                    4

                    Besides this: I am planning to run a few web and mailservers behind this pfsense machine. Which lists do you recommend to use?

                    Thanks,

                    Woger</infile></infile></infile></infile></infile></infile></infile></infile></infile></infile></infile></infile>

                    1 Reply Last reply Reply Quote 0
                    • T
                      tommyboy180
                      last edited by

                      70 seems like a really low number. What normally happens is the IP range is converted into CIDR and duplicates are removed. The number 70 doesn't mean 70 IP addresses it means 70 Networks since it's all converted to CIDR.

                      It's possible that the reading is correct. One way to see is to look at the ipfw.ipfw file. You can then see the raw list and determine why that reading is so low.

                      The next release will be CIDR native. This will eliminate weird stuff like this. If you want vote on the discussion to help me get a better understanding of what my userbase needs are at http://forum.pfsense.org/index.php/topic,39649.0.html

                      As far as lists go I recommend the spam, drop, and the forum spam list. Also I highly recommend running countryblock and blocking the major spamming countries.

                      -Tom Schaefer
                      SuperMicro 1U 2X Intel pro/1000 Dual Core Intel 2.2 Ghz - 2 Gig RAM

                      Please support pfBlocker | File Browser | Strikeback

                      1 Reply Last reply Reply Quote 0
                      • W
                        Woger
                        last edited by

                        Well it's 137 to be exactly. :)
                        The last line in ipfw.ipfw is 95.64.9.0/24 which is the second last line in sh_drop. The last line in sh_drop is DROP:95.64.98.0-95.64.99.255 . Exactly the same range is in bt_hijacked somewhere in the middle.
                        A wc -l on all three files gives:

                        358 bt_hijacked
                            139 ipfw.ipfw
                            469 sh_drop
                        Looks like it drops most off the lines during installation of the list files.

                        1 Reply Last reply Reply Quote 0
                        • W
                          Woger
                          last edited by

                          OK,
                          Probably found the problem.
                          both lists use a lot off the same ranges which are removed. This is because you use uniq -u in your convert script. The -u removes all duplicates. If I remove the -u it leaves 1 behind and I get 570 networks instead of 137. Is there a reason why you use the -u ?

                          Greetings,
                          Woger

                          1 Reply Last reply Reply Quote 0
                          • T
                            tommyboy180
                            last edited by

                            @Woger:

                            OK,
                            Probably found the problem.
                            both lists use a lot off the same ranges which are removed. This is because you use uniq -u in your convert script. The -u removes all duplicates. If I remove the -u it leaves 1 behind and I get 570 networks instead of 137. Is there a reason why you use the -u ?

                            Greetings,
                            Woger

                            Why wouldn't you use the unique switch?

                            -Tom Schaefer
                            SuperMicro 1U 2X Intel pro/1000 Dual Core Intel 2.2 Ghz - 2 Gig RAM

                            Please support pfBlocker | File Browser | Strikeback

                            1 Reply Last reply Reply Quote 0
                            • K
                              kyronax
                              last edited by

                              Hey everybody, first post here  ;)

                              I've put together my first pfSense box and it's pretty sweet. It's running on a Supermicro X7SPA-HF-D525 booting off a 2GB USB drive. I did a "hard drive" install to the USB (not embedded because it didn't boot properly). http://www.dropbox.com/gallery/9284282/1/pfSense?h=5dead0

                              Anyways, I have a little problem with IP-Blocklist. I have 2 lists installed and 1 whitelist (Steam). Maybe it's just my stupidity but whenever I reboot the router it would "forget" about the whitelist and continues blocking Steam. I have to hit the Save/Update button to get it working again. I searched but couldn't find anything regarding this.

                              Thanks!

                              2.0-RC3 (i386) built on Tue Aug 16 19:48:02 EDT 2011
                              IP-Blocklist Package 0.3.2

                              Supermicro X7SPA-HF-D525 Intel Atom D525 ICH9R 6XSATA Dual GBLAN 1XPCI-E 2.0 Matrox G200EW
                              Corsair CMSO2GX3M1A1333C9 2GB 1X2GB DDR3-1333 CL9-9-9-24 204PIN SODIMM Memory
                              MINI-BOX M350 Universal MINI-ITX Case
                              PICOPSU-80 with Power Kit Includes Pico PSU 80W DC-DC ATX PSU & AC-DC Switching Power Supp

                              1 Reply Last reply Reply Quote 0
                              • W
                                Woger
                                last edited by

                                @tommyboy180:

                                @Woger:

                                OK,
                                Probably found the problem.
                                both lists use a lot off the same ranges which are removed. This is because you use uniq -u in your convert script. The -u removes all duplicates. If I remove the -u it leaves 1 behind and I get 570 networks instead of 137. Is there a reason why you use the -u ?

                                Greetings,
                                Woger

                                Why wouldn't you use the unique switch?

                                uniq -u gets only the uniq lines from a file. So if there are double lines in a file these aren't uniq and will be discarded (as mentioned earlier in this thread). But without the -u switch, uniq will leave one of the double lines behind. For the 2 lists I use 358 lines in bt_hijacked and 469 lines in sh_drop there are more than 300 identical lines. With uniq -u these are all removed which ends in 139 lines in ipfw.ipfw. If I remove the -u switch the 300 identical lines are made uniq again which leaves more than 400 uniq lines in ipfw.ipfw.

                                hmm.. not very explaining  ???. I'll try it like this:

                                file "test" containg these lines:
                                ip-1
                                ip-1
                                ip-2
                                ip-3
                                ip-3

                                uniq -u test > test2 will result in a file which contains
                                ip-2

                                uniq test > test2 will result in a file which contains
                                ip-1
                                ip-2
                                ip-3

                                1 Reply Last reply Reply Quote 0
                                • T
                                  tommyboy180
                                  last edited by

                                  Well that is a major problem. I will hopefully push an update ASAP. Thanks for taking the time to explain it!

                                  -Tom Schaefer
                                  SuperMicro 1U 2X Intel pro/1000 Dual Core Intel 2.2 Ghz - 2 Gig RAM

                                  Please support pfBlocker | File Browser | Strikeback

                                  1 Reply Last reply Reply Quote 0
                                  • T
                                    tommyboy180
                                    last edited by

                                    Version 3.3 Released.

                                    • Fixed uniq -u bug

                                    • Removed email tab - no longer needed

                                    Thank you Woger!

                                    -Tom Schaefer
                                    SuperMicro 1U 2X Intel pro/1000 Dual Core Intel 2.2 Ghz - 2 Gig RAM

                                    Please support pfBlocker | File Browser | Strikeback

                                    1 Reply Last reply Reply Quote 0
                                    • W
                                      Woger
                                      last edited by

                                      No thanks,
                                      Glad I could help  :)
                                      Keep up the good work!

                                      1 Reply Last reply Reply Quote 0
                                      • S
                                        Shuey
                                        last edited by

                                        When I try turning this on with one list added, I get the error:

                                        /tmp/rules.debug:86: cannot load "/usr/local/www/packages/countryblock/lists/countries.txt": No such file or directory

                                        I am running pfSense 2.0-RC3 (i386) Aug 18, 2011
                                        I have package IP-Blocklist 0.3.3
                                        I'm trying to use this list: http://www.tomschaefer.org/temp/pfsense/IP-Blocklist-ForumSpam.txt

                                        I don't have the "country block" package installed at all, so I don't get why I'm seeing this error…

                                        1 Reply Last reply Reply Quote 0
                                        • T
                                          tommyboy180
                                          last edited by

                                          Did you have the countryblock package installed at any point?

                                          -Tom Schaefer
                                          SuperMicro 1U 2X Intel pro/1000 Dual Core Intel 2.2 Ghz - 2 Gig RAM

                                          Please support pfBlocker | File Browser | Strikeback

                                          1 Reply Last reply Reply Quote 0
                                          • S
                                            Shuey
                                            last edited by

                                            @tommyboy180:

                                            Did you have the countryblock package installed at any point?

                                            Yes, and I could never get it to work properly.  But it was uninstalled and I also upgraded pfSense from RC2 to RC3.  I figured those two things would've fixed the problem, but pf obviously still sees some junk left over from CB.

                                            What can I do?

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.