Navigation

    Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    PfBlocker not updating after initial list download and only trying once a day

    pfSense Packages
    7
    25
    5128
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      mallwri last edited by

      Hello.

      pfSense 2.1 and pfBlock 1.02 on both standard 32-bit and NanoBSD releases do not update lists after initial download and only update once a day at 12h30 inspite of being set to update every 12 hours.

      The logs say:

      Oct 20 12:30:44 php: rc.update_urltables: /etc/rc.update_urltables: pfBlockerMDL does not need updated.
      Oct 20 12:30:44 php: rc.update_urltables: /etc/rc.update_urltables: pfBlockerET does not need updated.
      Oct 20 12:30:44 php: rc.update_urltables: /etc/rc.update_urltables: pfBlockerDShield10 does not need updated.
      Oct 20 12:30:44 php: rc.update_urltables: /etc/rc.update_urltables: pfBlockerSpamHausDrop does not need updated.

      the "crontab" file only has an entry that will trigger at 12h30:

      30 12 * * * root /usr/bin/nice -n20 /etc/rc.update_urltables

      And it is set to update every 12 hours in the configuration XML file.

      The number of CIDRs has not changed for any of the list since first installing pfBlocker about 6 weeks ago. The log always indicates "does not need updated"; I have manually download the current lists and they have changed since the initial version.

      Anybody else seeing this?  I have disabled and re-enabled pfBlocker - no change.  Rebooting - no change.  Tried on
      different hardware (also 2.1 pfSense) - same result.  Anybody have any other ideas to fix this?

      Thanks.

      Mark.

      1 Reply Last reply Reply Quote 0
      • marcelloc
        marcelloc last edited by

        Did you tried to change update frequency on list configuration tab? ???

        1 Reply Last reply Reply Quote 0
        • M
          mallwri last edited by

          Hello.

          Yes I tried that.  See the attached image.  Both systems were previously on pfSense 2.0 and upgraded to pfSense 2.1 before the pfBlocker package was installed.  Only package I have installed is pfBlocker.  I do have some rules that have time schedules but the rest is pretty standard both boxes.

          In the XML configuration code the "updatefreq" is "32" for pfBlocker; seems an odd value for 12 hours.

          -<aliases>-<alias><name>pfBlockerET</name><url>https://127.0.0.1:443/pfblocker.php?pfb=pfBlockerET</url><updatefreq>32</updatefreq>

          <address>-<type>urltable</type>- <detail></detail> -<alias><name>pfBlockerMDL</name><url>https://127.0.0.1:443/pfblocker.php?pfb=pfBlockerMDL</url><updatefreq>32</updatefreq>

          <address>-<type>urltable</type>- <detail></detail>

          And the snip for the "crontab":

          -<minute>30</minute><hour>12</hour><mday></mday><month></month><wday>*</wday><who>root</who><command></command>/usr/bin/nice -n20 /etc/rc.update_urltables

          And the pfBlocker snip:

          <menu><name>pfBlocker</name><tooltiptext>Configure pfblocker</tooltiptext>Firewall<url>/pkg_edit.php?xml=pfblocker.xml</url></menu>

          -<tab><text>General</text><url>/pkg_edit.php?xml=pfblocker.xml&id=0</url><active></active></tab>-<pfblocker>-<config><enable_cb>on</enable_cb><enable_log>on</enable_log><inbound_interface>wan</inbound_interface><inbound_deny_action>block</inbound_deny_action><outbound_interface>opt1,opt2,opt3,opt4,opt5</outbound_interface><outbound_deny_action>reject</outbound_deny_action><credits><donation></donation></credits></config></pfblocker>-<pfblockerlists>-<config><aliasname>ET</aliasname>--<row><format>txt</format><url>http://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt</url></row>-<row><format>txt</format><url>http://rules.emergingthreats.net/blockrules/compromised-ips.txt</url></row><action>Deny_Both</action><cron>12hours</cron></config>-<config><aliasname>MDL</aliasname>--<row><format>txt</format><url>http://www.malwaredomainlist.com/hostslist/ip.txt</url></row><action>Deny_Both</action><cron>12hours</cron></config></pfblockerlists>

          Thanks.

          Mark.


          </address></alias> </address></alias></aliases>

          1 Reply Last reply Reply Quote 0
          • marcelloc
            marcelloc last edited by

            Did you enabled pfblocker on general tab?

            I've tested on 2.1 and it's working fine.

            1 Reply Last reply Reply Quote 0
            • M
              mallwri last edited by

              Hello.

              pfBlocker is enabled in the general tab.

              Looking at "ls -l /var/db/aliastables" it is clear the tables are being updated.  There does not appear to be a direct log entry for each table with a "updated" or "does not need updated" entry for each time this runs; I guess this is what initially confused me.

              What I do not understand is why there is an additional update at 12h30 every day with a "updated" or "does not need updated" log entry every day.  There is an entry in the "/etc/crontab" file that starts this process.

              Thanks for the help.

              Mark.

              1 Reply Last reply Reply Quote 0
              • marcelloc
                marcelloc last edited by

                This is a system update for url tables.  It's not related to pfblocker update code.

                if you can see a pfblocker php on cron tab, then it's working fine.

                1 Reply Last reply Reply Quote 0
                • M
                  mallwri last edited by

                  Hello.

                  OK, that makes sense.

                  In summary pfBlocker is working fine with pfSense 2.1.

                  Thanks for the help.

                  Mark.

                  1 Reply Last reply Reply Quote 0
                  • M
                    MarkVLK last edited by

                    I'm actually seeing the same issue on 2.1.4

                    In the system logs, it says none of the three lists that I have need updating but I've compared the rules of at least one of the lists to the actual list at the link I provided it and the IPs are not the same. I had it set to update every 24 hours but changed it to 12 hours though seems like nothing has changed and the IPs are still outdated.

                    1 Reply Last reply Reply Quote 0
                    • A
                      ashes00 last edited by

                      @MarkVLK:

                      I'm actually seeing the same issue on 2.1.4

                      In the system logs, it says none of the three lists that I have need updating but I've compared the rules of at least one of the lists to the actual list at the link I provided it and the IPs are not the same. I had it set to update every 24 hours but changed it to 12 hours though seems like nothing has changed and the IPs are still outdated.

                      MarkVLK - I'm seeing the same thing on 2.1.3.  None of the lists are updating even thought their file date changes.  DI you ever find a solution to this?  Thanks
                      Ash,

                      1 Reply Last reply Reply Quote 0
                      • M
                        MarkVLK last edited by

                        @ashes00:

                        @MarkVLK:

                        I'm actually seeing the same issue on 2.1.4

                        In the system logs, it says none of the three lists that I have need updating but I've compared the rules of at least one of the lists to the actual list at the link I provided it and the IPs are not the same. I had it set to update every 24 hours but changed it to 12 hours though seems like nothing has changed and the IPs are still outdated.

                        MarkVLK - I'm seeing the same thing on 2.1.3.  None of the lists are updating even thought their file date changes.  DI you ever find a solution to this?  Thanks
                        Ash,

                        Unfortunately I have not yet found a solution. The lists do appear to update occasionally, just not every time. Not sure what's causing this behavior.

                        1 Reply Last reply Reply Quote 0
                        • A
                          ashes00 last edited by

                          @MarkVLK:

                          @ashes00:

                          @MarkVLK:

                          I'm actually seeing the same issue on 2.1.4

                          In the system logs, it says none of the three lists that I have need updating but I've compared the rules of at least one of the lists to the actual list at the link I provided it and the IPs are not the same. I had it set to update every 24 hours but changed it to 12 hours though seems like nothing has changed and the IPs are still outdated.

                          MarkVLK - I'm seeing the same thing on 2.1.3.  None of the lists are updating even thought their file date changes.  DI you ever find a solution to this?  Thanks
                          Ash,

                          Unfortunately I have not yet found a solution. The lists do appear to update occasionally, just not every time. Not sure what's causing this behavior.

                          MarkVLK - Quick Question are you using an embedded version (NANO)?
                          Ash,

                          1 Reply Last reply Reply Quote 0
                          • T
                            tobi64 last edited by

                            I see this behaviour on the embedded version 2.1.5 (nanobsd) whenever the media read/write status is read-only.

                            1 Reply Last reply Reply Quote 0
                            • A
                              ashes00 last edited by

                              @tobi64:

                              I see this behaviour on the embedded version 2.1.5 (nanobsd) whenever the media read/write status is read-only.

                              tobi64 - Thanks for the heads up.  I am starting to think this is a NANO platform related issue.  I have changed my NANO file system to RW, and tried the rc.updatealiastables command, and nothing changes.  I ordered a SSD, and will be installing the Full version, and testing against that in the coming weeks.

                              1 Reply Last reply Reply Quote 0
                              • T
                                tobi64 last edited by

                                May be I'm wrong, but in my eyes the command 'rc.updatealiastables' has nothing to do with pfblocker. The periodically update uses the command 'php -q /usr/local/www/pfblocker.php cron'. I tried it on page 'Diagnostics / Command Prompt' without success. But it works over ssh using Putty.exe. But only if the media read/write status is read-write.

                                1 Reply Last reply Reply Quote 0
                                • A
                                  ashes00 last edited by

                                  tobi64 - I found in the pf forums that pfblocker relies on pfsense to do the actual list update.  That command was the one that people were using to force update the URL alias lists.  I think its broke as it has never forked for me though.  Ill continue with my plan to swap out pfsense HDD with Full install HDD, and continue trouble shooting.

                                  On another note….. I have another PFsense (Full Install) that just had every list, other than Country lists, disappear in all location (Aliases>URLs, PFblocker widget, and /var/db/aliatables/*.  They are still referenced in Firewall>Pf blocker>lists though!?!  WTH  Getting tired of this PFblocker acting like a fool.  I think I'm just going to add/remove the package,and set it up from fresh.  Grrrr...

                                  1 Reply Last reply Reply Quote 0
                                  • BBcan177
                                    BBcan177 Moderator last edited by

                                    Hi ashes00,

                                    @ashes00:

                                    tobi64 - I found in the pf forums that pfblocker relies on pfsense to do the actual list update.

                                    pfBlocker should update on its own. the rc.update_urltables will update once every 32 days only.

                                    What settings are you using for the "Action" in the Alias TAB?

                                    On another note….. I have another PFsense (Full Install) that just had every list, other than Country lists, disappear in all location (Aliases>URLs, PFblocker widget, and /var/db/aliatables/*.  They are still referenced in Firewall>Pf blocker>lists though!?!  WTH  Getting tired of this PFblocker acting like a fool.  I think I'm just going to add/remove the package,and set it up from fresh.  Grrrr...

                                    From the comment above, either pfBlocker was Disabled, or the Action is set as "Disabled" or the Update Frequency is set to "Never"

                                    If that is not the case, reply back…

                                    1 Reply Last reply Reply Quote 0
                                    • A
                                      ashes00 last edited by

                                      BBcan177 - Thanks for responding! :)

                                      What settings are you using for the "Action" in the Alias TAB?

                                      General Settings TAB Inbound Action = Block, and Outbound action - reject. Lists TAB Deny_Both for all but 1 list.

                                      From the comment above, either pfBlocker was Disabled, or the Action is set as "Disabled" or the Update Frequency is set to "Never"

                                      If that is not the case, reply back… --> This is my 2nd pfsense,and the actions are the same as 1st pfsesne system.

                                      My update frequency is always set to 4 Hours for both pfsense routers.

                                      Hope this helps.

                                      Also I have upgraded pfsesne unit 1 from NANO build v2.1.3 to Full Install v2.1.5 on Friday night.  I still have 2 lists that are not updating to what www.iblocklist.com says the CIDRs count should equal.  Lists not updating = spyware and hijacked (bluetrack version).

                                      Ash

                                      1 Reply Last reply Reply Quote 0
                                      • BBcan177
                                        BBcan177 Moderator last edited by

                                        Which lists are you having issues with? What are you expecting the count to be?

                                        1 Reply Last reply Reply Quote 0
                                        • A
                                          ashes00 last edited by

                                          @BBcan177:

                                          Which lists are you having issues with? What are you expecting the count to be?

                                          1. I am using the following iblocklist.com lists. (with a paid subscription)

                                          hijacked - Bluetack (p2p, gz) [Number of ranges = 496, Last updated 11/11/14]
                                          spyware - Bluetrack (p2p, gz) [Number of ranges = 3201, Last updated 11/11/14]

                                          On 2 different PFsense systems report the following CIDR counts

                                          PF01 (Full install v2.1.5)
                                          hijacked = 536 CIDRs
                                          spyware = 3565 CIDRs

                                          PF02 (Full install v2.1.3)
                                          hijacked = 536 CIDRs
                                          spyware = 3565 CIDRs

                                          I am expecting to see the following on both PFsense systems.
                                          hijacked = 496 CIDRs
                                          spyware = 3201 CIDRs

                                          2. Side note, what is the best/proper iblocklist.com (paid subscription) File Format?  p2p, dat, cidr, or hosts?  I picked p2p after reading tons of PFsense PFblocker threads, but not 100% sure what the best choice is here.

                                          3. Also thanks so much for all you do.  From what I read you seem to be leading the path for the rebirth/redesign of PFBLOCKER.  Keep up the great work, and THANK YOU!!!!!!  Is there anytime line on when the newer PFBLOCKER might come out in "Available Packages"?

                                          Ash

                                          1 Reply Last reply Reply Quote 0
                                          • BBcan177
                                            BBcan177 Moderator last edited by

                                            You would use p2p,gz format.

                                            With my version of pfBlockerNG, I have the following counts for those Lists:

                                            http://list.iblocklist.com/?list=usrcshglbiilevmyfhse&fileformat=p2p&archiveformat=gz
                                            wc -l IBlock_BT_Hijack.orig
                                              536 IBlock_BT_Hijack.orig

                                            http://list.iblocklist.com/?list=llvtlsjyoyiczbkjsxpf&fileformat=p2p&archiveformat=gz
                                            wc -l IBlock_BT_Spy.orig
                                              3565 IBlock_BT_Spy.orig

                                            If you want to help Beta Test, send me a PM. The Devs are busy with 2.2, so when they have time I hope they will review.

                                            1 Reply Last reply Reply Quote 0
                                            • A
                                              ashes00 last edited by

                                              @BBcan177:

                                              You would use p2p,gz format.

                                              With my version of pfBlockerNG, I have the following counts for those Lists:

                                              http://list.iblocklist.com/?list=usrcshglbiilevmyfhse&fileformat=p2p&archiveformat=gz
                                              wc -l IBlock_BT_Hijack.orig
                                                536 IBlock_BT_Hijack.orig

                                              http://list.iblocklist.com/?list=llvtlsjyoyiczbkjsxpf&fileformat=p2p&archiveformat=gz
                                              wc -l IBlock_BT_Spy.orig
                                                3565 IBlock_BT_Spy.orig

                                              Hey BBcan177, Well that matches up with what I have when I do wc -l on both lists and both PFsense systems.  Great you and I are experiencing the same situation then (Great Base line).  When I go over to the iblocklist.com website the stated Number of ranges are different.

                                              https://www.iblocklist.com/list.php?list=usrcshglbiilevmyfhse –> Number of ranges = 496, not 536
                                              https://www.iblocklist.com/list.php?list=llvtlsjyoyiczbkjsxpf --> Number of ranges = 3201, not 3565

                                              When I download the list manually to my Win7 PC, and do line count I get 496, & 3201 respectively.  I could understand if our lists were smaller if PFblocker was dedupping some ranges, but both our lists are larger than what we are being given.  This leads me to believe that PBflocker is not updating the list correctly.  Is my logic flawed?

                                              Both my PFsense systems are production, so I can't Beta test at the moment.  If I can find some hardware to spare I'll PM you later.

                                              Again thanks for all your help!
                                              Ash,

                                              1 Reply Last reply Reply Quote 0
                                              • W
                                                wcrowder last edited by

                                                @ashes00:

                                                @BBcan177:

                                                You would use p2p,gz format.

                                                With my version of pfBlockerNG, I have the following counts for those Lists:

                                                http://list.iblocklist.com/?list=usrcshglbiilevmyfhse&fileformat=p2p&archiveformat=gz
                                                wc -l IBlock_BT_Hijack.orig
                                                  536 IBlock_BT_Hijack.orig

                                                http://list.iblocklist.com/?list=llvtlsjyoyiczbkjsxpf&fileformat=p2p&archiveformat=gz
                                                wc -l IBlock_BT_Spy.orig
                                                  3565 IBlock_BT_Spy.orig

                                                Hey BBcan177, Well that matches up with what I have when I do wc -l on both lists and both PFsense systems.  Great you and I are experiencing the same situation then (Great Base line).  When I go over to the iblocklist.com website the stated Number of ranges are different.

                                                https://www.iblocklist.com/list.php?list=usrcshglbiilevmyfhse –> Number of ranges = 496, not 536
                                                https://www.iblocklist.com/list.php?list=llvtlsjyoyiczbkjsxpf --> Number of ranges = 3201, not 3565

                                                When I download the list manually to my Win7 PC, and do line count I get 496, & 3201 respectively.  I could understand if our lists were smaller if PFblocker was dedupping some ranges, but both our lists are larger than what we are being given.  This leads me to believe that PBflocker is not updating the list correctly.  Is my logic flawed?

                                                Both my PFsense systems are production, so I can't Beta test at the moment.  If I can find some hardware to spare I'll PM you later.

                                                Again thanks for all your help!
                                                Ash,

                                                http://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing

                                                1 Reply Last reply Reply Quote 0
                                                • A
                                                  ashes00 last edited by

                                                  @wcrowder:

                                                  @ashes00:

                                                  @BBcan177:

                                                  You would use p2p,gz format.

                                                  With my version of pfBlockerNG, I have the following counts for those Lists:

                                                  http://list.iblocklist.com/?list=usrcshglbiilevmyfhse&fileformat=p2p&archiveformat=gz
                                                  wc -l IBlock_BT_Hijack.orig
                                                    536 IBlock_BT_Hijack.orig

                                                  http://list.iblocklist.com/?list=llvtlsjyoyiczbkjsxpf&fileformat=p2p&archiveformat=gz
                                                  wc -l IBlock_BT_Spy.orig
                                                    3565 IBlock_BT_Spy.orig

                                                  Hey BBcan177, Well that matches up with what I have when I do wc -l on both lists and both PFsense systems.  Great you and I are experiencing the same situation then (Great Base line).  When I go over to the iblocklist.com website the stated Number of ranges are different.

                                                  https://www.iblocklist.com/list.php?list=usrcshglbiilevmyfhse –> Number of ranges = 496, not 536
                                                  https://www.iblocklist.com/list.php?list=llvtlsjyoyiczbkjsxpf --> Number of ranges = 3201, not 3565

                                                  When I download the list manually to my Win7 PC, and do line count I get 496, & 3201 respectively.  I could understand if our lists were smaller if PFblocker was dedupping some ranges, but both our lists are larger than what we are being given.  This leads me to believe that PBflocker is not updating the list correctly.  Is my logic flawed?

                                                  Both my PFsense systems are production, so I can't Beta test at the moment.  If I can find some hardware to spare I'll PM you later.

                                                  Again thanks for all your help!
                                                  Ash,

                                                  http://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing

                                                  wcrowder - Maybe I have this backwards,  But, the use of CIDR would allow my PFsense list to be SMALLER than that of the Original list.  The fact that my PFsense list is larger does not make any sence if the PFsense is summarizing the IP ranges in order to reduce the amount of data to hold.  In addition the other lists I have are the same qty on pfsense and iblocklists.com.  Sorry I just do not see how this is applicable.  I see see my PFsense as not updating to the correct number of ranges provided by iblocklist.com.  Please understand I mean no disrespect, just don't see how CIDR is going to change only 2 lists.  If I am blindly missing something, please feel free to explain.  Thanks for your help.

                                                  BBcan177 - Do you think that my logic in my reply on "November 12, 2014, 12:18:45 pm"  makes since?  Thanks again for your help.

                                                  Ash,

                                                  1 Reply Last reply Reply Quote 0
                                                  • BBcan177
                                                    BBcan177 Moderator last edited by

                                                    wcrowder was leading you in the right direction  :)

                                                    Take for example this Range from the IBlock BT Spyware List

                                                    Range Format
                                                    221.181.73.214-221.181.73.221:

                                                    Converts to the following in CIDR Notation
                                                    221.181.73.214/31
                                                    221.181.73.216/30
                                                    221.181.73.220/31

                                                    So comparing Line Count in Range to CIDR is not going to be exact depending on the Ranges in a particular list.

                                                    Hope this makes it clearer.

                                                    1 Reply Last reply Reply Quote 0
                                                    • A
                                                      ashes00 last edited by

                                                      @BBcan177:

                                                      wcrowder was leading you in the right direction  :)

                                                      Take for example this Range from the IBlock BT Spyware List

                                                      Range Format
                                                      221.181.73.214-221.181.73.221:

                                                      Converts to the following in CIDR Notation
                                                      221.181.73.214/31
                                                      221.181.73.216/30
                                                      221.181.73.220/31

                                                      So comparing Line Count in Range to CIDR is not going to be exact depending on the Ranges in a particular list.

                                                      Hope this makes it clearer.

                                                      BBcan177 - Thanks, and I guess that settles that. :)  So it looks like the only way I have to validate that lists are updating is just if they change from time to time.

                                                      wcrowder - Sorry, & Thank you.  I guess I had that backwards.

                                                      Ash,

                                                      1 Reply Last reply Reply Quote 0
                                                      • First post
                                                        Last post

                                                      Products

                                                      • Platform Overview
                                                      • TNSR
                                                      • pfSense
                                                      • Appliances

                                                      Services

                                                      • Training
                                                      • Professional Services

                                                      Support

                                                      • Subscription Plans
                                                      • Contact Support
                                                      • Product Lifecycle
                                                      • Documentation

                                                      News

                                                      • Media Coverage
                                                      • Press
                                                      • Events

                                                      Resources

                                                      • Blog
                                                      • FAQ
                                                      • Find a Partner
                                                      • Resource Library
                                                      • Security Information

                                                      Company

                                                      • About Us
                                                      • Careers
                                                      • Partners
                                                      • Contact Us
                                                      • Legal
                                                      Our Mission

                                                      We provide leading-edge network security at a fair price - regardless of organizational size or network sophistication. We believe that an open-source security model offers disruptive pricing along with the agility required to quickly address emerging threats.

                                                      Subscribe to our Newsletter

                                                      Product information, software announcements, and special offers. See our newsletter archive to sign up for future newsletters and to read past announcements.

                                                      © 2021 Rubicon Communications, LLC | Privacy Policy