User definable refresh time for IP lists


  • Moderator

    Hi,

    I'd propose a more custom approach for IP list update intervals :)
    As I have that problem ATM I'd really like for pfBlockerNGs default Cronjob to run at least every 2 or 5 minutes (I suppose it only checks if it has something to refresh, doesn't it?) and give the user the ability - at least for IP4/6 lists - to refresh them how he likes at a minimum that is defined by the cronjob (so 2 or 5 minutes or - if suitable even every minute).

    Why that? As the example PRI1 list (in the devel tree) is quite a nice job, pfBNG can then actually be used as a black- or whitelist backend/service for self-hosted lists. We do this for example in combination with Fail2Ban and a custom self-written Ban/Unban action and using the DB capability of Fail2Ban. So if an IP gets blocked, it doesn't get blacklisted on the host itself but the host writes it to a central database. That database has a web frontend that can now get called to create an IP list for the whole customer service and then blocks these IPs for all services and hosts of said customer, not only the one that triggered the action. That is a good solution for clustered or loadbalanced services as well.

    BUT to use that to full extend we need to import those lists in a timing that is closer to 5min instead of only every hour because if we detect e.g. an SQL injection and blacklist the IP the firewall should block it (almost) immediatly. As pfSense itself has no way to read lists < 1d in interval, pfBNG could fill that gap quite nicely!

    Thanks
    Jens


  • Moderator

    @jegr

    I have written an api script a few years ago that can add IPs to an IP customlist. I have not released it yet, but should do that at some point. This can add/remove IPs from any IP customlist. This would have to be scripted to fit your network requirements.

    Its can be problematic to run the pfBlockerNG cron task every 5 mins depending on how many IP Blocklists are being used, and if the Reputation IP options are being used (which takes some time to process). So it was that reason they I didn't add more cron options. The problem is also exacerbated when DNSBL is used, and depending on the number of feeds and if the TLD option is enabled.


  • Moderator

    Then how about splitting up IP and DNSBL tasks into two jobs? As DNSBL links into unbound, any "refresh" or reload results in the resolver loosing its cache so reloading it too often perhaps isn't that desirable anyway ;) So we could have two jobs, one for those jobs with delays >1h or even >4h and one only working on IP lists.

    Running every 5min also would not be that hard, as you already have timers in place, that most lists don't want to be updated every 30min to 1h. So it would only be some kind of advanced option or option that is only allowed with a warning or for internal use/custom lists that allow such high refresh counts. That would also IMHO minimize the problems with reputation IP etc. as the fast refresh times would only be used with very few custom lists?

    Greets
    Jens



  • @jegr said in User definable refresh time for IP lists:

    every 2 or 5 minutes

    Pure IP-blacklisting (like Fail2Ban + iptables) should not involve any interactions with the DNS (Resolver) => no reload or cache issue.

    Not related, but : I like the Resolver (unbound) but what about using the real thing for ones and for all : bind.


  • Moderator

    I don't have any dealings with unbound in my use case. I was just pointing out to @BBcan177 that work on DNSBL parts of pfBNG don't have happen on the same timetable/cronjob as IP-only based processing.

    I'd like to see pfSense-core have the ability to import IP lists via URL on a 2-5min interval but until that may or may not be happening, pfBNG is the next best thing to do that. Even our loadbalancer has the ability to include custom black and whitelists and refresh them every minute (configurable per list) and block those IPs from accessing the balanced-IP. But as I'd like to have IP processing on a central point - firewall that is - I'd love to see the ability come to pfSense or pfBNG so you have the ability for a quick-reaction list to take action without waiting an hour or more to refresh.

    BTW: Devel version already includes techniques for just "refreshing" instead of reloading unbound without cache-loss. But besides that I don't see a use for BIND anywhere in this thread ;)

    Greets,
    Jens