Good results with URL Table Aliases package



  • All,

    I've been testing URL Table Aliases for about a week now and it is working well.

    Thanks to jimp for putting this together.

    FYI, as an example, one can only allow access in a rule to US IP blocks.  In that case, a URL might be:  http://www.countryipblocks.net/e_country_data/US_cidr.txt

    The countryipblocks.net site is a valuable resource for this package.  I am sure there are alternatives though.

    In order to update these lists, you do need to install the cron package and put something like this in your cron list:

    1 0 * * * root /etc/rc.update_urltables

    IMPORTANT:  It appears you cannot currently use this package with negation rules.

    thanks,
    Dave



  • I like this package too but i've found that there is a limit on the amount of IP addresses :-( Would like to use the full Level1 and China IPs list but it goes over a limit that I think is hard set within pfconf…

    Have you tried a really big list yet? Maybe its just my setup


  • Rebel Alliance Developer Netgate

    It defaults to 100,000 I think, but it can be set higher in pf. I'd just need to figure out the best way to get that done. You just need to add the right command to the pf ruleset:

    set limit table-entries 200000
    

    That would set the limit to 200,000. There are some memory considerations for increasing the size, but the pf site (http://www.openbsd.org/faq/pf/options.html) implies that 100,000 is only needed as a default if the system has less than 100MB of RAM.



  • Memory shouldn't be an issue right now, other theb maybe a performance hit.. My box has 2gigs of memory. Is there a way to manually up that limit? I did a search for the pf.conf file and can't seem to find it. Would be a nice addition to the Advanced options under the System meny.. Being able to increase this limit would benefit this package and the IP-Blocklist package(If it gets added to the package manager)

    thanks again!!


  • Rebel Alliance Developer Netgate

    The pf ruleset is generated by the system automatically, so this setting would have to be coded into filter.inc somewhere during the ruleset generation.

    If I bothered to do that though I may as well make a GUI field somewhere it could be tuned, though that may just be on 2.0



  • Something to add to the list for 2.0? The feature isn't there now, so I wouldn't mind waiting for 2.0 to have it and be stable. Thanks again!!


  • Rebel Alliance Developer Netgate

    URL Tables are in 2.0 as-is, but the table size tweak is not (yet).



  • Hello,

    Just tried, seems to work very well :)
    What kind of list format is supported ?
    Do you use some kind of regexp to find ip on each line, whatever the format used ?
    Does list such as http://list.iblocklist.com/?list=cn works ?
    Also, does list in compressed format (http://…./list.gz) works ?

    And please, where is the file to put 'set limit table-entries 200000', as rc.conf doesn't seems to be used in pfSense (?). Is it inc/filter.inc.
    I don't think starting to modify the php code is a good idea, but it seems it's not possible to change this setting through the web configurator.

    Thank you.


  • Rebel Alliance Developer Netgate

    It only supports plain text files with lists of IP addresses or CIDR masked subnets. No compressed files.

    The setting to increase the table entries goes in the pf ruleset. I added that as a GUI option in 2.0 but haven't added that to the 1.2.x package yet.



  • Thanks for your reply, but how can i change the pf ruleset ?
    Do i need to manually create a pf.conf file ?


  • Rebel Alliance Developer Netgate

    You cannot manually change it. You'd have to add the code to filter.inc.



  • after using pfsense for years now , this is a great addon, which I missed. I feel better to have a blocklist on the firewall beside moblock or iplist on the desktop.

    I asked me about the supported Blocklistformat. Jimp mentioned above that only plain text files are supported-what a pitty! Most availible blocklist are compressed and in IP RANGE format with additional information such as nipfilter.dat.gz or level1.gz .
    Has the  list to be only in CIDR format or Ip range too?
    I can download uncompress, convert the list manually and upload to pfsense box by sftp, but to which location?

    thanks
    pete



  • @jimp: ok, i modified filter.inc the following way :
    $rules .= "set limit table-entries 300000\n";
    in the filter_configure_sync function
    and it seems to work.
    @pete : you can also store the updated list to a local web server, then use the corresponding url into the table alias.


  • Rebel Alliance Developer Netgate

    @PFS00:

    after using pfsense for years now , this is a great addon, which I missed. I feel better to have a blocklist on the firewall beside moblock or iplist on the desktop.

    I asked me about the supported Blocklistformat. Jimp mentioned above that only plain text files are supported-what a pitty! Most availible blocklist are compressed and in IP RANGE format with additional information such as nipfilter.dat.gz or level1.gz .
    Has the  list to be only in CIDR format or Ip range too?
    I can download uncompress, convert the list manually and upload to pfsense box by sftp, but to which location?

    thanks
    pete

    Someone else has another package ("ip blocklist") which can work on the level1 list and ranges and such. The author of that package converted it to pf (at my suggestion) and it may be viable for that now. I haven't used it myself yet.

    The problem with ranges is they take a lot of time to calculate. IPs and CIDRs are what pf supports natively. Ranges and compressed files have to be processed before they can be used by pf, which increases the burden on the router.



  • @jideel:

    @jimp: ok, i modified filter.inc the following way :
    $rules .= "set limit table-entries 300000\n";
    in the filter_configure_sync function
    and it seems to work.
    @pete : you can also store the updated list to a local web server, then use the corresponding url into the table alias.

    Where in the filter_configure_sync section did you put your statement in?

    I was thinking of adding it before
    $rules .= "\n";
    $rules .= "set skip on pfsync0\n";



  • I notice that is package can not be uninstalled and I was wondering what were the reasons and could it possibly be detrimental in anyway.


  • Rebel Alliance Developer Netgate

    If you uninstall it and leave a URL table alias configured, the behavior would be unpredictable and could result in the filter rules failing to load.



  • I installed it.  I installed cron.  I added a US CIDR txt url for allowing only us ips. I added the cron job specified on the first post.  It does not seem to be working.  Using a proxy to test, i could still gain access to pcs behind the firewall.


  • Rebel Alliance Developer Netgate

    Did you use the alias in a firewall rule? Can you post a screencap of your firewall rules? If it got through, then it either matched a rule above it in the ruleset that passed it, it wasn't used in a rule properly, or the proxy you used was really in the US and not another country.



  • i havent had a chance to check yet, but just to mention I am running NanoBSD version.  Does that matter?


  • Rebel Alliance Developer Netgate

    @killervette:

    i havent had a chance to check yet, but just to mention I am running NanoBSD version.  Does that matter?

    I thought it was safe to use on NanoBSD but I don't recall at the moment.



  • So how should i add an alias to my rules?  I want to only allow us ips to connect.  So, i add the CIDR US list to the URL alias.  Then what?


  • Rebel Alliance Developer Netgate

    You make a new alias, choose the URL table type, put in the URL for the US IPs list.

    Then use the alias in a rule like any other alias. You'd make a rule on WAN like so:

    pass <protocol>from <single host="" or="" alias,="" us_ip_alias="">, port: any, to <local_ip>, port: <whatever>.

    The real contents of that rule are up to you and whatever your app is.</whatever></local_ip></single></protocol>



  • Will that by default block all other ips? do i need to put a rule below allow us ips to block all?


  • Rebel Alliance Developer Netgate

    All traffic is blocked on pfSense by default.

    If you have no other pass rule that matches the same traffic, then all other traffic will be blocked.


Locked