URL Table - Update Frequency trick



  • I needed a much smaller update frequency for a certain URL table (URL alias named "addresses_permit") in my system (every 5 minutes instead of every 1 day).

    What I did was:

    • install Cron package
    • modify the cron for /usr/bin/nice -n20 /etc/rc.update_urltables to */5  *  *  *  *
    • add a new cron with the command /usr/bin/touch -t 1001031305 /var/db/aliastables/addresses_permit.txt also with */5  *  *  *  *

    Since the rc.update_urltables script seems to look for the date of the downloaded file, I'm fooling it by setting the access/modification time well back in the past, thus triggering a forced update each time the script is ran.

    Did I proceed correctly?

    PS:
    Actually, my problem was that the txt contains FQDN addresses, which are not resolved every 5 minutes as described in the documentation. Re-downloading the whole table from the given URL alias triggers re-resolving the FQDNs too.



  • Setting the modification time back in the past is a nice idea. I'll keep it in mind.  :)

    I searched for a way to get a shorter update frequency, but didn't find anything. To get just one specific url table updated more frequently, I then modified the pfsense-utils.inc and build in a query for the table file name and changed the age for it.

    But after a pfSense update I'll have to do this modification again.



  • @viragomann:

    Setting the modification time back in the past is a nice idea. I'll keep it in mind.  :)

    I searched for a way to get a shorter update frequency, but didn't find anything. To get just one specific url table updated more frequently, I then modified the pfsense-utils.inc and build in a query for the table file name and changed the age for it.

    But after a pfSense update I'll have to do this modification again.

    Or you could push the modification to the maintainers, they may accept it as a patch and include it in a "next" release.



  • The skeleton is in the code for a "forceupdate" option to be easily added.  Pull request 1720 - Diagnostic Tables has this added feature so that when doing a manual url table update from diagnostics - tables, the update is forced regardless of age (file time stamp).

    It's been up there since 6/15 but hasn't been accepted.  If someone where make a pull request that includes just the force update feature, perhaps they would merge it.

    Then you could just put the option flag (forceupdate) and the table name to update on an rc.update_urltables command in cron.