pfBlockerNG re-plays ip_block.log file after reloading/updating?

cukal

pfsense 2.6.0 & pfBlockerNG-devel 3.2.0_4

I have configured syslog-ng to send /var/log.pfblockerng/ip_block.log to a dedicated udp syslog input on Graylog. But my plan to further process the received events in Graylog seems foiled by some strange logic that escapes me.

It seems that when pfBlockerNG reloads/updates it replays the already outputted lines from the logfile and re-dumps them verbatim in the same logfile again?

I tested it by truncating the file, hitting reload and nothing gets appended as the log is empty at that point. Trigger a block rule, it outputs the block event to the log, hit reload pfBlockerNG and at the end of the reload/update it replays each logline with the same timestamps again?

Is this expected behaviour and can this be controlled/configured somehow?
From an event standpoint each time it reload/updates it triggers new events, albeit without changing the timestamp but from a tailing point-of-view these are new events being produced... I'm genuinely intrigued to know the origin of this unique behaviour, as I've never encountered an application behaving this way before.

Thanks,
C

jrey

@cukal

This is caused by the way pfBlockerNG manages the log files.

Part of the problem is that the logs are only "trimmed back" to number of lines shown on the configuration page. So typically the default is 20,000 lines.

depending on how and when you are sending the records to your syslog in relationship to the cron job (or perhaps as part of that process) you could drop or create duplicates.

you might see if the solution presented here works for you:

https://forum.netgate.com/topic/163248/issue-entire-unified-log-monitored-by-syslog-ng-gets-transmitted-every-time-pfblockerng-php-job-is-executed-in-cron?_=1699921042105

Setting the files to null every time cron update runs certainly stops duplications, but also pretty much destroys any local reporting (if there is value in it for you)

Another approach I have seen, but can't locate right now is to have it only send the records that are being "trimmed off" during the cron process. Of course log file is then off by the number of records remaining in the file until the next trim.

Setting the records retained lower kills local logging, setting it higher and particularly on smaller systems kills performance (especially local reporting)

Ideally it would be better to have pfB send them records in real time, but it is not currently built that way. i seem to recall seeing a feature request for this to be added.

cukal

Thanks for your reply!

Seems it's done in /usr/local/pkg/pfblockerng/pfblockerng.inc : pfb_log_mgmt.
Fixed my use case there by zeroing out ip_block.log file and not retaining the max log lines.
Just another 'remark' when upgrading pfSense!