Pfsense sending tens of thousands of notifications a day

Armstrong

I am using a custom notification script to track OpenVPN logins, which has worked flawlessly for the past few years, but recently I have begun to receive tens of thousands of emails from the system repeatedly sending the same notifications.

<?php
        require_once("/etc/inc/notices.inc");
        $local_connect_value = " user_name: " . getenv('common_name') . " vpn_client_ip: " . getenv('ifconfig_pool_remote_ip') . " from: " . getenv('trusted_ip') . " on " . date('F j, Y, g:i a');
        if ( strrchr (__FILE__ , 'disconnect') ) {
        $local_connect_value .= ", during : " . getenv('time_duration') . " seconds, received : " . getenv('bytes_received') . " bytes, send : " . getenv('bytes_sent') ." bytes. DISCONNRECTED.";
        }
        notify_all_remote($local_connect_value);
?>

The only way I have found to stop it is to kill the notify_monitor.php process via the shell. Disabling SMTP notifications, removing the SMTP server details and even removing the notification script from OpenVPN does not stop it, I have to kill the process itself.

I have tried to dig in to Pfsense source code but am a bit confused. Apparently the notification queue should be at /tmp/notices but this doesn't exist. I can't figure out where this repeat notification is being stored or what is causing it to trigger repeatedly every few seconds. The notification lock files in /tmp are equally empty.

Has anyone else experienced this behaviour?

stephenw10

Looks like this: https://redmine.pfsense.org/issues/14031

Are you running 2.6?

Armstrong

@stephenw10 I sure am.

Thanks for this! Thing even got me listed on Spamhaus!

Armstrong

Anyone know how to add a patch from Redmine? The post states to but I've never done it before and it isn't explained.

@stephenw10 are you aware?

viragomann

@Armstrong
Install the System_Patches package.
There you can enter the patch ID or URL.

Armstrong

@viragomann Thanks, that doesn't work unfortunately, the "Apply" button never appears.

viragomann

@Armstrong
After entering the ID c5faa351c1ef6d4555478a7f50b3a16ece7e0b2a and a description, I can save it.
Then I get a fetch button.

This doesn't work on your installation?

Armstrong

@viragomann Yes I get a fetch button but no Apply button, and the patch isn't applied when I verify the file itself. Is this normal behaviour?

Armstrong

@viragomann OK doesn't seem like normal behaviour. If I debug I get the following:

Patch does not apply cleanly (detail)
Patch does not revert cleanly (detail)

Patching file usr/local/bin/notify_monitor.php using Plan A...
Hunk #1 failed at 20.
1 out of 1 hunks failed while patching usr/local/bin/notify_monitor.php

viragomann

@Armstrong
Yes, I get the same.
Obviously the patch is now for 2.7.

Armstrong

@viragomann Thanks.

Who needs reliability in production anyway! Guess I will wait for 2.7 to resolve this if the patch isn't even available any more.

stephenw10

Hmm, curious. Digging....

stephenw10

Ah, it's because it's a patch against 23.01. It's in the recommended patch list there.

It's in 23.05 and 2.7-beta so you can upgrade to either of those now. Otherwise 2.7 is not far off.

Armstrong

@stephenw10 Thanks for that, appreciate your time.

I assume I can also achieve something of the same effect by simply disabling the NUT notifications?

When is 2.7 due?

stephenw10

Simply reducing the notifications will certainly help.

2.7 is in Beta and it's looking good. So unless something show stopping is found I would anticipate 'weeks'. Hard to be more accurate than that.

Steve