pfSense email flood issue

stephenw10

Yes, I agree.

Do they just continually add up or is it sending you, say, all the alerts from the last 10mins each time?

Samlink

@stephenw10 they keep adding up, the last mail had:
-pppoe disconnected
-pppoe reconnected
-updated DNS record
-the start of the reboot (for stopping this email flood)
-the completion of the bootup

stephenw10

That's still all inside the last 10mins though. Does it eventually stop sending the oldest alerts? How long does that take?

Samlink

@stephenw10 I don't know, I had to restart it.
I'm trying to replicate manually what happened today to see if this behavior keeps happening.

Another thing I didn't mentioned before: I also have telegram notifications enabled, and they are working normally, they are not being replicated.

EDIT:
it's happening again: unplugged the ups (simulating a blackout), unplugged the wan2 eth, plugged in the ups after 15 seconds and then plugged in the wan2.
It's been 12 min and I received over 60 emails.

Is there anything I can look while the problem is ongoing?

stephenw10

Do you see those in /tmp/notices?

Do you see a huge list of alerts in the GUI?

Samlink

@stephenw10 sorry I had to reboot it and didn't check this. For a little over 20 min I received the same text 98 times. They kept adding up to the same emails.

Tomorrow I'll do another try and check the tmp folder as you suggested.

Samlink

@stephenw10
I didn't found any notices file(s) or directory under /tmp.
Attached are the tmp list during the problem, after ups and wan2 reconnection and after a complete reboot.
during.txt after_reboot.txt after.txt

DT 0

Similar problem here after 2.6.0 update.

Triggered by (NUT) UPS CyberPower on battery and/or DynDNS updated IP Address on WAN (pppoe0).

Sample Gateway Log Entries:
May 18 22:17:54 dpinger 26971 WAN_PPPOE : sendto error: 65
May 18 22:17:53 dpinger 26971 WAN_PPPOE : sendto error: 65
May 18 22:17:53 dpinger 26971 WAN_PPPOE : sendto error: 65
May 18 22:17:52 dpinger 26971 WAN_PPPOE : sendto error: 65
May 18 22:17:52 dpinger 26971 WAN_PPPOE : sendto error: 65
May 18 22:17:51 dpinger 26971 WAN_PPPOE : sendto error: 65
May 18 22:17:51 dpinger 26971 WAN_PPPOE : sendto error: 65
May 18 22:17:50 dpinger 26971 WAN_PPPOE : sendto error: 65
May 18 22:17:50 dpinger 26971 WAN_PPPOE : sendto error: 65
May 18 22:17:49 dpinger 26971 WAN_PPPOE : sendto error: 65
...and on and on.

Only a reboot will stop the emails.
Also, the continuous email notification "DynDNS updated IP Address on WAN (pppoe0)" is indicating a stale WAN IP, not the current/updated one.
Also, after the 2.6.0 update, I still require the patch in order for my Dynamic DNS Clients (No-IP) to update.
I have disabled notifications for the time being.
Thanks for your help.

Samlink

Hello, any update/advice on this issue?

DT 0

My issue was solved with a fresh install of 2.6.0 and a restore of the configuration. Only took a few minutes and all is well now. The re-install also gave me the opportunity to update the file system to zfs.

Samlink

@dt-0 thank you, definitely going to give it a try the next week

stephenw10

Unclear still if you're seeing the same alerts being continually sent or just a massive number of alerts.
Have you been able to check the dashboard when this happens yet?

Steve

Samlink

@stephenw10 I'm receiving the same alerts over and over. Even today, my pppoe went down only 1 time in 6 minutes, but I received the same email every 10 seconds or so.

stephenw10

Check the file: /var/db/notices_lastmsg.txt

That file should store the last message sent and prevent sending the same message twice.

Steve

Samlink

@stephenw10 at 7:09 I unplugged ups and wan2 and started flooding.
First email I received is from the UPS, followed by the wan2 down:

The file notices_lastmsg.txt is updated with:

MONITOR: WAN2_PPPOE has packet loss, omitting from routing group WAN_FAILOVER

The following minutes the same last email is getting repeated every 10 sec or so, here are the next 3:
[Edited]
(date in the email is the same 7:09:43, but I received them up until 7:13 before I rebooted).

During the flood, the file notices_lastmsg.txt didn't change unless something new happened, only then it changed with the new notification, like at 7:12, where I plugged back in the wan2:

Now the emails are being sent also with the new notifications, they are adding up.
Afterwards I rebooted and got the last email:

This is the content of notices_lastmsg.txt now:

7:09:43 MONITOR: WAN2_PPPOE has packet loss, omitting from routing group WAN_FAILOVER
142.250.184.99|xxx.xxx.xxx.xxx|WAN2_PPPOE|10.029ms|1.699ms|24%|down|highloss
7:12:47 MONITOR: WAN2_PPPOE is available now, adding to routing group WAN_FAILOVER
142.250.184.99|xxx.xxx.xxx.xxx|WAN2_PPPOE|11.602ms|0.092ms|0.0%|online|none
7:12:48 DynDNS updated IP Address on WAN2 (pppoe0) to xxx.xxx.xxx.xxx
7:14:01 pfSense is rebooting now.
7:15:18 Bootup complete

PS:
The ups came back online at 7:12:52, I've got only one email relative to it, ,in the middle of the other wan2 ones:

stephenw10

Hmm, this should prevent that happening unless the notices are sent as 'forced':
https://github.com/pfsense/pfsense/blob/RELENG_2_6_0/src/etc/inc/notices.inc#L334

Probably time to open a bug report and get some developer eyes on it. I failed to replicate it here though.

https://redmine.pfsense.org/

Steve

Samlink

@stephenw10 I'll briefly explain my setup and how I can replicate the problem.
-2 PPPoE WANs (tier1 and tier2).
-One ups (by cyberpower, I don't think this matter but the user @DT-0 happened to also have one).

Unplug the ups , after about 5 seconds unplug the wan2. Doing so I'm able to replicate the email flood problem.

I'm going to open a bug report for now, thanks

stephenw10

Are you running NUT or apcupsd? I assume you must be from the alerts you have.

Samlink

@stephenw10 I'm running NUT.

DT 0

The problem persists. Just had the WAN go down for a few minutes and come back up. Email notification was sent to advise "DynDNS updated IP Address on WAN (pppoe0) to" This is where the problem (bug) is -- the same email notification keeps on repeating. Perhaps because this is the last notification sent. Only way to stop is disable SMTP notifications and reboot. I thought a re-install of 2.6.0 solved this but no. This issue was introduced in 2.6.0

My dynamic DNS client is No-IP so I don't know if the bug is unique to No-IP. I am also using NUT (CyberPower).

Also, contrary to the release 2.6.0 notes indicating the No-IP complex password issue was solved, it is not. The patch is still required. I have yet to try another DNS client but I might.