Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    pfSense email flood issue

    Scheduled Pinned Locked Moved
    General pfSense Questions
    5
    29
    2.2k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • stephenw10S
      stephenw10 Netgate Administrator
      last edited by

      Yes, I agree.

      Do they just continually add up or is it sending you, say, all the alerts from the last 10mins each time?

      S 1 Reply Last reply Reply Quote 0
      • S
        Samlink @stephenw10
        last edited by

        @stephenw10 they keep adding up, the last mail had:
        -pppoe disconnected
        -pppoe reconnected
        -updated DNS record
        -the start of the reboot (for stopping this email flood)
        -the completion of the bootup

        9ed9db9e-8a46-43e4-955d-ef6a5605643f-immagine.png

        1 Reply Last reply Reply Quote 0
        • stephenw10S
          stephenw10 Netgate Administrator
          last edited by

          That's still all inside the last 10mins though. Does it eventually stop sending the oldest alerts? How long does that take?

          S 1 Reply Last reply Reply Quote 0
          • S
            Samlink @stephenw10
            last edited by Samlink

            @stephenw10 I don't know, I had to restart it.
            I'm trying to replicate manually what happened today to see if this behavior keeps happening.

            Another thing I didn't mentioned before: I also have telegram notifications enabled, and they are working normally, they are not being replicated.

            EDIT:
            it's happening again: unplugged the ups (simulating a blackout), unplugged the wan2 eth, plugged in the ups after 15 seconds and then plugged in the wan2.
            It's been 12 min and I received over 60 emails.

            Is there anything I can look while the problem is ongoing?

            1 Reply Last reply Reply Quote 0
            • stephenw10S
              stephenw10 Netgate Administrator
              last edited by

              Do you see those in /tmp/notices?

              Do you see a huge list of alerts in the GUI?

              S 2 Replies Last reply Reply Quote 0
              • S
                Samlink @stephenw10
                last edited by

                @stephenw10 sorry I had to reboot it and didn't check this. For a little over 20 min I received the same text 98 times. They kept adding up to the same emails.

                Tomorrow I'll do another try and check the tmp folder as you suggested.

                1 Reply Last reply Reply Quote 1
                • S
                  Samlink @stephenw10
                  last edited by

                  @stephenw10
                  I didn't found any notices file(s) or directory under /tmp.
                  Attached are the tmp list during the problem, after ups and wan2 reconnection and after a complete reboot.
                  during.txt after_reboot.txt after.txt

                  1 Reply Last reply Reply Quote 0
                  • D
                    DT 0
                    last edited by

                    Similar problem here after 2.6.0 update.

                    Triggered by (NUT) UPS CyberPower on battery and/or DynDNS updated IP Address on WAN (pppoe0).

                    Sample Gateway Log Entries:
                    May 18 22:17:54 dpinger 26971 WAN_PPPOE : sendto error: 65
                    May 18 22:17:53 dpinger 26971 WAN_PPPOE : sendto error: 65
                    May 18 22:17:53 dpinger 26971 WAN_PPPOE : sendto error: 65
                    May 18 22:17:52 dpinger 26971 WAN_PPPOE : sendto error: 65
                    May 18 22:17:52 dpinger 26971 WAN_PPPOE : sendto error: 65
                    May 18 22:17:51 dpinger 26971 WAN_PPPOE : sendto error: 65
                    May 18 22:17:51 dpinger 26971 WAN_PPPOE : sendto error: 65
                    May 18 22:17:50 dpinger 26971 WAN_PPPOE : sendto error: 65
                    May 18 22:17:50 dpinger 26971 WAN_PPPOE : sendto error: 65
                    May 18 22:17:49 dpinger 26971 WAN_PPPOE : sendto error: 65
                    ...and on and on.

                    Only a reboot will stop the emails.
                    Also, the continuous email notification "DynDNS updated IP Address on WAN (pppoe0)" is indicating a stale WAN IP, not the current/updated one.
                    Also, after the 2.6.0 update, I still require the patch in order for my Dynamic DNS Clients (No-IP) to update.
                    I have disabled notifications for the time being.
                    Thanks for your help.

                    1 Reply Last reply Reply Quote 0
                    • S
                      Samlink
                      last edited by

                      Hello, any update/advice on this issue?

                      1 Reply Last reply Reply Quote 0
                      • D
                        DT 0
                        last edited by

                        My issue was solved with a fresh install of 2.6.0 and a restore of the configuration. Only took a few minutes and all is well now. The re-install also gave me the opportunity to update the file system to zfs.

                        S 1 Reply Last reply Reply Quote 2
                        • S
                          Samlink @DT 0
                          last edited by

                          @dt-0 thank you, definitely going to give it a try the next week👍

                          1 Reply Last reply Reply Quote 0
                          • stephenw10S
                            stephenw10 Netgate Administrator
                            last edited by

                            Unclear still if you're seeing the same alerts being continually sent or just a massive number of alerts.
                            Have you been able to check the dashboard when this happens yet?

                            Steve

                            S 1 Reply Last reply Reply Quote 0
                            • S
                              Samlink @stephenw10
                              last edited by

                              @stephenw10 I'm receiving the same alerts over and over. Even today, my pppoe went down only 1 time in 6 minutes, but I received the same email every 10 seconds or so.

                              1 Reply Last reply Reply Quote 0
                              • stephenw10S
                                stephenw10 Netgate Administrator
                                last edited by

                                Check the file: /var/db/notices_lastmsg.txt

                                That file should store the last message sent and prevent sending the same message twice.

                                Steve

                                S 1 Reply Last reply Reply Quote 0
                                • S
                                  Samlink @stephenw10
                                  last edited by stephenw10

                                  @stephenw10 at 7:09 I unplugged ups and wan2 and started flooding.
                                  First email I received is from the UPS, followed by the wan2 down:
                                  4f2e5292-1863-4019-980f-3735d506ab39-immagine.png
                                  179ec81d-6243-402d-aecc-46d3d15a6d11-immagine.png
                                  The file notices_lastmsg.txt is updated with:

                                  MONITOR: WAN2_PPPOE has packet loss, omitting from routing group WAN_FAILOVER

                                  The following minutes the same last email is getting repeated every 10 sec or so, here are the next 3:
                                  [Edited]
                                  (date in the email is the same 7:09:43, but I received them up until 7:13 before I rebooted).

                                  During the flood, the file notices_lastmsg.txt didn't change unless something new happened, only then it changed with the new notification, like at 7:12, where I plugged back in the wan2:
                                  f9530525-82b5-4ea5-9632-582db66b0b77-immagine.png
                                  Now the emails are being sent also with the new notifications, they are adding up.
                                  Afterwards I rebooted and got the last email:
                                  6660cc68-ec22-4b0c-bf25-ed4aa226132e-immagine.png
                                  This is the content of notices_lastmsg.txt now:

                                  7:09:43 MONITOR: WAN2_PPPOE has packet loss, omitting from routing group WAN_FAILOVER
142.250.184.99|xxx.xxx.xxx.xxx|WAN2_PPPOE|10.029ms|1.699ms|24%|down|highloss
7:12:47 MONITOR: WAN2_PPPOE is available now, adding to routing group WAN_FAILOVER
142.250.184.99|xxx.xxx.xxx.xxx|WAN2_PPPOE|11.602ms|0.092ms|0.0%|online|none
7:12:48 DynDNS updated IP Address on WAN2 (pppoe0) to xxx.xxx.xxx.xxx
7:14:01 pfSense is rebooting now.
7:15:18 Bootup complete

                                  PS:
                                  The ups came back online at 7:12:52, I've got only one email relative to it, ,in the middle of the other wan2 ones:
                                  95371b0b-6a63-413c-8b80-fed8efd46ed6-immagine.png

                                  1 Reply Last reply Reply Quote 0
                                  • stephenw10S
                                    stephenw10 Netgate Administrator
                                    last edited by

                                    Hmm, this should prevent that happening unless the notices are sent as 'forced':
                                    https://github.com/pfsense/pfsense/blob/RELENG_2_6_0/src/etc/inc/notices.inc#L334

                                    Probably time to open a bug report and get some developer eyes on it. I failed to replicate it here though.

                                    https://redmine.pfsense.org/

                                    Steve

                                    S 1 Reply Last reply Reply Quote 1
                                    • S
                                      Samlink @stephenw10
                                      last edited by

                                      @stephenw10 I'll briefly explain my setup and how I can replicate the problem.
                                      -2 PPPoE WANs (tier1 and tier2).
                                      -One ups (by cyberpower, I don't think this matter but the user @DT-0 happened to also have one).

                                      Unplug the ups , after about 5 seconds unplug the wan2. Doing so I'm able to replicate the email flood problem.

                                      I'm going to open a bug report for now, thanks

                                      1 Reply Last reply Reply Quote 0
                                      • stephenw10S
                                        stephenw10 Netgate Administrator
                                        last edited by

                                        Are you running NUT or apcupsd? I assume you must be from the alerts you have.

                                        S 1 Reply Last reply Reply Quote 0
                                        • S
                                          Samlink @stephenw10
                                          last edited by

                                          @stephenw10 I'm running NUT.

                                          1 Reply Last reply Reply Quote 0
                                          • D
                                            DT 0
                                            last edited by

                                            The problem persists. Just had the WAN go down for a few minutes and come back up. Email notification was sent to advise "DynDNS updated IP Address on WAN (pppoe0) to" This is where the problem (bug) is -- the same email notification keeps on repeating. Perhaps because this is the last notification sent. Only way to stop is disable SMTP notifications and reboot. I thought a re-install of 2.6.0 solved this but no. This issue was introduced in 2.6.0

                                            My dynamic DNS client is No-IP so I don't know if the bug is unique to No-IP. I am also using NUT (CyberPower).

                                            Also, contrary to the release 2.6.0 notes indicating the No-IP complex password issue was solved, it is not. The patch is still required. I have yet to try another DNS client but I might.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post