Gateway down - no notification received via SMTP



  • Hi,
    I have configured pfsense to send SMTP notifications via my LAN SMTP server. It works perfectly for the test notification and also for dyndns ip changes. However, pfsense does not notify me if my gateway goes down:

    Apr 27 03:45:35 rc.gateway_alarm 59722 >>> Gateway alarm: WAN_DHCP (Addr:80.109.220.1 Alarm:1 RTT:10.218ms RTTsd:4.909ms Loss:21%)
    Apr 27 03:45:35 check_reload_status updating dyndns WAN_DHCP
    Apr 27 03:45:35 check_reload_status Restarting ipsec tunnels
    Apr 27 03:45:35 check_reload_status Restarting OpenVPN tunnels/interfaces
    Apr 27 03:45:35 check_reload_status Reloading filter
    Apr 27 03:45:36 check_reload_status Linkup starting igb0
    Apr 27 03:45:36 kernel igb0: link state changed to DOWN

    Do I need to activate that somewhere?

    I am using:

    2.4.4-RELEASE-p2 (amd64)
    built on Wed Dec 12 07:40:18 EST 2018
    FreeBSD 11.2-RELEASE-p6

    Best regards!



  • Hi,

    I think you need to install "mailreport" under Package Manager.



  • Hi, I checked it out, but "mailreport" just sends out periodic reports. I would like to be instantly notified if the gateway goes down. I read in other articles that this is working for other users with SMTP notifications turned on. Thus I am wondering why it does not work for me...



  • If WAN is down, it can't be sent... You need a failover connection for that.



  • Hmm, why? The LAN interface (still up) and the SMTP server are in the same private subnet. I even configured the static IP of the SMTP server instead of the FQDN to prevent any DNS related issues.


  • LAYER 8 Global Moderator

    Do you have pfsense set to reset states on gateway down?

    system / adv / misc

    State Killing on Gateway Failure
    Flush all states when a gateway goes down The monitoring process will flush all states when a gateway goes down if this box is checked.
    

    Wonder if that could conflict with talking to your smtp server?



  • Nope.

    de2d99a0-b4e7-4db8-95e2-cf7b69526c09-image.png



  • Normally you don‘t need that. If your WAN is down, even a smtp on LAN can‘t send the mail.



  • @mrsunfire said in Gateway down - no notification received via SMTP:

    Normally you don‘t need that. If your WAN is down, even a smtp on LAN can‘t send the mail.

    So why is it then possible to successfully send and receive a test notification (within the LAN) with a disabled WAN interface?

    b3f0fd60-2bee-4be9-9db8-a4de64c28f2b-image.png



  • @phaze75

    Assuming that your email server logs never log the email attempt?



  • @chpalmer
    No, the SMTP server didn't log an email attempt at the incident "Apr 27 03:45:36 kernel igb0: link state changed to DOWN". But it logged my manually triggered SMTP test notifications properly.

    I suspect pfsense did not try to send an email at all (bug?). What supports this suspicion is that pfsense properly logged my test notification (see below), but it did not log a notification of the "igb0: link state changed to DOWN" incident:

    Apr 27 09:04:19 php-fpm /system_advanced_notifications.php: Message sent to @****. OK



  • Under System/Advanced/Notifications you're sure you did not check the first box "Disable SMTP"? Sometimes this happens because you think check a box enables something.




  • Netgate Administrator

    Do you see notifications when the WAN comes back up?

    Do you see gateway alarms logged when the WAN is down? If you just disable the WAN removing the gateway monitor on it I don't believe you will see an alert.

    Steve



  • @stephenw10 said in Gateway down - no notification received via SMTP:

    Do you see notifications when the WAN comes back up?

    No.

    Do you see gateway alarms logged when the WAN is down?

    Yes, see my first post. The first line of the log contains the gateway alarm.

    If you just disable the WAN removing the gateway monitor on it I don't believe you will see an alert.

    That is clear. I just disabled the WAN manually to prove that SMTP (test) notifications can still be sent by pfsense even if the WAN adapter is down/disabled.

    I suspect there might be an issue of pfsense preventing a gateway alarm to trigger the corresponding SMTP notification.


  • Netgate Administrator

    You don't have gateway action disabled in the gateway setup?

    Though I would still expect the alarm to trigger an alert in that case. Not sure I've ever tested that though.

    Steve



  • @stephenw10 said in Gateway down - no notification received via SMTP:

    You don't have gateway action disabled in the gateway setup?

    No. See below.

    e2d1f2c1-96f7-497f-ae4b-0d7e51428e89-image.png



  • @stephenw10 So, where do we go from here? Should I file a bug?



  • I guess this is it :
    @phaze75 said in Gateway down - no notification received via SMTP:

    I suspect pfsense did not try to send an email at all (bug?)

    I took a look at what happens when a gateway (WAN) goes down.
    I'm pretty sure that there are no mail notifications attempts in that case.


  • LAYER 8 Global Moderator

    This is kind of been a gray area for a while - I am not aware of any docs that spell out what triggers a notification.

    Would be a nice if there was some doc listing what exactly will trigger a smtp notification... Better yet would be a way to toggle different notifications on or off..

    Notification of wan down is prob never thought of since more than likely smtp notification would be going to outside smtp server. I just use a "free" outside monitoring service that send me email when it can not ping my wan IP..



  • @phaze75
    /etc/inc/gwlb.inc - line https://github.com/pfsense/pfsense/blob/30335336358db3bcdc0ede634a4f81b7f3273c7b/src/etc/inc/gwlb.inc#L120
    add

    		notify_all_remote("WAN went down (and when I receive this, it will be up again !)");
    

    This part of the file will now look like :

    ....
    	if (!$gateway['action_disable']) {
    		$params .= "-C \"{$alarm_cmd}\" ";	/* Command to run on alarm */
    		notify_all_remote("WAN went down !");
    	}
    ....
    
    Works for me ^^
    I ripped out the WAN cable - counted to 35, put it back in .... 
    The logs mentioned a mail being send - several, actually - my DynDNS's also kicked in.
    I received also :
    

    Notifications in this message: 1

    11:22:06 WAN went down (and when I receive this, it will be up again !)

    
    Btw : I'm not advising you to mess up your pfSense ;)
    
    
    @johnpoz said in [Gateway down \- no notification received via SMTP](/post/840825):
    > Would be a nice if there was some doc listing what exactly will trigger a smtp notification... Better yet would be a way to toggle different notifications on or off..
    
    +1 !


  • @Gertjan said in Gateway down - no notification received via SMTP:

    Btw : I'm not advising you to mess up your pfSense ;)

    Fiddling around with the code/scripts will bring me exactly there. So I will refrain from leaving the standard. Thanks anyway!


  • LAYER 8 Global Moderator

    If you just want an easy way to get notified when your wan goes offline - other than your users screaming at you ;) There are many a FREE place that will do this for you..



  • @johnpoz said in Gateway down - no notification received via SMTP:

    your wan goes offline

    Yeah !! Sending a mail with WAN down means : receiving the mail when WAN comes up. Rather useless.



  • @Gertjan said in Gateway down - no notification received via SMTP:

    Yeah !! Sending a mail with WAN down means : receiving the mail when WAN comes up. Rather useless.

    Guys, am I writing in Chinese? 😉 I am repeating myself (see the quote of my second post below) when I tell you that the pfsense's LAN interface, the SMTP-Server, the Mail-Server and all Email-Clients are all in the same private LAN, that continue to work even with the WAN down. So no, it is not useless, because I would instantly receive pfsense's email notification within my private LAN, if pfsense would just send it. So simple, no strings attached. 😉

    @phaze75 said in Gateway down - no notification received via SMTP:

    Hmm, why? The LAN interface (still up) and the SMTP server are in the same private subnet. I even configured the static IP of the SMTP server instead of the FQDN to prevent any DNS related issues.



  • Non, not Chinese, I understood that part : pfSense can deliver to a local mail on server just fine.
    What I do understand just know : your mail box is also on LAN.
    I saw your "mail server", and was things "oh, that would be a mail forwarder".

    My ISP is quiet tolerant, but hosting a mail server locally is impossible : port 25 is blocked for incoming connection. Outgoing 25 is only allowed to the ISP mail servers. In Europe, most ISP's handle this way.



  • @Gertjan said in Gateway down - no notification received via SMTP:

    My ISP is quiet tolerant, but hosting a mail server locally is impossible : port 25 is blocked for incoming connection. Outgoing 25 is only allowed to the ISP mail servers. In Europe, most ISP's handle this way.

    I agree regarding to ISP's private contracts. I am lucky. ISP's business contracts have to enable the full IP port range, otherwise companies could not host their SMTP servers on premise. So there will be others too hosting on premise SMTP servers behind pfsense firewalls who could use that feature... 😉



  • It might be nice (for the rest of us) to receive an email once the gateway is back online, simply stating the gateway was down at <datetime> for #h:#m.
    perhaps an event on the gateway up event with some log analysis to calc the duration?

    Any way to do that without an external NMS watching everything?


  • LAYER 8 Global Moderator

    @Gertjan said in Gateway down - no notification received via SMTP:

    Outgoing 25 is only allowed to the ISP mail servers. In Europe, most ISP's handle this way.

    That is not just EU, most consumer lines are the same way in the US. And even if your isp doesn't block - most major players block you sending them mail if your IP is listed as being dynamic.



  • @johnpoz said in Gateway down - no notification received via SMTP:

    That is not just EU, most consumer lines are the same way in the US. And even if your isp doesn't block - most major players block you sending them mail if your IP is listed as being dynamic.

    Aren't there any business users of pfsense using business lines that are not blocked by the ISP? I did not understand that pfsense was only for home users using consumer lines.

    By the way: since almost 8 years our business is extensively using the on-premise SMTP server via a dynamic IP address including a DynDNS setup. In all those years there has never been an occasion where one of our emails has been blocked. And we have also communicated to major players.



  • What is now to do with the result of this long controversial discussion that even contains a potential solution from @Gertjan (see below)? Is this now taken care of? Do I need to submit a bug report/feature request?

    @Gertjan said in Gateway down - no notification received via SMTP:

    ....
    	if (!$gateway['action_disable']) {
    		$params .= "-C \"{$alarm_cmd}\" ";	/* Command to run on alarm */
    		notify_all_remote("WAN went down !");
    	}
    ....
    

    Works for me ^^
    I ripped out the WAN cable - counted to 35, put it back in ....
    The logs mentioned a mail being send - several, actually - my DynDNS's also kicked in.



  • It's just a feature request.
    These belong here : pfSense redmine feature requests.

    Btw, this :

     		notify_all_remote("WAN went down !");
    

    isn't really perfect.
    A setup can have more then one WAN interface. It should at least mention which interface.
    I didn't check if this part of the code only executes ones, or could execute several times.
    And of course, the feature would make sense if a local mail server is available. IMHO : this is very rare.


  • Netgate Administrator

    In 2.4 you can set a failover group as the default gateway so you should be able to send and email as long as one WAN is still up. Though I've never tested that.

    Steve


Log in to reply