System/Advanced/Notifications/Email ... "Test SMTP Settings" - no emails
-
Hi All,
I have been fussing with this for the last day or two, lost at first like I usually am. I thought I wasn't receiving emails from my pfSense servers for the usual events, UPS offline, CARP notifications etc.
So I used the "Test SMTP Settings¨ and found that it worked on 3 of my 5 server installs.
Great, well sort of ...
But on the other two I got
On port 465
Could not send the message to admin@catalan1.lan -- Error: Failed to connect to ssl://mail.catalan1.lan:465 [SMTP: Failed to connect socket: Operation timed out (code: -1, response: )]
On port 25
Could not send the message to admin@catalan1.lan -- Error: Failed to connect to mail.catalan1.lan:25 [SMTP: Failed to connect socket: Operation timed out (code: -1, response: )]
So what is happening with the other two? Gee, I don't know ...
I tried changing the usual stuff, ports 25, 465, certificate stuff, passwords, DNS ... nothing.
Then I did the SMTP test ...
telnet mail.catalan1.lan 25
Trying 10.3.0.15...
Connected to mail.catalan1.lan.
Escape character is '^]'.
220 catalina-sme.catalan1.lan ESMTP
HELO mail.catalan1.lan
250 catalan1.lan Hi pc-00216.catalan1.lan [10.3.0.216]; I am so happy to meet you.
MAIL FROM: test@railsware.com
250 test@railsware.com, sender OK - how exciting to get mail from you!or from a remote location
telnet mail.catalan1.lan 25
Trying 10.3.0.15...
Connected to mail.catalan1.lan.
Escape character is '^]'.
220 catalina-sme.catalan1.lan ESMTP
HELO mail.catalan1.lan
250 catalan1.lan Hi Unknown [10.2.0.49]; I am so happy to meet you.
MAIL FROM: test@railsware.com
250 test@railsware.com, sender OK - how exciting to get mail from you!Nice friendly programmers those SMTP folks.
Thanks for any suggestions.
Roy
-
What do you see in the system logs when you run that test?
What pfSense version are you running?
Steve
-
@stephenw10 Hi Steve,
I am using pfSense 2.7.2 Community Version. HA/CARP
The logs reflected the same errors, but
Well I was wrong, the emails are not getting through, but I had a thought. I did not do my testing from BSD but from a machine on the lan.
So I got into BSD and did the same tests.
Bingo ...
The ping fails and the SMTP test fails from the BSD prompt.
So I am trying to do this through the OPENVPN tunnels. The email notification fails through those tunnels. The tunnels are working otherwise for my administration of the servers.
They WERE all working for email too.
When I started tweaking the connections they stopped sending. One would come back sporadically. Just weird.
So how did I break it?? I am not sure.
I have been fussing with the tunnels. I usually just have OPENVPN tunnels. However, I am connecting with IPSEC to a Dream Machine and I also have begun pushing DNS over the road warriors on pfSense, but I don't think the DNS is the problem. I am beginning to focus on the IPSEC connection. I made it available over my entire OPENVPN network, but I am still a little green on doing IPSEC to OPENVPN.
And tunnels configured badly can have bizarre results.
So I put things back to the way they were before my fussing with extending IPSEC to the OPENVPN tunnels. Still no emails.
It just occurred to me to trying clearing the state tables.
I will look at the logs for IPSEC and OPENVPN and clear the state tables when I can.
It is Saturday so I am trying to stand back and rest a little. It is so easy to spend too much time on these things.
I still appreciate your comments and seasoned wisdom.
Roy
-
@reberhar pfSense 2.7.2 Community Version
-
Traffic from pfSense itself will always use the default route. Is that set via one of the VPNs or via the WAN?
Do you have rules to block outbound traffic that isn't using the VPN?
-
@stephenw10 Hi Steve,
I think that the default is the WAN so that is probably where the problem lies. I can ping from the LAN and LAN-CARP interfaces but not from the WAN of course.
So to expose my ignorance, where do I change the default interface for the firewall? There are bells ringing in my head saying, "Don't you remember, you did that?", but with some much water under the dam, you can save me a little time by reminding me.
Thanks,
Roy
-
You can set the default gateway in System > Routing > Gateways. But if you set that as a VPN you may need to add static routes to the VPN servers to allow them to connect.
-
@stephenw10 Hi Steve,
Ok, yeah, I remember. It also occured to me that I might need to create a new gateway for the tunnel.
I'm running a policy based connection, so that probably is not gonna happen. Maybe I could put 127.0.0.1 in my list of addresses that bypass the policy connection on the LAN. I certainly don't want that one sent out to the Internet, if that is what is happening, or maybe I need to do that on the CARP/WAN interface, just grab it and send it to the system routing table which seems more like it, if I can even do that.
Before I crash the system with introducing an unknown like that, I need to think about it.
What do you think?
Thanks for your input.
Roy
-
You can't policy route traffic from the firewall itself. It doesn't hit firewall rules on the LAN and outbound rules are applied after the routing decision.
It always uses the routing table to determine which gateway to use.
-
@stephenw10 Hi Steve,
Now please fix some of my ignorance.
"You can't policy route traffic from the firewall itself."
This I suspected as I fussed with this. I know that it makes no sense to even think it. "It (The firewall) can't be filtered by firewall rules on the LANs"
Such routing might cause an infinite loop and the booleans might also be confused.
"and outbound (WAN) rules are applied after the routing decision."
Yes, of course. You make it very obvious. No solution for me there.
"It always uses the routing table to determine which gateway to use."
Except with policy routing, here the routing table is bypassed right?And still no solution for me there.
I certainly don't want to make the tunnels the default gateway.
The odd thing is that it did work. Was it an artifact from before 2.7.0 when you all began to be more strict with the enforcement of the firewall rules?
And it does work on one server set still.
I suppose I should send you a netstat so that you can see what BSD sees.
Roy
-
@reberhar said in System/Advanced/Notifications/Email ... "Test SMTP Settings" - no emails:
"It always uses the routing table to determine which gateway to use."
Except with policy routing, here the routing table is bypassed right?I meant for traffic originating from the firewall itself. Traffic coming from internal subnets can be policy routed by pf yes.
Hmm, nothing much changed between 2.7.0 and 2.7.2 that might cause this.
Can I assume then that you do have outbound rules to block traffic leaving without using the VPN? And that they might be catching the SMTP traffic?
-
@stephenw10 Hi Steve,
I meant to delete that part about policy routing. I understand that is not involved with the firewall.
"Can I assume then that you do have outbound rules to block traffic leaving without using the VPN? And that they might be catching the SMTP traffic?"
Yes, I catch all my tunnel traffic before it gets to the routing policy rule and send it to the system routing table.
10.3.0.15 is the email server.
netstat -rWn
Routing tablesInternet: Destination Gateway Flags Nhop# Mtu Netif Expire default 10.12.1.1 UGS 17 1500 igb0 8.8.4.4 10.12.1.1 UGHS 11 1500 igb0 10.0.0.0/24 10.56.0.1 UGS 16 1500 ovpnc2 10.1.0.0/22 10.56.0.1 UGS 16 1500 ovpnc2 10.2.0.0/22 link#6 U 4 1500 igb5 10.2.0.1 link#9 UHS 5 16384 lo0 10.2.0.3 link#9 UHS 5 16384 lo0 10.3.0.0/23 10.56.0.1 UGS 16 1500 ovpnc2 10.5.0.0/23 10.56.0.1 UGS 16 1500 ovpnc2 10.12.1.0/24 link#1 U 1 1500 igb0 10.12.1.1 10.12.1.1 UGHS 11 1500 igb0 10.12.1.50 link#9 UHS 3 16384 lo0 10.12.1.51 link#9 UHS 3 16384 lo0 10.12.3.0/24 link#3 U 6 1500 igb2 10.12.3.1 10.12.3.1 UGHS 12 1500 igb2 10.12.3.50 link#9 UHS 7 16384 lo0 10.12.3.51 link#9 UHS 7 16384 lo0 10.13.0.0/23 10.56.0.1 UGS 16 1500 ovpnc2 10.22.10.3 link#9 UH 14 16384 lo0 10.56.0.0 link#9 UHS 13 16384 lo0 10.56.0.0/24 link#12 U 10 1500 ovpnc2 127.0.0.1 link#9 UH 2 16384 lo0 172.16.1.0/24 link#7 U 8 1500 re0 172.16.1.2 link#9 UHS 9 16384 lo0 Internet6: Destination Gateway Flags Nhop# Mtu Netif Expire ::1 link#9 UHS 1 16384 lo0 fe80::%igb0/64 link#1 U 5 1500 igb0 fe80::92e2:baff:fe10:7d48%lo0 link#9 UHS 4 16384 lo0 fe80::%igb2/64 link#3 U 9 1500 igb2 fe80::92e2:baff:fe10:7d4c%lo0 link#9 UHS 8 16384 lo0 fe80::%igb5/64 link#6 U 7 1500 igb5 fe80::6eb3:11ff:fe1c:24d7%lo0 link#9 UHS 6 16384 lo0 fe80::%re0/64 link#7 U 11 1500 re0 fe80::92b1:1cff:fe6c:f2c%lo0 link#9 UHS 10 16384 lo0 fe80::%lo0/64 link#9 U 3 16384 lo0 fe80::1%lo0 link#9 UHS 2 16384 lo0 fe80::%ovpnc2/64 link#12 U 13 1500 ovpnc2 fe80::92e2:baff:fe10:7d48%lo0 link#9 UHS 12 16384 lo0 -
Probably OK then.
Side note: If you have 6 igb NICs available don't use the re NIC.
-
@stephenw10 Thanks for your help.
6 igbs yes, and the one re (realtek) nic is only used for sync. It seems to handle that ok.
Concerning the email problem, I have someplace to start with the predictable failure at the firewall, I will fuss with it and probably eventually fix it. I have route, and traceroute to help me. I will do some packet capture too. It will be an opportunity to learn BSD better.
The pfSense servers are themselves solid and I have access to the logs over the tunnels. There are much worse headaches.
Thanks for your help.
Roy
-
This post is deleted! -
@reberhar Hello Steve,
I have been trying to get back to this firewall email failure through my OPENVPN tunnels.
I have been tinkering with it and noted that if I restart the tunnels, the client and sometimes the server, the emails work for a few minutes, and then they fail. I am going to try to study the routing table before and after, but I don't think that is changing.
I know that I need to fool with the system routing table carefully. I have found several ways to hang up the tunnels and even bomgar. Once, when I switched to TLS, I hung up the LAN on a server 2000 miles away.
On the positive side, I have learned how to make IPSEC work with OPENVPN, which, for me is helpful.
I am going to be checking the system patches that I have installed. I haven't noted anything in the logs, but will be checking again. Maybe I need to bump up the verbosity of the logs.
I checked the floating rules and didn't notice anything.
Does anything ring a bell with you?
Roy
-
Not really. I've never tried to send email over VPN though. But I would expect it to work as long as the system route to the email server is via the VPN gateway.
-
-
@reberhar Hi Steve,
So I have chased this problem and have found some helpful information, at least for me.
The network administrator before me setup individual OpenVPN site to site server tunnels for every remote connection. This was with shared certificates. There were 4 servers. Everything worked fine with the Email from the servers. Each route was unique with unique port numbers.
Enter the deprecation of shared certificates in OPEN VPN in pfSense and the threat of and update that would not include my install. We were directed to change to TLS.
So I reseached how to make the change and decided to use only one server which works fine execpt for this one glitch, of which I was unaware.
Open VPN does not keep multiple localhost connections alive. If I restart the openvpn the localhost works on whatever server sends mail first. As soon as a different remote unit sends mail, the first connection is lost until another openvpn restart.
It does not treat the same problem but something related.
Since I don't want to create 3 more openvpn servers I am going to switch to an external email server.
-
Hmm, well that second link seems quite clear but I don't think it applies to your situation. You're not trying to access services across the tunnel.
The first link is less clear. It seems to imply that OpenVPN takes traffic destined for localhost on the same box. And that if it fails the localhost traffic also fails. Which applies to you when trying to send an email to the sendmail service. But I'm not sure how that would affect different clients differently...
