System/Advanced/Notifications/Email ... "Test SMTP Settings" - no emails
-
You can set the default gateway in System > Routing > Gateways. But if you set that as a VPN you may need to add static routes to the VPN servers to allow them to connect.
-
@stephenw10 Hi Steve,
Ok, yeah, I remember. It also occured to me that I might need to create a new gateway for the tunnel.
I'm running a policy based connection, so that probably is not gonna happen. Maybe I could put 127.0.0.1 in my list of addresses that bypass the policy connection on the LAN. I certainly don't want that one sent out to the Internet, if that is what is happening, or maybe I need to do that on the CARP/WAN interface, just grab it and send it to the system routing table which seems more like it, if I can even do that.
Before I crash the system with introducing an unknown like that, I need to think about it.
What do you think?
Thanks for your input.
Roy
-
You can't policy route traffic from the firewall itself. It doesn't hit firewall rules on the LAN and outbound rules are applied after the routing decision.
It always uses the routing table to determine which gateway to use.
-
@stephenw10 Hi Steve,
Now please fix some of my ignorance.
"You can't policy route traffic from the firewall itself."
This I suspected as I fussed with this. I know that it makes no sense to even think it. "It (The firewall) can't be filtered by firewall rules on the LANs"
Such routing might cause an infinite loop and the booleans might also be confused.
"and outbound (WAN) rules are applied after the routing decision."
Yes, of course. You make it very obvious. No solution for me there.
"It always uses the routing table to determine which gateway to use."
Except with policy routing, here the routing table is bypassed right?And still no solution for me there.
I certainly don't want to make the tunnels the default gateway.
The odd thing is that it did work. Was it an artifact from before 2.7.0 when you all began to be more strict with the enforcement of the firewall rules?
And it does work on one server set still.
I suppose I should send you a netstat so that you can see what BSD sees.
Roy
-
@reberhar said in System/Advanced/Notifications/Email ... "Test SMTP Settings" - no emails:
"It always uses the routing table to determine which gateway to use."
Except with policy routing, here the routing table is bypassed right?I meant for traffic originating from the firewall itself. Traffic coming from internal subnets can be policy routed by pf yes.
Hmm, nothing much changed between 2.7.0 and 2.7.2 that might cause this.
Can I assume then that you do have outbound rules to block traffic leaving without using the VPN? And that they might be catching the SMTP traffic?
-
@stephenw10 Hi Steve,
I meant to delete that part about policy routing. I understand that is not involved with the firewall.
"Can I assume then that you do have outbound rules to block traffic leaving without using the VPN? And that they might be catching the SMTP traffic?"
Yes, I catch all my tunnel traffic before it gets to the routing policy rule and send it to the system routing table.
10.3.0.15 is the email server.
netstat -rWn
Routing tablesInternet: Destination Gateway Flags Nhop# Mtu Netif Expire default 10.12.1.1 UGS 17 1500 igb0 8.8.4.4 10.12.1.1 UGHS 11 1500 igb0 10.0.0.0/24 10.56.0.1 UGS 16 1500 ovpnc2 10.1.0.0/22 10.56.0.1 UGS 16 1500 ovpnc2 10.2.0.0/22 link#6 U 4 1500 igb5 10.2.0.1 link#9 UHS 5 16384 lo0 10.2.0.3 link#9 UHS 5 16384 lo0 10.3.0.0/23 10.56.0.1 UGS 16 1500 ovpnc2 10.5.0.0/23 10.56.0.1 UGS 16 1500 ovpnc2 10.12.1.0/24 link#1 U 1 1500 igb0 10.12.1.1 10.12.1.1 UGHS 11 1500 igb0 10.12.1.50 link#9 UHS 3 16384 lo0 10.12.1.51 link#9 UHS 3 16384 lo0 10.12.3.0/24 link#3 U 6 1500 igb2 10.12.3.1 10.12.3.1 UGHS 12 1500 igb2 10.12.3.50 link#9 UHS 7 16384 lo0 10.12.3.51 link#9 UHS 7 16384 lo0 10.13.0.0/23 10.56.0.1 UGS 16 1500 ovpnc2 10.22.10.3 link#9 UH 14 16384 lo0 10.56.0.0 link#9 UHS 13 16384 lo0 10.56.0.0/24 link#12 U 10 1500 ovpnc2 127.0.0.1 link#9 UH 2 16384 lo0 172.16.1.0/24 link#7 U 8 1500 re0 172.16.1.2 link#9 UHS 9 16384 lo0 Internet6: Destination Gateway Flags Nhop# Mtu Netif Expire ::1 link#9 UHS 1 16384 lo0 fe80::%igb0/64 link#1 U 5 1500 igb0 fe80::92e2:baff:fe10:7d48%lo0 link#9 UHS 4 16384 lo0 fe80::%igb2/64 link#3 U 9 1500 igb2 fe80::92e2:baff:fe10:7d4c%lo0 link#9 UHS 8 16384 lo0 fe80::%igb5/64 link#6 U 7 1500 igb5 fe80::6eb3:11ff:fe1c:24d7%lo0 link#9 UHS 6 16384 lo0 fe80::%re0/64 link#7 U 11 1500 re0 fe80::92b1:1cff:fe6c:f2c%lo0 link#9 UHS 10 16384 lo0 fe80::%lo0/64 link#9 U 3 16384 lo0 fe80::1%lo0 link#9 UHS 2 16384 lo0 fe80::%ovpnc2/64 link#12 U 13 1500 ovpnc2 fe80::92e2:baff:fe10:7d48%lo0 link#9 UHS 12 16384 lo0 -
Probably OK then.
Side note: If you have 6 igb NICs available don't use the re NIC.
-
@stephenw10 Thanks for your help.
6 igbs yes, and the one re (realtek) nic is only used for sync. It seems to handle that ok.
Concerning the email problem, I have someplace to start with the predictable failure at the firewall, I will fuss with it and probably eventually fix it. I have route, and traceroute to help me. I will do some packet capture too. It will be an opportunity to learn BSD better.
The pfSense servers are themselves solid and I have access to the logs over the tunnels. There are much worse headaches.
Thanks for your help.
Roy
-
This post is deleted! -
@reberhar Hello Steve,
I have been trying to get back to this firewall email failure through my OPENVPN tunnels.
I have been tinkering with it and noted that if I restart the tunnels, the client and sometimes the server, the emails work for a few minutes, and then they fail. I am going to try to study the routing table before and after, but I don't think that is changing.
I know that I need to fool with the system routing table carefully. I have found several ways to hang up the tunnels and even bomgar. Once, when I switched to TLS, I hung up the LAN on a server 2000 miles away.
On the positive side, I have learned how to make IPSEC work with OPENVPN, which, for me is helpful.
I am going to be checking the system patches that I have installed. I haven't noted anything in the logs, but will be checking again. Maybe I need to bump up the verbosity of the logs.
I checked the floating rules and didn't notice anything.
Does anything ring a bell with you?
Roy
-
Not really. I've never tried to send email over VPN though. But I would expect it to work as long as the system route to the email server is via the VPN gateway.
-
-
@reberhar Hi Steve,
So I have chased this problem and have found some helpful information, at least for me.
The network administrator before me setup individual OpenVPN site to site server tunnels for every remote connection. This was with shared certificates. There were 4 servers. Everything worked fine with the Email from the servers. Each route was unique with unique port numbers.
Enter the deprecation of shared certificates in OPEN VPN in pfSense and the threat of and update that would not include my install. We were directed to change to TLS.
So I reseached how to make the change and decided to use only one server which works fine execpt for this one glitch, of which I was unaware.
Open VPN does not keep multiple localhost connections alive. If I restart the openvpn the localhost works on whatever server sends mail first. As soon as a different remote unit sends mail, the first connection is lost until another openvpn restart.
It does not treat the same problem but something related.
Since I don't want to create 3 more openvpn servers I am going to switch to an external email server.
-
Hmm, well that second link seems quite clear but I don't think it applies to your situation. You're not trying to access services across the tunnel.
The first link is less clear. It seems to imply that OpenVPN takes traffic destined for localhost on the same box. And that if it fails the localhost traffic also fails. Which applies to you when trying to send an email to the sendmail service. But I'm not sure how that would affect different clients differently...
-
@stephenw10 Hi Steve,
No, neither one applies directly. The second one especially. It just showed that someone tried to fix a similar problem on Linux.
What is clear is that OpenVPN treats Localhost differently than a standard ipv4 or ipv6 number.
Of course that IP number is always the same no matter what machine. So localhost is a separate case. it was never meant to be an endpoint from outside the machine ... if I really understand its function correctly.
I will delete the second link.
-
Hmm, wait are you saying that when you set the clients to route all traffic over the VPN that grabs localhost traffic somehow? But it still works as long as the tunnel stays up?
That wouldn't apply to the server end if so. Maybe I've misread that....
-
@stephenw10 Hi Steve,
The post I include shows how Openvpn changes the BSD routing table. I still have to look at mine again and compare them.
I am however, reticent, to tweak the system at this level. Persistence is certain to be a problem.
As I type this, I guess what I really need to do is dive into the BSD documentation and just learn this stuff.
One thing is certain, OpenVpn does not necessarily maintain a stable dependable connection at the Localhost level.
This seems obvious.
Roy
-
@stephenw10 Only the pfSense email connection drops.
I have an email server internally.
-
Yeah, it seems likely there is is something more subtle at work here because it would break a lot of other things if localhost became completely inaccessible.
-
@stephenw10 Truly.
The routing table in BSD is sending the email query to the OpenVPN gateway address.
The next place to look is in OpenVPN, then the email server.
Now to try to find time to figure out what is happening in OpenVPN.
I wish there was a way to send these messages through the LAN interface. Then there would be a unique IP arriving at the email server. Making LAN a gateway creates a potential infinite loop.
Here is part of the Dovecot log. I wish it had more information.
My wife says that I am "too persistent." Maybe I just need to take the easy way around this obscure problem and just delete this post. Nuts, I like clean resolutions.
Dec 11 01:33:16 catalina-sme dovecot: imap-login: Login: user=<roy>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=18233, secured, session=<0aj3cvoosop/AAAB>
Dec 11 01:33:16 catalina-sme dovecot: imap(roy): Connection closed (No commands sent) in=0 out=360
Dec 11 12:01:06 catalina-sme dovecot: imap-login: Login: user=<roy>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=19902, secured, session=<upJDOAMpuox/AAAB>
Dec 11 12:01:06 catalina-sme dovecot: imap(roy): Connection closed (No commands sent) in=0 out=360The ne is in OpenVPN.
Dec 11 12:01:39 catalina-sme dovecot: imap-login: Login: user=<roy>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=19912, secured, session=<8vo8OgMpvIx/AAAB>
Dec 11 12:01:39 catalina-sme dovecot: imap(roy): Connection closed (No commands sent) in=0 out=360
Dec 11 12:03:51 catalina-sme dovecot: imap-login: Login: user=<roy>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=19924, secured, session=<DNsjQgMpvox/AAAB>
Dec 11 12:03:51 catalina-sme dovecot: imap(roy): Connection closed (No commands sent) in=0 out=360
Dec 11 12:05:00 catalina-sme dovecot: imap-login: Login: user=<roy>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=19935, secured, session=<U/48RgMpwIx/AAAB>
Dec 11 12:05:00 catalina-sme dovecot: imap(roy): Connection closed (No commands sent) in=0 out=360
Dec 11 12:05:49 catalina-sme dovecot: imap-login: Login: user=<roy>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=19951, secured, session=<bq0lSQMpxox/AAAB>
Dec 11 12:05:49 catalina-sme dovecot: imap(roy): Connection closed (No commands sent) in=0 out=360
Dec 11 12:06:46 catalina-sme dovecot: imap-login: Login: user=<roy>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=19963, secured, session=<PbSPTAMpyIx/AAAB>
Dec 11 12:06:46 catalina-sme dovecot: imap(roy): Connection closed (No commands sent) in=0 out=360
Dec 12 01:33:09 catalina-sme dovecot: imap-login: Login: user=<roy>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=21469, secured, session=<pvRokA4pUo9/AAAB>
Dec 12 01:33:09 catalina-sme dovecot: imap(roy): Connection closed (No commands sent) in=0 out=360
