Broken port forwarding

[eableson]

@netblues Yup - The mail server has no problems going out and getting stuff and I can send email to it from the internal network which get properly forwarded along to the defined smarthost. So basically anything initiated by the mail server works fine, but the SYN ACK responses get to the pfSense box and it's trying to go out through a route that doesn't work for it.

Here I can see after each initiated connection that the gateway tried to return the ACK, but can't find a route to the originator.

[netblues]

@eableson Need to check nat outbound.
Switch to manual and create rules for your two connections.

[eableson]

@netblues Like this?

Still no go...

Even with the two rules:

[netblues]

I don't get it. Why 104.47.9.51 is unreachable from pf?
From the command line of pf, can you ping the world from adsl wan interface?
Can you ping your adsl wan side from remote?

[eableson]

@netblues Ah, now things are getting interesting. Current state :

• Disconnected the 4G connection (but still in the configuration)
• Turned the OpenVPN connections back on to be sure I'm fixing the problem in a state that won't just get broken again once I turn it back on

So at the moment from the pfSense box:

• ping 8.8.8.8 from WANADSL IP: OK
• ping 8.8.8.8 from LAN IP: No route to host

But from a computer on the LAN, ping to 8.8.8.8 works fine... because I had set the priority routing up so that it would take the default route only when trying to go to other internal networks. When I look at the trace route, the first hop is the ADSL WAN Gateway, not the internal LAN IP address.

So the current routing table looks like this:

Destination        Gateway            Flags     Netif Expire
78.226.49.0/24     link#2             U           re1
78.226.49.193      link#2             UHS         lo0
127.0.0.1          link#4             UH          lo0
172.16.11.1        link#8             UH       ovpnc2
172.16.11.2        link#8             UHS         lo0
172.16.12.1        link#9             UH       ovpnc3
172.16.12.2        link#9             UHS         lo0
192.168.2.0/24     172.16.12.1        UGS      ovpnc3
192.168.5.0/24     172.16.11.1        UGS      ovpnc2
192.168.6.0/24     172.16.11.1        UGS      ovpnc2
192.168.10.0/24    link#7             U           ue0
192.168.10.1       link#7             UHS         lo0

And attached are the current rules NAT & FW:

0_1545226719685_pfctl-rules.txt

So the issues seems to be that in a multi-wan setup that has VPN connections, the default routing table will only include known internal networks and if you need to go to the internet, it has to be via a gateway group. So the question I'm asking myself is if I should be declaring the OpenVPN connections as physical interfaces so that their routing rules get handled like regular interfaces.

[eableson]

@eableson @netblues Found it! After staring at the combination of the routing table and the "no route to host" message, I realised that there was no default route listed in the routing table.

Digging around, this appears to be a side of effect of using gateway groups and setting the Default gateway IPv4 to a gateway group instead of setting it to a physical interface. Setting the Default gateway to the ADSL WAN connection add it as the default route and things start working again...

[netblues]

@eableson So, it has nothing to do with natting and port forward, as expected :)

[eableson]

@netblues As usual, the source of the problem was elsewhere, but brought to light by adding NAT and Port Forwarding to the mix. This one could stand to have a tech note - but before I ask I'm going to go back and reread the documentation about setting up multi-wan

[netblues]

@eableson Well, the norm is sending your default gw to the internet and not the vpn routes. (if being a server)

[eableson]

@netblues Well - it's not terribly clear about the correct use of the default gateway option when in multi wan mode. The trick is that in this configuration, most traffic should go out the 4G (high performance) connection, failing back to the ADSL when it drops (usually once every day or two). So in that case, the default configuration would be the default gateway as this gateway group, not a physical interface.

Then there are services that have to run on the ADSL line since it's a direct connection with a fixed IP. But they are very much the exception. But in all cases, they point to the pfSense as the default gateway from their perspective and it's responsible for knowing the best route to take.

[KOM]

If we all got $1 for every "Gah! pfSense is hacked/broken/whatever!" and it turned out to be a configuration issue, we would all be able to retire.