Thank you so much for you answer, jimp. You're always very helpful.

"3. The mobile IPsec tunnel would need to be set to use the same failover group as the dyndns entry"

I've tried setting it up just as you said in this topic:
https://forum.pfsense.org/index.php?topic=58784.msg315915#msg315915

Everything works fine, except that the ipsec.conf won't reload automatically when the DynDNS is updated (https://forum.pfsense.org/index.php?topic=58784.msg628621#msg628621). I had to manually reload configs/service in order for it to acknowledge the group's new active IP. I'd appreciate if you could help me out with that too.

Anyway, there's no way to make the VPN accessible simultaneously through 2 different IPs when using mobile Ipsec, right? Is OpenVPN the only way I can make it work in pfSense?

4. You'll probably need to activate default gateway switching under System > Advanced on the Misc tab

I don't think that's needed. I configured a gateway group in load-balance mode (same tiers) and set it up as the Ipsec "interface". Obviously it wouldn't work, as there can only be one IP at a given time in my ipsec.conf's "left=" parameter, but I could see that the traffic always leaves through the same interface in which it came in. Needless to say, it works just the same when in failover mode. Not that it really matters, just saying that pfSense handles it very well.

It depends on what you did. If you added a static route that overlapped something used by another service such as OpenVPN, then removed it, you should be able to save/apply in the relevant section that originally created the route to get it back. In the case of OpenVPN, just restart the VPN with an edit/save to bring that back.

Thanks!
Exactly what I wanted. :D

(Now I'll have to duplicate some firewall rules to outgoing NAT module)

That is entirely up to the client side in these cases. If you can configure the client to only send traffic for your far-side VPN subnets across, then it should be closer to what you want. But with IKEv2 in this sort of setup the client makes all those decisions.

If you only have your server network(s) in the P2 list and have checked "Provide a list of networks…" on the mobile clients tab, clients might respect that and stop sending all traffic over IKEv2, but support varies by client.

Depends on what the data center is giving you. If it is two different interfaces then no problem.

If it is two different gateways on the same interface I think you can still do Multi-WAN but it'd be an uncommon scenario. And it would be a single point of failure on your side.

I don't see any reason you can't put multiple gateways on one interface, make a gateway group, and policy route to it though I have never seen it done (which doesn't mean it hasn't).

Only one of them can be the default gateway.

Thank you for your answer.

It is what currently happening … reaching the GUI from the inside on any of the Virtual IP's that are meant to be NAT to some LAN server (common way of NAT'ing)

The only things I set up are :

NAT on those 3 Virtual IPs some rules to have it accessed from the outside

To get it maybe much more clear, here is some weird behavior linked :

Situation :
Ping from my laptop connected onto the VPN (L2TP/IPSec) with some 192.168.x.x address to a public IP that can be reached perfectly from the outside 91.134.183.74 (em0) which is NAT'ed to 10.0.10.4 in the LAN (em1).

Here is the output :

taken on the if l2tp0 :

tcpdump -n -i l2tp0 host 192.168.x.x and port not 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on l2tp0, link-type NULL (BSD loopback), capture size 65535 bytes

10:10:46.586013 IP 192.168.x.x > 91.134.183.74: ICMP echo request, id 41134, seq 77, length 64
10:10:47.590574 IP 192.168.x.x > 91.134.183.74: ICMP echo request, id 41134, seq 78, length 64
10:10:48.594678 IP 192.168.x.x > 91.134.183.74: ICMP echo request, id 41134, seq 79, length 64

taken on the LAN (to see if it reaches the targeted server) :

tcpdump -n -i em1 host 192.168.x.x and port not 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em1, link-type EN10MB (Ethernet), capture size 65535 bytes

(…. NOTHING here)

taken on the WAN interface :

tcpdump -n -i em0 host 192.168.x.x and port not 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 65535 bytes

10:09:58.470654 IP 91.134.183.74 > 192.168.x.x: ICMP echo reply, id 41134, seq 29, length 64
10:09:59.472724 IP 91.134.183.74 > 192.168.x.x: ICMP echo reply, id 41134, seq 30, length 64
10:10:00.475666 IP 91.134.183.74 > 192.168.x.x: ICMP echo reply, id 41134, seq 31, length 64
10:10:01.476897 IP 91.134.183.74 > 192.168.x.x: ICMP echo reply, id 41134, seq 32, length 64
10:10:02.474787 IP 91.134.183.74 > 192.168.x.x: ICMP echo reply, id 41134, seq 33, length 64
10:10:03.476701 IP 91.134.183.74 > 192.168.x.x: ICMP echo reply, id 41134, seq 34, length 64
10:10:04.480793 IP 91.134.183.74 > 192.168.x.x: ICMP echo reply, id 41134, seq 35, length 64

Hereby the routing table of the FW :

Internet:
Destination        Gateway            Flags      Netif Expire
default            91.134.183.254    UGS        em0
10.0.10.0/24      link#2            U          em1
10.0.10.1          link#2            UHS        lo0
91.134.183.0/24    link#1            U          em0
91.134.183.72      link#1            UHS        lo0
91.134.183.73/32  link#1            U          em0
91.134.183.74/32  link#1            U          em0
91.134.183.75/32  link#1            U          em0
91.134.210.0/24    link#1            U          em0
91.134.210.4      link#1            UHS        lo0
127.0.0.1          link#6            UH          lo0
192.168.x.x        link#7            UHS        lo0
192.168.x.x      link#7            UH        l2tp0

Seems like there is at least an issue here as the FW seems to send that traffic to the internet … which could be acceptable if at least they came back ...
.. or, better, to have the NAT rules applied on those addresses.
I did not add those routes at any point and cannot have them deleted neither.

Here the routing entries on the laptop :

91.134.183.72      10.0.0.1          UGHS            5    13449    en0
91.134.183.74      link#10            UHWIi          1    1199    ppp0

10.0.0.1 is the default GW of my CheckPoint here at home which is correct and works fine.
But for the other WAN IP, it sends it to the tunnel. Same story, I did not add those records. The VPN connection propagated it here for me.

So the first issue is pretty clear : the FW is sending its traffic onto the wrong interface (question #1 : how to get rid of that strange routing entries)

But the second one is at least as weird : the FW seems to answer to a request to 91.134.183.74 instead of transferring it to the according server on the LAN. He definitively things he is the guy that own that IP (which is technically correct , except the NAT part)

I already re-installed the entire pfSense twice from scratch and always the same issues … depression is watching me ...

Thanks

/O.

[edit]

Hmmmmm …. sounds like Reflection had not been enabled on that 1:1 NAT entry ...
So from the LAN, it is working now.

But still not from the VPN. It is allowed by the rules but a ping to some public VIP entering the l2tp0 if ig getting out to the WAN ... :-/

You have to assign the OpenVPN interface under Interfaces>assign in order for it to have gateways available.

The gateways' monitor IPs are basically the equivalent of IOS's tracking. If your backup path is going to be a VPN, you can use OpenVPN for that, which will give you an additional gateway.

@ipodgorny:

Thanks, that clears it up. I thought that Non-Local was both layer 2 and 3, now it all makes sense.

That's just the nature of IP routing, you can't tell a system to use a gateway that isn't on the same layer 2 network (or otherwise directly-connected, like a VPN).

Did you ever find a solution to your problem?

start by testing with a clean install (=no packages whatsoever)

if the problem is the same with clean install, check system logs for error, check interface configuration, check wiring, check cpu usage

if the problem is fixed, add packages, one at a time & try again until you find the culprit.

That never would have worked. Interface group rules don't have reply-to because they can't by their nature. One rule applies to multiple interfaces. So your non-default WAN wouldn't have gotten replies out, unless it was sourced from the IP facing you on on the PPPoE concentrator, where it'd know how to reply. It might have changed to a diff source IP on the ISP side that needed reply-to to route back.

Cool thanks.  That's what I thought.  How about the "Outgoing Network Interfaces" setting for Unbound … what's the current best practice on that?  I notice it defaults to "All" but I usually change it to "LAN + Localhost" otherwise DNS queries forwarded over IPSEC tunnels do not function.  Seems to work well enough but I don't know if that's something I should be doing differently?

Go to the routing gateway page (System / Routing / Gateways) and select the edit action (looks like a pencil) for the gateway you want to configure. Scroll down and click on the "Display Advanced" button and look for "Latency thresholds"

In general you want to start a new thread rather than using old threads. Old threads don't receive as much attention as new threads.

Give that you are using a satellite connection, you need to adjust the latency thresholds to higher values. Check the latency under load using ping, and then adjust the threshold values for the connection on the gateway edit page (System/Routing/Gateways/Edit) in the advanced section.

Pretty sure same as here https://forum.pfsense.org/index.php?topic=111108.0

I have tried pinging from the console and it was fine.
DHCP was fine.
Everything is up on the server.
Internet on my LAN is not working., it can't connect to the internet.