@SteveITS Final update. The issue definitely seems resolved. Thanks again.

Solution for you guys having the same problem:

Create an Interface on site A for both OVPN-Tunnels. Than assign the automativ created Gateways in the Gateway Group.

Dont forget to do NAT on the Cloud side.

@Bambos
pfSense itself never needs outbound NAT rules. It's rather the outside world, who needs it.

The point is to enable the outside world to communicate with your local devices, which probably resides inside a private subnet.
If the outside world has no route to your subnet pointing to your (VPN) interface IP, you need to masquerade the source IP on outgoing packets with the interface IP with an outbound NAT rule.

If you have a site-to-site VPN the remote site has usually a route for your local subnets. So there is no rule needed then.

@AndyRH said in SSH cant connect:

if it is enabled it is more or less just IPTables

If it was iptables or ufw I would agree with you - but this firewalld is zone based.. And such a firewall coming up in a different zone or not any zone would explain his symptoms exactly.

https://docs.fedoraproject.org/en-US/quick-docs/firewalld/

All Fedora Editions install, configure and activate the firewall by default. No further action is required. The only exception is Cloud Edition, which relies on the higher level cloud system.

That sounds like to me its using firewalld and not iptables or ufw, etc.

I would see if its running

systemctl status firewalld

if it is, shut it down, does ssh now work?

sudo systemctl stop firewalld

If it running, you should be able to see what zone its in and settings with

firewall-cmd --list-all

@jecker Can you show your gateway groups? And example rules?

https://docs.netgate.com/pfsense/en/latest/multiwan/load-balance-and-failover.html

@viragomann
I think "skip rules, when gateway is down" in system/Advanced/Miscellaneous that you mentioned is the point that i didn't know.
Thank you so much.

The following should work,

pfSense VIVO and Claro WAN interfaces are on different subnets (e.g., don't configure the LAN-side addresses of the VIVO and Claro gateways to both be 192.168.1.1) firewall rules permit access from your internal network to the VIVO and Claro management addresses

@rbthomas
Don't know about step by step instructions, but Netgate documentation is pretty good.
https://docs.netgate.com/pfsense/multiwan/load-balance-and-failover.html

@Gertjan You are right, the ipv6 adoption here is pitiful. I just wanted to try my hand on it too see what I can do with it. But, it breaks my running system so I will stick with v4 for now.

@dcreations you can use FRR package, OSPF and BFD.
BFD with default settings, will try a hello packet every 50ms, and if it looses 3 packets, will switch the traffic to the backup path.
You can also set one side to administratively down, and by doing that, you don't need to change the cost at the other side to shift the traffic.

I know the SG2100 only supports the older SIM cards

Looks like I may have found my issues.

I had setup a policy route for testing the connection. I had also deleted that route before opening this request here so I had discarded it. But I think because my computer that was used for the test kept re-establishing sessions, the state table kept re-adding the states.

I have since killed all states after bringing the secondary connection back online (I kept it unplugged unless testing as I didn't want to kill my metered connection) and rebooted the computer that was used in testing. now the only states I am seeing on the secondary are the expected ones for gateway monitoring.

Thank you @marcg and @viragomann for your assistance.

@marcosm said in Failback state killing with "Automatic" failover?:

When you use an IP for gateway monitoring, a route is created for it via the gateway.

Got it. I switched the secondary monitoring address to 9.9.9.9 since I don't use Quad9 for DNS resolution. The extra states on the secondary, while the primary is up, disappeared. Thanks!

[24.03-RELEASE][admin@pfSense.home.arpa]/root: pfctl -i igc1.95 -s state igc1.95 icmp 192.168.95.2:24256 -> 9.9.9.9:24256 0:0

@not-a-bot2024
Why do you bother with the ISP DNS if it doesn't work reliably?
The DNS Resolver on pfSense requests the DNS root servers directly, unless you're using the forwarding mode.

@johnpoz Thank you!

@Stee7ic So you have double NAT situation at all your sites?
As in Public IP -> Modem -> 192.168.100.1 -> pfsense -> LAN IP
So I'm assuming when you say pfsense is 10.120.10.254, that is the LAN IP?

It shouldn't matter what the pfsense WAN IP happens to be, which would be unique for each site as well (at least the public IP).

I'm assuming with double NAT that the modems are set up to do port forward of ports 500, 4500 or whatever you use for IPSec?