@Bob-Dig This solution worked also for me. Thank you!

Thanks so much Virgomann. It makes perfect sense.

@Ximulate said in After failover, delay recovery:

I'm using the default IP.

Some may not agree, but try another IP. I have had luck with 8.8.8.8 (Google DNS). I don't think it will solve this issue since it sounds like you have actual latency impacting you.
Edit, On second thought, leave it alone. Particularly since you will be lowering the threshold, using 8.8.8.8 may have moments of latency and packet loss that will be normal and your lower values will make matters worse.

Well at the moment we don't know anything about your setup /LAN network. So it's almost impossible to tell you exactly whats wrong.
Generally speaking, running through the pfSense wizard will leave you with a working default configuration, providing Internet access from the LAN Interface.
https://docs.netgate.com/pfsense/en/latest/book/config/setup-wizard.html

-Rico

Ok I think I have resolved this, after going through debugs and more logs I saw the inactivity timer was being triggered on PFsense. This to me said it was not getting hello's from the multicast / neighbor.. therefore I created a firewall rule to allow any from the source of the neighbor to pfsense and the recurring adj messages stopped.

OK, I got this finally. I've spend ridicules amount of time on this one and it seems to be some kind of bug.

So I'm starting with a clean installation. At the beginning instance has only ena0 configured and ena1 is disabled. Changing ena1 to enabled makes Anti-Lockout Rule travel to ena1 (1st thing that shouldn't happen if you ask me) and I can not access 443 using EIP on ena1, just private ip. Then I'm adding allow all rule to both interfaces so all traffic should be accepted and I still can not access 443.
Screenshot 2020-07-22 at 01.13.37.png

nc -zv $EIP 443 does not work either from pfsense or remote computer. It works using private ip.

Only after removing Anti-Lockout Rule things are getting back to normal.

Hello!

Member Down should trigger on the upper threshold for latency or packet loss. Please review and/or post your gateway settings - gateway ip, monitoring/action settings, advanced settings, etc...

John

@wolfsden3 This works as expected
Go to openvpn client config and change the interface to the charter port.
Or even better, create a failover group and assign it there, with the needed priorities.

@ahmetakkaya said in Different Network Structure:

@chpalmer

Will it work if we connect two devices with port and write route ?

Yes. The routed package writes the route for you. But it can be all done manually..

well, I didn't find any issue from your work, and you have some basic routing and switch problems. I will try to make it simple...

nat isn't for routing, nat is to prevent private networks to expose on public internet which cannot route private address, and nat is used to allow private networks to access public internet.
port forwarding is used to provide private services to public networks.

vlan is used for separate networks, vlan can't communicate with other vlan directly, they communicate by routing.

so, in your work, nat is used on wan port which replaces source (private network address) to public address, then the public address know how to transmit back.
port forwarding is like someone access your public address port 80, which is HTTP, then your firewall forward the request to the server in your private network
in your private network, your firewall performs like a router, who knows every network, so if an address from LAN want to access an address in SERV, then it will forward the packet to the gateway of the LAN, which is your firewall LAN address, then the firewall know where next-hop is, it will forward the packet to SERV network.

Thank you!

It gets weirder, made the factory LAN 10.0.13.254/24 and made the 172.16.120.1 another interface (basically flipped the drop downs, and adjusted the IP's and DHCP, etc). Now when attempting to reach 10.0.6.159, it originates from the VPN /30 IP, so weird.

And actually- Ive been wanting to experiment a little with the cable modem and have now figured out that issue as well..

Thanks Raffi_ You got me working on this and Ive solved an issue now. ☺

thank you so much

@knobby unfortunately, I moved on to fixed wireless internet that gives me better internet speed and stability. Good luck with your findings.

Just to close the loop on this - I re-enabled pf with any/any inbound and outbound with NAT disabled and have not found any resulting issues.

From a performance perspective, I saw about a 50% performance hit in throughput. Luckily, I'm running this instance as a VM so by adding a second core to this instance, I'm back to near wire speed with pf running.

ESXi 7 on AMD Ryzen 5 3600 CPU if anyone is interested.

Thanks for the replies on this.

@Stewart said in Failing over too early:

I'd assume that setting would be for if we select the High Latency option.

There is no such option. It is part of the gateway monitoring (dpinger) mechanism for determining the gateway status.

I agree with @serbus . There is more than one factor which will mark a gateway as down. Packet loss is only one factor, the other is latency. If packets are taking too long to get a response, this can also cause your gateway to be marked as down.

To confirm what the actual problem is, go to
Status/System Logs/System/Gateways. What does the log there say when your gateway is marked down? You can post a screenshot and don't forget to blank out your WAN IP.

You appear to be trying to configure an unsupported role. The /64 for NPt must be routed to pfSense. If the upstream expects it to respond to NDP on the WAN segment, that cannot work. pfSense does not support the concept of proxying NDP requests.

If you have a handful of static addresses on the inside, you could setup IP alias VIPs on the WAN for those, but automatic assignment wouldn't be possible.