Failover Internet - Just for two clients on the network.

Derelict

With both WANs on Tier 1 you have create a Load Balanced gateway group, not a failover.

https://docs.netgate.com/pfsense/en/latest/book/multiwan/index.html

billsecond

@Derelict not really sure what you mean.

KOM

The guide says:

Gateways that are load balanced will automatically failover between each other. When a gateway fails it is removed from the group, so in this case if any one of A, B, or C went down, the firewall would load balance between the remaining online gateways.

which would lead me to believe that it should failover to the other member and remove the bad member.

Derelict

https://docs.netgate.com/pfsense/en/latest/book/multiwan/index.html

Derelict

@KOM said in Failover Internet - Just for two clients on the network.:

The guide says:

Gateways that are load balanced will automatically failover between each other. When a gateway fails it is removed from the group, so in this case if any one of A, B, or C went down, the firewall would load balance between the remaining online gateways.

which would lead me to believe that it should failover to the other member and remove the bad member.

That would assume that the gateway monitors would accurately represent what can pass traffic.

When OP posts he wants failover and presents load balance, I have to question the solution.

KOM

@billsecond Anything in the System log or Gateway log that confirms the bad gateway was detected as down? If you really want it to primarily use the Xfinity link and failover to the 4G, then set the 4G tier to 2.

billsecond

@KOM @Derelict, Thanks for all of your help, I took a combination of what you said and applied! For future reference, here is what I ended up doing...

Interface > Assignments > Added my new 4G modem, kept everything as the defaults:
System > Routing > Gateway Groups > Created a new Gateway Group, Set a Tier 1, and Tier 2 (this is priority based on what I was reading, let me know if it is incorrect, it is working)
Firewall > Rules > Lan > Add (top or bottom, we have to adjust it later)
- Interface: Lan
- Protocol: Any
- Source: Single host or alias: Selected a PC's IP in my network
- Direction: Any
- Advanced Options
  - Gateway: Added my new Failover Gateway Group
Ordered the priority of the Failover PC's above the other traffic, but set the other traffic to go over the normal fast default gateway

Please validate that I did this right and didn't screw anything up. It does work!

KOM

Looks good to me. One thing to note IIRC is that if a failover happens and the tier 1 member comes back online, it won't switch back automatically.

billsecond

@KOM How do I switch it back automatically?

KOM

I'm not aware of any way to do it in the GUI. I've read about some scripts that you can configure to run on a gateway up event but I've never tried them. I think I have a copy of them floating around somewhere, maybe at work. Google it and maybe you will find something. Or I could be entirely wrong and giving you old information. Test it and see what happens.

Derelict

It WILL switch back automatically but it will not kill states that were established over the Tier 2 WAN so that traffic will continue to flow there.

New states will use the Tier 1 WAN.

KOM

Thanks for the clarification.

JeGr

@Derelict said in Failover Internet - Just for two clients on the network.:

It WILL switch back automatically but it will not kill states that were established over the Tier 2 WAN so that traffic will continue to flow there.

Other than manually resetting states - how can one do so in a less intrusive way? Especially VoIP is bound to stay glued to the failover connection as the state should never expire? ;)

KOM

There are homebrew solutions floating around:

https://github.com/mk-fg/pfsense-scripts

No idea if they work, no idea if they will give you Space AIDS, YMMV.

billsecond

I think it is fine to keep the open streams to continue to work over the failover gateway. It is my experience (based on this test) that everything will move back when the gateway comes back up on line. It is a very good approach.

JeGr

I'm thinking more along the lines of an advanced setting like the State killing on GW switch toggle. As some use WAN Failover links that are metered, staying online on the backup too long will have unnecessary costs that could otherwise be avoided.

billsecond

@JeGr said in Failover Internet - Just for two clients on the network.:

'm thinking more along the lines of an advanced setting like the State killing on GW switch toggle. As some use WAN Failover links that are metered, staying online on the backup too long will have unnecessary costs that could otherwise be avoided.

Good point! For my needs that does not apply, but I see where in the case of a metered connection, Yes, it would apply! However, I would be worries that it would kill the backup, and you'd have some unknown (at first) reprocution.

Derelict

As I understand it there is no good way to say "Hey, pf, kill all of the states on this interface." And there is also no good way to match those outside states up with the corresponding inside states.

So you are left with killing all states.

The Reset All States setting in System > Advanced, Networking was intended to provide some relief here.

JeGr

@Derelict I was afraid of that ;) But thanks!

stephenw10

As I understand it that used to be the case, no way to kill the states that selectively, and that's why it behaves as it does. However I believe current versions of pf can in fact do that so it may be possible to do exactly that now.

I have this issue myself. My backup WAN is data limited so I can't have connections using it continually after my main WAN glitches. Most traffic will fail back naturally as TCP connections close it's only really persistent UDP traffic that gets 'stuck', so VoIP and VPNs. My own WAN is good enough that I just don't failover that traffic but I understand that for many those are the most important connections.
That said if it did just kill everything on the backup WAN when the main WAN comes bacl up it would needlessly interrupt that traffic. A cron-job that kills traffic on the backup WAN that runs at some off-peak time might be better.

Steve