Add each WAN IPs you want to use to the WAN interface. Firewall >Virtual IPs. Use type "IP Alias".

Go to Firewall >NAT >Outbound. Switch into the manual mode. pfSense should take over the automically generated rules for each of your subnets into the manual mode. Edit each one, go down to the translation address and select the outbound IP from the drop-town you want to assign the respective source network.

Create the vlans on pfsense, with the IDs of your vlans and then assign them to the physical interface that is connected to switch.

nobody with ideas?

Thanks. I'll see what I can do.

This was not the fix. needed to adjust a routing statement in the VPN router to include the full / 24 I had mistakenly set it to /28

Okay, my bad. This seems to be an issue with my APs versus pfSense. When I run test-ipv6.com on a wired client, it passes. I'd delete this post, but it errors out. My apologies for the diversion.

Yeah you normally do not have access to manipulate routing inside the mpls network. But you could ask.

Proxy on your end, which you just run on pfsense would be easier way to go for sure ;)

Other solution would be to create a tunnel between their end your end where you could route internet through the tunnel. This removes any routing concerns inside the mpls path.. You could do openvpn from the branch pfsense to yours.

If it makes multiple outbound connections and the protocol doesn't like it coming from two different addresses you will have problems.

If it only makes one connection it should be fine.

Try it and see?

If it gives you issues you can policy route just that traffic out one WAN. You might also try sticky connections.

https://docs.netgate.com/pfsense/en/latest/book/multiwan/load-balancing-and-failover.html#problems-with-load-balancing

You have to use the tag because, as that blog describes, traffic heading out WAN has already had outbound NAT applied by the time the outbound floating rule is checked so you lose the ability to match on the hosts' inside IP addresses.

@Milan-M said in LAN Routes just disappear:

LANs 3-5 have been created by going to "Firewall -> Rules -> LAN" and creating the rules there.

That is not how you create anything.. Creating other lan would be done via interface assignments, be it a physical interface or a vlan you assign.

If you have other networks that are downstream that you want to get to via some other downstream router, then you wuld need to create a gateway in routing, and then the route(s) telling the networks at are available via that gateway.

Yes you would need to create rules to allow them access.. But that is not what "creates" them or routes to them.

Btw your rule there for "lan" isn't going to do anything - the source is set for the lan address, not the network.. So that says hey pfsense if you see traffic from your own lan address allow it ) Never going to work that way..

@dukynuky said in Problem changing gateway through rules:

pass in log quick on $OpenVPN reply-to ( ovpnc1 10.10.10.1 ) inet from any to any tracker 1560717223 keep state label "USER_RULE"
now it is working.. seems pfsense isnt creating the reply to rules.. :(

That will not survive filter rule rewrites.

The traffic coming into the lower pfsense MUST NOT MATCH rules on the OpenVPN tab or the state will not get reply-to.

The traffic must match the rules on the Assigned interface tab. If it matches a rule on the OpenVPN tab, those are processed first so the assigned interface rules will never be reached, and therefore no reply-to.

All of this works. It just has to be configured correctly.

I would just remove all rules from the OpenVPN tab and put the necessary rules on the appropriate assigned interface tab and never worry about it again. Some topologies support this method, some don't.

@Derelict said in Multiple public IP multiple routers...:

If all of this is jibber-jabber to you

My money is on this statement ;)

I still got this issue, now I can replicate it easily at 2 completely sites, all 2.4.4_p3 and both using;

FRR and OSPF

list itemHA pair

list itemIPSEC VTI tunnels bound to a CARP IP address

list itemFRR set to fllow the lan CARP address (so FRR off on the backup firewall)

Here's a continuous ping across the VPN from site A to site B.

Reply from 10.10.40.1: bytes=32 time=4ms TTL=253
Reply from 10.10.40.1: bytes=32 time=7ms TTL=253
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Reply from 10.10.40.1: bytes=32 time=4ms TTL=253
Reply from 10.10.40.1: bytes=32 time=3ms TTL=253
Reply from 10.10.40.1: bytes=32 time=4ms TTL=253
Reply from 10.10.40.1: bytes=32 time=3ms TTL=253

First timeed out, that's the primary firewall being rebooted, 4 pings lost and the backup completely takes over. Very acceptable. Excellent.

Now the slow bit... The primary comes up, CARP takes over and takes ages for things to settle and go online.

Reply from 10.10.40.1: bytes=32 time=3ms TTL=253
Reply from 10.10.40.1: bytes=32 time=17ms TTL=253
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Reply from 10.10.40.1: bytes=32 time=3ms TTL=253
Reply from 10.10.40.1: bytes=32 time=4ms TTL=253

After digging, I think the cause is the VPN, IPSEC, it's just not getting released from the backup firewall in a timely manner, it seems to hold on and on and on and keeps running IPSEC VPN tunnels. I can speed up the fail back by logging onto the backup firewall and in IPSEC status stopping the IPSEC tunnels.

I wonder if the issue is because my IPSEC tunnels are using a CARP IP address?

Ask the provider if they have an alternate subnet they can assign to the other subscription.