Finally, the problem was that this IP is from an NDS server (it is a DC) that is delivered by DHCP to Pfsense and it creates the route as local, although it is on the other side of the VPN tunnel.
We have configured the DNS of the manual Pfsense and we have not added that server and the problem has been fixed.

@jarlel Traffic is policy routed when it enters an interface.

Traffic sourced from localhost never enters an interface so it cannot be policy routed.

After I connected WAN back it stayed out of the group and marked as down. I released and renewed it's IP, that didn't work. I then just had to go to system, routing, and on the gateways page change nothing, just hit save and then apply. Then it came right back online into the gateway group, routing started working to it, and dynamic dns picked up and also updated the ip. This should have happened automatically and is I guess another issue but maybe related.

Doesn't seem I'm having a lot of success with automatic things with dual WAN's on a LAGG with an XG-7100

As long as nothing else uses that port, you can tie destion IP or Range/Netblock with that port.

Kind of problematic if they use say 443 or 80 or any other port that some other site/service will use. So you need to use enough variables to only route that specific traffic and not traffic you don't want to go out that gateway.

Why source IP is used - is it simple that if that changes its completely under your control. Problem with destination IP is most stuff is served via some sort of CDN these days, and IPs used could be in the 1000's or 10's of thousands - and they can change all the time.

Same with port, they are not always unique to whatever site/service you would want to route out a specific gateway.

While you know your source IP is what you set it to be, and won't be changing unless you change it.

But any combination you can come up with that makes the traffic unique enough to identify can be used.

Please delete this my issue is solved.

@evgeniysk said in server from LAN can't access themself via Virtual IP on WAN interface:

Ok, is it possible to change this behavior?

Yes, with NAT reflection. That means that a NAT rule on an specific interface (mostly WAN) is also implicitly applied on other interfaces. Not preferred, but there is no other option, it's a way to go.

You can activate it either in the respective NAT rule (at the bottom) or globally in System > Advanced > Firewall & NAT.
You can try the pure NAT mode, but if the server needs to access himself you possibly need the proxy mode.

@evgeniysk said in server from LAN can't access themself via Virtual IP on WAN interface:

Server pings itself by public IP, that configured on pfSense, so traffic must flow through it some way.

Without a NAT rule for ICMP + reflection, there is no possibility for the server to ping himself by using the public IP. You may be able ping the public IP though from the server, but this is owned by pfSense, so the firewall might response to such pings.
You may sniff the traffic on the internal pfSense interfaces to verify. If the server himself respond to the ping, you would see the packet twice, one time from server to pfSense and a second time back to the server.

Hi, I solved it like this :

create /usr/local/bin/reset_voip_states.sh

#!/bin/sh

#Kill Udp Sip States after new wan IP
echo "Killing States from ASTERISK pbx to SIPPROVIDER" |logger;

#kill freepbx connection
/sbin/pfctl -k ASTERISKIP

/sbin/pfctl -k ASTERISKIP -k SIPPROVIDER
/sbin/pfctl -k WAN1IP -k SIPPROVIDER
/sbin/pfctl -k WAN2IP -k SIPPROVIDER

chmod 755 /usr/local/bin/reset_voip_states.sh

Edit config file /conf/config.xml

<system>
...
<afterfilterchangeshellcmd>/usr/local/bin/reset_states.sh</afterfilterchangeshellcmd>
</system>

works like a charm
greetings

@rico said in Two providers, three links:

Are link 1 and 2 sharing the same ISP gateway IP ?
Check https://docs.netgate.com/pfsense/en/latest/multiwan/considerations.html

Excuse me, I missed your answer somehow!
Gateways are different, different subnets.
On the first link of provider # 1 there is a white IP address, on the second link of provider # 1 the IP address is gray, as a temporary one.

Thank you, I'll check!

@nunu There was an outbound NAT rule from LAN1 to a subnet not in LAN1 and it took a long time to realize. All in order for now. It helps writing it down somewhere. Cisco devices sometimes use loopback.

@heper

Thanks, I did notice that and set the WAN_DHCP to default yet as soon as I add the new GW it becomes the default. I assume I can set the weight of the new one lower and that might help. Need to wait till evening to try again.

@m0l50n
You need to add a route for the mobile VPNs IP pool to the remote site. So if it is pfSense you have to add it to the "Remote networks" in the OpenVPN settings.

It also is required that the remotes VPN endpoint is the default gateway in its network. Otherwise you need another solution.

@floydque

You could assign that URL to an Alias.
And then policy route packages that have that alias as destination , out of the desired gateway.

/Bingo

@shepherdkai said in Dual WAN at home? Anyone have stories on their experience?:

I have a Ubiquiti EdgeRouter Lite sitting in a box that I plan to break out for this use case.

Just curious... If you're going to use that box as your main router/firewall, why are you asking these questions on a pfsense forum?

Solved my own problem. Forgot to put additional static routes on my home router for the additional networks.... Silly me...