Just remember doing this can be a bit harder on the resources of the box. Especially if you have allot of LAN to LAN traffic..

Add another MX record to point to your isp b WAN address. That way mail should be able to reach your mail server no matter which gateway is up.

It didn't help. Any traffic initiated from external sources via that tunnel ends up being responded via WAN interface. Any ideas?

You can't hairpin NAT like that with FreeBSD/pf. They won't be able to reach each other through the WAN address. With manual port forward rules, you can enable NAT reflection which adds more rules behind the scenes to cover that scenario. That is not possible with UPnP.

@KOM okay when I get the opportunity I will post that, I think I'm going to leave this pfsense VM a live just because I want to track down the real issue. Yeah I've seen a lot more users having issues with hyperv I agree.

I still don't know why the above firewall rules didn't work, but what did work was: flipping the rule order so that devices which are only allowed through the Tier 1 WAN gateway is the last rule enable Sys > Adv > Misc "Flush all states when a gateway goes down" was also needed for certain devices (i.e. VoIP phones) allowed through either gateway

@Derelict thanks for your answer. I'm not sure I understood the problem. After reading the article I think the problem is because I created groups for load balacing + failover while load balancing manage the failover itself: Gateways that are load balanced will automatically failover between each other. So I have to get rid of my groups + rules about failover. And for default gateway IPv4? I put automatic?

Well, i figured it out. I was doing the logical thing by adding the remote network to each side (Site B) and to the OpenVPN service (hosted on Site A). And that wasn't working. So I started messing around with the openvpn firewall. Turns out that you need an additional explicit route on the mobile client server config. Source: openvpn mobile client subnet (192.168.1.0/24 in this example) Destination: any Now the traffic routes. I'm sure that is documented somewhere but i couldn't come up with the right search phrase. I only figured it out with lucky guesses. Now those lucky bastards on OpenVPN Client can see the network resources on Site B. (And much more since this is a mesh setup.)

Based on my understanding (which might be wrong), set your Gateway Group as the default Gateway then you shouldn't need to modify the firewall rules. In my case, I need to modify the firewall rules to block all but high priority devices from using the backup (Tier 2) WAN. But, if thats not a concern of yours then I think you should be fine.

Found the problem. I swapped out SIM cards and everything worked fine. Therefore, I tracked it down to having an issue with the cellular provider.

@romal-amarkhail said in Can't ping/access internal network from LAN or WAN interface.: Any idea what might be wrong here? Do you know for a fact that the unit responds to pings? And if your LAN is 192.168.2.0, why is the wifi on 192.168.1.0? Are you using a /16 mask? Is your wifi in AP mode?

It's more interesting, i switch Trigger Level to High Latency, and some time later pfSense himself switch it to Packet Loss! I didn't understand, why it happens. And for sure it's dpinger bugs related, case i have checked "Disable Gateway Monitoring Action" and then made reconnect on router 2 (3 gateway) and in "Gateway status" i get on 3 gateway "Danger, Packetloss: 100%" on 3 gateway, i have check - traffic still goes through 3 gateway (router 2) without any problems, but dpinger thinks that it's dead for sure forever, till i make "save and apply" in any gateway settings. I didn't now how to make monitoring work in pfSense. :(

I guess this issue is then solved. Thanks for the help.

VLANs, VPNs, anything virtual.

GRE Tunnel Bonding Protocol [https://tools.ietf.org/html/rfc8157](link url) - "Single flow may use the combined bandwidth of the two connections. Can this be implemented in pfSense? It seems Layer2 bonding is one solution. " since load balancing in bonding takes places in Ethernet frames, even a single TCP/IP connection will enjoy an increased band thanks to the presence of multiple links." [https://zeroshell.org/load-balancing-failover/#vpn-bonding](link url) Can this be implemented in pfSense?

I believe I am now sorted. I have left don't pull roots unchecked and in firewall/rules/secondlan advanced options/gateway I chose WAN_PPPoE. I now have internet connection via VPN on igb1 and igb2 and connection not through VPN on igb4 just as I wanted. Many thanks for the help