Thank you for the response.
I took your advise and upgraded to one router with multiple NICs.

@benzepoj
pfSense Version
2.5.2-RELEASE (amd64)
built on Fri Jul 02 15:33:00 EDT 2021
FreeBSD 12.2-STABLE

@gertjan I think I've discovered the answer for my problem...

I came across a note about comcast blocking traffic which pings and the answer was to turn off monitoring on the gateway. Voila. That worked.

I tested it by turning monitoring back on and it still worked. That seems a little flaky to me and I suspect it will come back to haunt me... but for now i'm clad it's framed up pretty well.

I did create the 4084 vlan and assigned it to a default Interface OPT1. Then in the Interface settings I renamed the OPT1 to WAN2.

I also created a gateway for WAN2 manually and had to set a NAT in outbound for WAN2.

Thank you for your help.

In anyone is still interested, here is how I got it to work with 3 pfsense setup.

I wanted to setup an environment where I have a datacenter and a remote lab.
All machines in the datacenter have the domain datacenter.home.arpa.
All machines in the lab have the domain lab1.home.arpa.
I wanted machines in the lab to be able to reach machines in the datacenter.

pfSense1:

Hostname: pfSense Domain: home.arpa WAN (dhcp) LAN: 192.168.0.1 Block private networks and loopback addresses: Unchecked Forward packets for datacenter subnet 192.168.2.0/24 to datacenter router - 192.168.0.2 Added gateway Name: datacentergw Interface: LAN Gateway: 192.168.0.2 Added static route Network: 192.168.2.0/24 gateway: datacentergw

pfSense2:

Hostname: pfSense Domain: datacenter.home.arpa WAN: 192.168.0.2 (static) LAN: 192.168.2.1 Block private networks and loopback addresses: Unchecked NAT Forward ICMP and TCP/UDP from source:192.168.0.0/16, destination: LAN net to LAN Address This automatically added necessary firewall rules as well

pfSense3:

Hostname: pfSense Domain: lab1.home.arpa WAN: 192.168.0.3 LAN: 192.168.3.1 Block private networks and loopback addresses: Unchecked DNS Add a domain override for datacenter.home.arpa and send its queries to datacenter DNS: 192.168.2.1 DHCP Set lab1.home.arpa;datacenter.home.arpa as DNS Search

@stewart Just wanted to report that this was the solution. Thanks @johnpoz!

@dataideas-josh Yeah I need to get back to testing this soon.

@operations no one with an idea?

pfSense from what I've seen won't work if the gateway is the same on both WAN interfaces.
Are you doing this in a VM environment or BareMetal?

thanks so much for the help @viragomann and @johnpoz , I seem to have a working route out now with FW rules using policy route!

@jaspery Based on my 2nd episode with crash, I suspect it was crash that caused my dpinger to fail (in this case).

@ashtonianagain Can't speak to Wireguard but we've used it for our office (behind our building router) for many years and have had port forwards set up at several clients that put the router in a DMZ.

There is a guide at https://docs.netgate.com/pfsense/en/latest/troubleshooting/nat-port-forwards.html but if it connects initially it would seem the forwarding is correct. Unless maybe it's trying to use additional ports?

There are examples for Wireguard setup.

@latimeria

I think you will get the most out of this video on YouTube.

How to use Multiple WAN on pfsense for Fail over and or Load Balancing

@chrisjx

Maybe I'm over thinking it and it's just a different way to do what DDNS does but for a non-ip CGNAT service.

You need a so called jump host in the internet, free to reach from else where, that is connected to you home network.

Thats it, at a "Hoster" of your choice for some coin
per month and all is done.

@steveits Yes, it was surprisingly easy to set up the 1:1 NAT logic. For the Medusa, its used for someone who rents single office tenant spaces to their own clients so lots of small VLAN's with one or two clients requesting public IP's directly.

@cool_corona
IPsec Site-to-Site VPN Example with Pre-Shared Keys

If you want to allow access to a small segment of the LAN subnet you can state this in the phase 2 at "Local Network", type "Network".
Additionally you need a firewall rule on the IPSec tab to allow access. Here you can also state an alias with single IPs and ports as destination to lock permission down to the necessary destinations only.

Guys, i am still working on this trying to configure it. I think i am doing some kind of progress. Please bear with me as today i don't have that much time. I'll come back tomorrow.

Thank you for all your advises!

@computingdon You'll need to post details. The source address of the connection, the route back to it, the firewall rules passing that traffic when it enters pfSense, and the outbound NAT rules.

@lnguyen said in Can't Route Site To Site:

@dma_pf What are the allowed networks under "Peers" for both sites?

Thanks for pointing me in this direction...that was it! There was an error in one of the peer IP addresses:

00155179-c7b4-4b05-be81-a0a7f79d6e1c-image.png

The Site 2 network should have been 192.168.164.0.

I made the error of seeing that the Wireguard handshake was completed and made the assumption that by doing so it was confirming that: 1) the cryptographic keys matched and 2) that the peer trying to connect had come from the Allowed IP networks. As a result I never rechecked the peer Allowed IPs because I saw a successful handshake.

But now I've got to dig deeper into the Wireguard protocol as it appears that the handshake only requires the keys to match and the Allowed IPs are only used as a routing ACL to allow or reject traffic across the tunnel.

Thanks again for your help!

@jazzl0ver ahhh ok not available in the kernel. That makes sense.

@viragomann

Ok yeah, that makes sense, now that you mention it, I've seen that before. Just not something I typically pay attention to. Guess that leaves me pretty well stumped here.

@rustydusty1717 Accurate, comprehensive, numbered diagrams always help.

@viragomann
One more question please. VLANs. What is the general concept if I want VLANs to work through PFSENSE? In this case, PFSENSE, as the core of the network, has OVPN and IPSEC clients. Should I want VLAN111 on the OVPN1 client to see VLAN111 on the OVPN2 client, or even more VLAN111 to see on the IPSEC1 client?

Got everything up and working now with the LAN block as virtual IP's.

FYI: For anyone changing the WAN adapter assignment, I found that I needed to go back through the CLI instead of the web browser to reassign all adapters before it would start routing traffic. Initially, I made the change on the adapter in Esxi, but nothing connected to the internet. Then, I created a completely new adapter and assigned it as the WAN interface in the web browser, but still, nothing happened. Eventually, I went through the CLI assignment for just the WAN and LAN, and then traffic started routing again. After that, I was able to reassign and reset the interfaces with the web browser.

@darkcorner said in WAN 1 and 2 Up, but no internet access:

Or leave the two Google DNS without associating them to the Gateways?

Exactly this one.

@rcoleman-netgate said in "Disable gateway monitoring action" NOT working:

You have to monitor an IP address with dual-WAN to make sure the interface is up

I do have IP addresses setted as you can check in the first 3 images.

@rcoleman-netgate said in "Disable gateway monitoring action" NOT working:

Taking it offline from monitoring will treat it as though it is always up and the "member down" setting is made redundant.

That is the configuration I am using now but only because the "disable gateway monitoring action" is not working.

I read all the documentation about multiple wans. But still, the "disable gateway monitoring action" checkmark should be self evident. If you mark that option it should never take any action when monitoring and should never set the interface down. But it does...

@johnpoz

Thanks I will look into the FFR package.

As to them using RIP like I said they are a crappy ISP... They used to use all just regular SurfBoard docsis 3.0 modems and everything was great. But they got sold on this shitty taiwanese Askey hardware and management software (I think because they took over time warner and roadrunner, this is the junk they were using). Well I think I might have gave it away to you which ISP this is, hopefully I don't piss off the ISP overlords and they zap my hardware for being an ungrateful customer LMAO...

@pduk82 well its not the end of the world - double nat is not all that bad, while there are some special apps that might have problems. Generally speaking most users can be behind a double nat, even triple or have seen quadruple and never notice any issues.. More nats in your chain before public IP can be problematic for allowing inbound traffic via port forwards, but still able to do - just have to port forward on the device(s) upstream of your pfsense

@dtnb Simply means the path to your monitor IP or maybe the monitor IP itself is not healthy.

edit: I see the monitor IP is 192.168.1.254 which is your ATT modem in this case. Perhaps a cabling problem? Have you tried swapping the patch cable?

@jegr said in small gateway bug after 23.01 upgrade?:

@jimp Applied the patch yesterday and had to change back all and every changed GW that was saved with lowercase back to uppercase. As manually created GWs can be lowercase, too and Interface names can have that, why not simply make it case insensitive to both work? That way one wouldn't have to switch back and forth between upper or lowercase? Made the whole setup break again :/

That is a much more significant (and likely disruptive!) change than putting the automatic names back the way they should have been.

Manual entries would not have changed and wouldn't have had any issues (upper or lower), only fully dynamic entries were a problem.