Using pfsense with multiple WANs

stephenw10 · Mar 17, 2022, 9:43 PM

How are you testing?

Check the routing table at each end (Diag > Routes). Make sure a route to the other side exists.

lewis · Mar 17, 2022, 10:20 PM

Hmm, I'm not sure where that /32 is coming from with pf01 10.102.0.2/32.

I see the mask weirdness but not sure how to handle that since I want pf01 to be reachable by more than one pfxx later.

stephenw10 · Mar 17, 2022, 10:20 PM

The 10.100.0.2 VIP should be /24 on pf02. The subnet should be the same everywhere.
That would not stop it reaching pfo1 though since it's inside the /30

Do you see any packets on the GRE interfaces in Status > Interfaces at either end?

How are you testing link?

lewis · Mar 17, 2022, 10:39 PM

I changed that to /24.

I've been testing by pinging only.

pf02
GRE Interface (opt1, gre0)
Status     up 
IPv4 Address     10.102.0.2 
Subnet mask IPv4     255.255.255.252 
Gateway IPv4     10.102.0.1 
IPv6 Link Local     fe80::225:90ff:fe0e:a370%gre0 
MTU     1476 
In/out packets     0/91731 (0 B/6.42 MiB) 
In/out packets (pass)     0/91731 (0 B/6.42 MiB) 
In/out packets (block)     0/33 (0 B/3 KiB) 
In/out errors     0/0 
Collisions     0 

pf01
GRE Interface (opt3, gre0)
Status     up 
IPv4 Address     10.102.0.1 
Subnet mask IPv4     255.255.255.252 
Gateway IPv4     10.102.0.2 
IPv6 Link Local     fe80::ae16:2dff:feb8:8400%gre0 
MTU     1476 
In/out packets     0/77642 (0 B/6.22 MiB) 
In/out packets (pass)     0/77642 (0 B/6.22 MiB) 
In/out packets (block)     0/75 (0 B/6 KiB) 
In/out errors     0/0   
Collisions     0

stephenw10 · Mar 17, 2022, 10:54 PM

Ok so both sides are sending packets and not seeing anything incoming.

Do you see blocked traffic on either firewall?

GRE uses it's own protocol (gre) so if you have rules that allow only icmp/udp/tcp on the DCLAN connected interfaces that will block it.
If so adds rules to allow gre. Or just allow all IPv4 as a test.

Ste ve

stephenw10 · Mar 17, 2022, 10:57 PM

Looking at the screenshots you posted above the pf02 LAN rules are only allowing traffic from LANnet. And the gre traffic will be coming from the VIP subnet 10.100.0.0/24 so you will need another rule there if you haven't added one since then.

lewis · Mar 17, 2022, 11:37 PM

I just don't know enough about this to make it work. I'm going to end up breaking pf02 and really messing things up.

stephenw10 · Mar 17, 2022, 11:58 PM

Try filtering by gre. You should see the states with traffic both ways:

And if it's passing correctly you should see both those gateway monitoring pings replying:

It looks like there is no incoming state there so either the firewall rules are blocking it, check the firewall logs, or one side has no route for some reason.

Steve

stephenw10 · Mar 18, 2022, 12:04 AM

I note that you have set 'add a static route' in the GRE setup at the pf01end. I have not done that in my test setup here.

lewis · Mar 18, 2022, 4:49 PM

My main fear at this point is messing up pf02 and causing down time since that is production.

I've removed the static. I think I enabled that because of something I read. And that's one problem with reading about something like this. People have different variations so when you've never done it and don't get the concept yet, it's easy to make a mess of tests.

I think this is where I'm at now and I've removed all rules, again, because I'm nervous about messing up prod traffic. I'll re-add slowly.

I added an ICMP rule allow all on both LAN and GRE on pf02 and cannot ping 10.102.0.2 from the pf01 CLI. However, I can ping 10.100.0.2 from pf01 and vise versa.

stephenw10 · Mar 18, 2022, 7:51 PM

Do you see gre states open on both firewalls?

lewis · Mar 18, 2022, 8:18 PM

stephenw10 · Mar 18, 2022, 8:20 PM

Not states on the GRE interfaces, gre states on the LAN and DCLAN interfaces.

Just filter by gre on all interfaces as I showed above.

lewis · Mar 18, 2022, 8:34 PM

All this time, I'm still pinging from one pf to the other.

lewis · Mar 18, 2022, 9:06 PM

You said most times that I seemed to have things set right other than the masks. I've updated those but still cannot ping between GRE sides. What in the world am I missing?

stephenw10 · Mar 19, 2022, 12:26 AM

Ok, so the problem becomes apparent. On one side (looks like pf02) the GRE tunnel is incorrectly using 10.0.0.1 when it should be using the VIP address, 10.100.0.2.
So probably it is set to be on the LAN there and should be on the VIP.

Steve

lewis · Mar 19, 2022, 12:31 AM

One guy tells me, why bother with a GRE tunnel? Just use DNAT.
I gave him an example of wanting to reach a vm at 10.0.0.120 port 443 behind pf02 from the LAN on pf01.

He says, create a dnat on pf02 10.100.0.120:443 to 10.0.0.120:443

Not really sure what he's talking about but he's got a point. I can ping 10.100.0.2 on pf02 so doesn't that mean there's already a way to get this to work? I just don't understand how to map/route private IPs to private IPs.

Maybe I just need to create a new network, 10.100.0.1/24 and map pf02 IPs to that?

Mind blown here. All week on this.

lewis · Mar 19, 2022, 12:35 AM

This post is deleted!

lewis · Mar 19, 2022, 12:37 AM

@stephenw10 said in Multi LAN networks to one pfsense:

Ok, so the problem becomes apparent. On one side (looks like pf02) the GRE tunnel is incorrectly using 10.0.0.1 when it should be using the VIP address, 10.100.0.2.
So probably it is set to be on the LAN there and should be on the VIP.

Steve

I see that in the image I shares but not in the config anywhere.

stephenw10 · Mar 19, 2022, 1:04 AM

@lewis said in Multi LAN networks to one pfsense:

He says, create a dnat on pf02 10.100.0.120:443 to 10.0.0.120:443

10.100.0.120 doesn't exist but assuming you create it as a VIP on pf01 you can do that. But it will only work for one pfSense instance forwarding traffic because that is the gateway on pf01 DCLAN.

If you try to something similar from pf03 without a GRE tunnel replies will go back via pf02 resulting in an asymmetric route and blocked traffic.

Steve