Using pfsense with multiple WANs

[lewis]

I forgot to add...
I can ping from pf01 to 10.100.0.2
I can ping from pf02 to 10.100.0.1

[stephenw10]

We have paid support if you need it. That pretty much rules out paying me here.

You're correct, you don't need a VIP on pf01 because it's DCLAN interface is already in the subnet directly.

If you've assigned the GRE interfaces at each end they should show as gateways and will be trying to ping each other. If you have added firewall rules on those pings should succeed and the gateways will show as on-line.

If that's the case you can try forwarding something from pf02 to the GRE tunnel IP at pf01 10.102.0.1. And from there to a server behind it.

Steve

[lewis]

If there's an hourly option available, I'd do that. I'm sure this could be done in 20 minutes so don't really need an ongoing plan. Besides, I send all kinds of business this way since I cannot speak highly enough about pfsense to all network folks I talk with.

In terms of what I have set up, it's as the images show. I posted another comment about being able to ping, apparently between tunnel interfaces now.

I didn't need a rule to allow the pings however so I assume it's automatically allowed because of the GRE at both ends.

Based on what you said, I needed a rule for pings to work so that must mean I've got something messed up.

[stephenw10]

You need to be able to ping 10.102.0.2 from pf01 and the other way, 10.102.0.1 from pf02. So pinging inside the tunnel. You need rules on the GRE interfaces to allow that.

Steve

[lewis]

Sounds like I have everything done right but must still be missing something small.

Cannot ping from one to the other.

[stephenw10]

How are you testing?

Check the routing table at each end (Diag > Routes). Make sure a route to the other side exists.

[lewis]

Hmm, I'm not sure where that /32 is coming from with pf01 10.102.0.2/32.

I see the mask weirdness but not sure how to handle that since I want pf01 to be reachable by more than one pfxx later.

[stephenw10]

The 10.100.0.2 VIP should be /24 on pf02. The subnet should be the same everywhere.
That would not stop it reaching pfo1 though since it's inside the /30

Do you see any packets on the GRE interfaces in Status > Interfaces at either end?

How are you testing link?

[lewis]

I changed that to /24.

I've been testing by pinging only.

pf02
GRE Interface (opt1, gre0)
Status     up 
IPv4 Address     10.102.0.2 
Subnet mask IPv4     255.255.255.252 
Gateway IPv4     10.102.0.1 
IPv6 Link Local     fe80::225:90ff:fe0e:a370%gre0 
MTU     1476 
In/out packets     0/91731 (0 B/6.42 MiB) 
In/out packets (pass)     0/91731 (0 B/6.42 MiB) 
In/out packets (block)     0/33 (0 B/3 KiB) 
In/out errors     0/0 
Collisions     0 

pf01
GRE Interface (opt3, gre0)
Status     up 
IPv4 Address     10.102.0.1 
Subnet mask IPv4     255.255.255.252 
Gateway IPv4     10.102.0.2 
IPv6 Link Local     fe80::ae16:2dff:feb8:8400%gre0 
MTU     1476 
In/out packets     0/77642 (0 B/6.22 MiB) 
In/out packets (pass)     0/77642 (0 B/6.22 MiB) 
In/out packets (block)     0/75 (0 B/6 KiB) 
In/out errors     0/0   
Collisions     0 

[stephenw10]

Ok so both sides are sending packets and not seeing anything incoming.

Do you see blocked traffic on either firewall?

GRE uses it's own protocol (gre) so if you have rules that allow only icmp/udp/tcp on the DCLAN connected interfaces that will block it.
If so adds rules to allow gre. Or just allow all IPv4 as a test.

Ste ve

[stephenw10]

Looking at the screenshots you posted above the pf02 LAN rules are only allowing traffic from LANnet. And the gre traffic will be coming from the VIP subnet 10.100.0.0/24 so you will need another rule there if you haven't added one since then.

[lewis]

I just don't know enough about this to make it work. I'm going to end up breaking pf02 and really messing things up.

[stephenw10]

Try filtering by gre. You should see the states with traffic both ways:

And if it's passing correctly you should see both those gateway monitoring pings replying:

It looks like there is no incoming state there so either the firewall rules are blocking it, check the firewall logs, or one side has no route for some reason.

Steve

[stephenw10]

I note that you have set 'add a static route' in the GRE setup at the pf01end. I have not done that in my test setup here.

[lewis]

My main fear at this point is messing up pf02 and causing down time since that is production.

I've removed the static. I think I enabled that because of something I read. And that's one problem with reading about something like this. People have different variations so when you've never done it and don't get the concept yet, it's easy to make a mess of tests.

I think this is where I'm at now and I've removed all rules, again, because I'm nervous about messing up prod traffic. I'll re-add slowly.

I added an ICMP rule allow all on both LAN and GRE on pf02 and cannot ping 10.102.0.2 from the pf01 CLI. However, I can ping 10.100.0.2 from pf01 and vise versa.

[stephenw10]

Do you see gre states open on both firewalls?

[lewis]

[stephenw10]

Not states on the GRE interfaces, gre states on the LAN and DCLAN interfaces.

Just filter by gre on all interfaces as I showed above.

[lewis]

All this time, I'm still pinging from one pf to the other.

[lewis]

You said most times that I seemed to have things set right other than the masks. I've updated those but still cannot ping between GRE sides. What in the world am I missing?