Using pfsense with multiple WANs
-
You have no idea how frustrated I am having to keep asking questions and how much I appreciate that you've helped me all this time. I really should have opened a new question for the GRE.
Something about this is either confusing or I've already got it right.
First, we want to create a tunnel between two firewalls using GRE. The GRE tunnel needs to communicate between the two firewalls so IPs pf01 10.100.0.1/24 (24 since I'll have other pf on the same tunnel/network in different subnets) and pf02 10.0.0.1/24.
The tunnel then has its own internal IPs so pf01 10.102.0.1/30 and pf02 10.102.0.2/30 to create one net.
Adding additional pfxx will be simpler since there are no hosts behind that network, we're just using pf as a means to get IP traffic from one network firewall to another.
So if I add pf03, then pf03 can have a LAN 10.3.0.1/24 connecting to DCLAN 10.100.0.1/24. Then a GRE tunnel, pf03 10.103.0.2/30 and pf02 will have 10.103.0.1/30.
IF I have this, then the following which is how I have it all set up should be ok.
IF this is ok, then the only thing I'm missing is one or more rules.
IF this is correct, I just need to start pointing hosts behind pf02 from 10.0.0.x/24 to 10.102.0.1/30.This is a bit simple and mind boggling at the same time because it's all text. Seeing one done would really simplify everything for me and anyone else wanting to understand this. I suspect I'm still screwing something up because from pf02 cli, I cannot ping 10.102.0.1 and from pf01 cli, I cannot ping 10.102.0.2 but it could be because I don't have any rule/s yet. Truly not sure what kind of rule I'll need anyhow.
-
The parent interface for the GRE tunnel on pf02 must be the VIP
10.100.0.2. The LAN interface there as you have it set there is not in the same subnet.On pf03 the LAN (is that's what's on the DCLAN VLAN) should be 10.100.0.3. Or you can add that as a VIP on top of some other subnet like pf02.
The GRE tunnel there will then be between pf03 and pf01.Otherwise that looks right. Can you ping across the tunnels?
Steve
-
Sorry, I only brought up pf3 in terms of your comment about trying to use a numbering scheme. Right now, there are only pf01 and pf02 at the moment to keep things simpler.
On pf02, I have a VIP of 10.100.0.2/30 on the LAN.
Do I need a VIP on pf01 also? I thought the DCLAN. I only have the DCLAN static set which is 10.100.0.1/24.I'm probably not getting this yet but it seems that the DCLAN interface just gets the one static IP.
However, GRE tunnels get their own end to end IPs and then another set for communications inside the tunnel.
Could I just hire you for an hour to get this done over a phone call?
It would save us 100 messages :).I was hoping it could be something useful for anyone else that is wanting to understand how to do this. There's a lot of information out there but it's pretty vague when you've never done it before.
-
I forgot to add...
I can ping from pf01 to 10.100.0.2
I can ping from pf02 to 10.100.0.1 -
We have paid support if you need it. That pretty much rules out paying me here.
You're correct, you don't need a VIP on pf01 because it's DCLAN interface is already in the subnet directly.
If you've assigned the GRE interfaces at each end they should show as gateways and will be trying to ping each other. If you have added firewall rules on those pings should succeed and the gateways will show as on-line.
If that's the case you can try forwarding something from pf02 to the GRE tunnel IP at pf01
10.102.0.1. And from there to a server behind it.Steve
-
If there's an hourly option available, I'd do that. I'm sure this could be done in 20 minutes so don't really need an ongoing plan. Besides, I send all kinds of business this way since I cannot speak highly enough about pfsense to all network folks I talk with.
In terms of what I have set up, it's as the images show. I posted another comment about being able to ping, apparently between tunnel interfaces now.
I didn't need a rule to allow the pings however so I assume it's automatically allowed because of the GRE at both ends.
Based on what you said, I needed a rule for pings to work so that must mean I've got something messed up.
-
You need to be able to ping 10.102.0.2 from pf01 and the other way, 10.102.0.1 from pf02. So pinging inside the tunnel. You need rules on the GRE interfaces to allow that.
Steve
-
Sounds like I have everything done right but must still be missing something small.
Cannot ping from one to the other.
-
How are you testing?
Check the routing table at each end (Diag > Routes). Make sure a route to the other side exists.
-
Hmm, I'm not sure where that /32 is coming from with pf01 10.102.0.2/32.
I see the mask weirdness but not sure how to handle that since I want pf01 to be reachable by more than one pfxx later.
-
The 10.100.0.2 VIP should be /24 on pf02. The subnet should be the same everywhere.
That would not stop it reaching pfo1 though since it's inside the /30Do you see any packets on the GRE interfaces in Status > Interfaces at either end?
How are you testing link?
-
I changed that to /24.
I've been testing by pinging only.
pf02 GRE Interface (opt1, gre0) Status up IPv4 Address 10.102.0.2 Subnet mask IPv4 255.255.255.252 Gateway IPv4 10.102.0.1 IPv6 Link Local fe80::225:90ff:fe0e:a370%gre0 MTU 1476 In/out packets 0/91731 (0 B/6.42 MiB) In/out packets (pass) 0/91731 (0 B/6.42 MiB) In/out packets (block) 0/33 (0 B/3 KiB) In/out errors 0/0 Collisions 0 pf01 GRE Interface (opt3, gre0) Status up IPv4 Address 10.102.0.1 Subnet mask IPv4 255.255.255.252 Gateway IPv4 10.102.0.2 IPv6 Link Local fe80::ae16:2dff:feb8:8400%gre0 MTU 1476 In/out packets 0/77642 (0 B/6.22 MiB) In/out packets (pass) 0/77642 (0 B/6.22 MiB) In/out packets (block) 0/75 (0 B/6 KiB) In/out errors 0/0 Collisions 0 -
Ok so both sides are sending packets and not seeing anything incoming.
Do you see blocked traffic on either firewall?
GRE uses it's own protocol (gre) so if you have rules that allow only icmp/udp/tcp on the DCLAN connected interfaces that will block it.
If so adds rules to allow gre. Or just allow all IPv4 as a test.Ste ve
-
Looking at the screenshots you posted above the pf02 LAN rules are only allowing traffic from LANnet. And the gre traffic will be coming from the VIP subnet 10.100.0.0/24 so you will need another rule there if you haven't added one since then.
-
I just don't know enough about this to make it work. I'm going to end up breaking pf02 and really messing things up.
-
Try filtering by
gre. You should see the states with traffic both ways:
And if it's passing correctly you should see both those gateway monitoring pings replying:
It looks like there is no incoming state there so either the firewall rules are blocking it, check the firewall logs, or one side has no route for some reason.
Steve
-
I note that you have set 'add a static route' in the GRE setup at the pf01end. I have not done that in my test setup here.
-
My main fear at this point is messing up pf02 and causing down time since that is production.
I've removed the static. I think I enabled that because of something I read. And that's one problem with reading about something like this. People have different variations so when you've never done it and don't get the concept yet, it's easy to make a mess of tests.
I think this is where I'm at now and I've removed all rules, again, because I'm nervous about messing up prod traffic. I'll re-add slowly.
I added an ICMP rule allow all on both LAN and GRE on pf02 and cannot ping 10.102.0.2 from the pf01 CLI. However, I can ping 10.100.0.2 from pf01 and vise versa.
-
Do you see gre states open on both firewalls?
-
