Backhaul

afreaken

New to pfSense, a little unsure of how to configure it or if it can do what I want it to.

Here is the setup:

Why the CISCO Firewall, because that's how the network is set up currently for VPN access and VPN tunnel between LANs, and I doubt my supervisor would want to throw them out.
Also, is it possible to pass through an internet connection over a WiFi backhaul in this setup as a failover?

Networking isn't my specialty, trying to learn.

Derelict

@afreaken:

Why the CISCO Firewall?

I guess another question would be why the pfSense?

kejianshi

Bang for the buck. Cost.

I'd keep Cisco if I already had it as long as it was working well, but I'd prefer pfsense if I was having to buy new as long as the functions I needed were supported.

Derelict

My point is why not just failover on the Ciscos?

afreaken

@Derelict:

My point is why not just failover on the Ciscos?

That's possibly a valid question and may be the route we end up going, but there is a little bit of complexity in our setup that does not need to be there, I am simply trying to replace a different linux system which we have little control over and is currently dieing. Once the pfsense is in place and working, the next step may be to remove them and have the CISCO's only since the pfsense cost nothing

I did not create this setup, I am simply trying to mend it whenever I am given room to do so.

Derelict

Then yes, pfSense should be able to monitor connectivity to each remote bridge and fail over. Look at the techniques for Multi-WAN. Your situation will be similar.

https://doc.pfsense.org/index.php/Multi-WAN

afreaken

@Derelict:

Then yes, pfSense should be able to monitor connectivity to each remote bridge and fail over. Look at the techniques for Multi-WAN. Your situation will be similar.

https://doc.pfsense.org/index.php/Multi-WAN

Sure, but that's about as far as I can get. I get a little fuzzy when it comes to the routing of traffic from 1 side to the other when it comes to having a fail-over backhaul link:

pfSense2(192.168.10.66) –>> (192.168.10.65)pfSense1(192.168.10.2) -->> (192.168.10.1)CISCO(192.168.1.1)
pfSense2(192.168.10.34) -->> (192.168.10.33)pfSense1(192.168.10.2) -->> (192.168.10.1)CISCO(192.168.1.1)

The current Linux box we have is in need of replacing, it has next hop routes + NAT'ing to accomplish this routing. I am not sure how to configure the equivalent in pfSense. As I said, I'm still learning networking, it's not something that was covered extensively in school. Does pfSense handle the routing if a fail-over occurs? I've hit issues where I was trying to set up routes for both links and had it give me an error due to the route existing on the other link.

Derelict

Did you read the document?

mikeisfly

Pay for PfSense support. You are getting hardware for free, and this helps the project out. As far learning if you make your diagram a little clearer might be easier to help. But PfSense can do:

1. NAT
2. RIP Routing
3. OSPF Routing
4. BGP Routing

So you should be able to use PfSense to do what you want.

afreaken

@mikeisfly:

Pay for PfSense support. You are getting hardware for free, and this helps the project out. As far learning if you make your diagram a little clearer might be easier to help. But PfSense can do:

1. NAT
2. RIP Routing
3. OSPF Routing
4. BGP Routing

So you should be able to use PfSense to do what you want.

Unfortunately my understanding of networking is very basic. I am trying to follow this logically, and have come to the understanding that I need to create firewall rules to pass traffic to the correct interfaces + multiWAN gateway. Please correct me if I am wrong, good chance I will be. However if this is true, I am still unsure on setting up the rules and whether I need to explicitly account for the fact that the pfSense device is not directly connected to the end network. If someone could guide me for 1 route from 1 side to the other, that would be awesome.

I have setup and configured GW's and a Load-Balanced group + all interfaces + DNS.

This is the route it takes from our 1.0/24 LAN to our 2.0/24 LAN:

1.0/24 Network –> [1.1 [u]CISCO 10.1] –> [10.2(LAN) ([u]pfSense1)*** MultiWAN Interfaces (WAN1)10.33 + (WAN2)10.65***]

–>WiFi BackHaul-->

[MultiWAN Interfaces (WAN1)10.34 + (WAN2)10.66([u]pfSense2) (LAN)10.98] –> [10.97 [u]CISCO 2.1] –> 2.0/24 Network

Currently I have a floating firewall rule in place set to:
Pass
Interfaces: WAN1 + WAN2
Direction: out
Protocol: any
Source: 1.0/24 -- (Using an alias with multiple local networks)
Destination: 2.0/24 -- (Using an alias for multiple remote networks)
Adv. GW: MultiWAN GW's

Do I need to add incoming rules? additional rules? Is this rule even correct?

Derelict

What are all the netmasks everywhere?

You will want to disable NAT, too.

You can just disable NAT and add pass IPv4 any source any dest any rules to all the pfSense interfaces in question.

Do new connections need to be established in both directions? If so you will need to configure "Multi-WAN" on both pfSenses, with each side testing both links and failing over if necessary. I've never done that before but it should work.

afreaken

@Derelict:

What are all the netmasks everywhere?

You will want to disable NAT, too.

You can just disable NAT and add pass IPv4 any source any dest any rules to all the pfSense interfaces in question.

Do new connections need to be established in both directions? If so you will need to configure "Multi-WAN" on both pfSenses, with each side testing both links and failing over if necessary. I've never done that before but it should work.

Thanks for the input, I will try disabling NAT. And yes, I don't see many examples of a loadbalanced backhaul on the internet, maybe my searching skills suck, idk.
And yes, I will multiWAN the remote site possibly later today if I can at least get this side configured right. I have to drive there since I screwed up a rule and lost connection, with no IT staff at that location /sadface.

Could you be a little clearer on the "pass IPv4 any source any dest any rules to all the pfSense interfaces in question"
Are you implying a single rule Any Source any Dest + All Interfaces? This is confusing me since there is the MultiWAN GW.
Interfaces in question? LAN + WAN1 + WAN2?
Remember I am learning. So treat me like I'm stupid.

As for the net masks:
1.0/24
2.0/24
10.0/27
10.33/27
10.64/27
10.97/27

I did not set up this network, clearly.

Derelict

Pass rules just let traffic into the interfaces. They have nothing to do with routing. The rules, however, can set gateways which DO affect routing.

What are all your pfSense interface names on both sides and what are they connected to?

While you're at the remote site you should assign someone local and show them how to log into pfSense (make a bookmark) so you can talk them through fixing something without having to go there.

afreaken

@Derelict:

Pass rules just let traffic into the interfaces. They have nothing to do with routing. The rules, however, can set gateways which DO affect routing.

What are all your pfSense interface names on both sides and what are they connected to?

While you're at the remote site you should assign someone local and show them how to log into pfSense (make a bookmark) so you can talk them through fixing something without having to go there.

Unfortunately that's not an option, the last time I asked the most technical person there to do something, walked him through it, he ended up changing the IP address of the machine to the IP of the server that he was trying to connect to. I was having him change the IP and DNS of the machine since it had been configured for a different network, and he was trying to connect to a terminal server on our side.

pfSense1 Interfaces
–---------------
LAN 10.2 (192.168.10.2/27) --- Connected to CISCO 10.1 (192.168.10.1/27) Has GW 1.1 (192.168.1.1/24)
WiFi1 10.33 (192.168.10.33/27) --- Connected to WiFi1T 10.34 (192.168.10.34/27)
WiFi2 10.65 (192.168.10.65/27) --- Connected to WiFi2T 10.66 (192.168.10.66/27)
(Also have a management interface for testing since I do not want to connect the LAN up while the rules are not set and the current Interface IP is in use)
192.168.1.180 --- On local network

pfSense2 Interfaces

LAN 10.98 (192.168.10.98/27) --- Connected to CISCO 10.97 (192.168.10.97/27) Has GW 2.1 (192.168.2.1/24)
WiFi1T 10.34 (192.168.10.34/27) --- Connected to WiFi1 10.33 (192.168.10.33/27)
WiFi2T 10.65 (192.168.10.65/27) --- Connected to WiFi2 10.65 (192.168.10.65/27)
(Also have a management interface for testing since I do not want to connect the LAN up while the rules are not set and the current Interface IP is in use)
192.168.2.180 --- On local network

Was reading another thread and amended my current rule for the time being:
Currently I have a floating firewall rule in place set to:
Pass
Interfaces: LAN
Direction: out
Protocol: any
Source: Any
Destination: Any
Adv. GW: MultiWAN GW's

Derelict

And all of that is currently up and running? You just need to add failover?

afreaken

@Derelict:

And all of that is currently up and running? You just need to add failover?

Currently our system we have is "functioning" but it doesn't do it very well; it doesn't handle load balancing/failover/link aggregation very well at all. Also 2 ports are dead on the remote side, possibly due to a surge from the cable modem.

I will know later today if the pfsense box is working if I am confident with the current rule setup, I will drive to the other location and fix the box there and mirror the rules.

I have set the 2 WiFi WAN interfaces up on the same tier. From my previous testing they should have roughly the same throughput, though 1 varies wildly. I'll need to do some fine tuning later once I actually get the link up.

So again, could you clarify my question about your comment in the previous post for the rules. And when I set the adv. GW –> MultiWAN setting for the rule, I have to choose either in or out, do I need to setup both? or OUT for LAN (what I already set up) and IN for the WAN's?

Also, thank you for responding to my post and trying to be helpful.

Derelict

In/Out is for limiters and is totally unrelated to multi-wan. All you have to do is set the gateway group in the rule.

afreaken

@Derelict:

In/Out is for limiters and is totally unrelated to multi-wan. All you have to do is set the gateway group in the rule.

Well that's where I get confused because when I set the GW group for the rule, it says I need to state a direction IN/OUT.

Could you clarify "the rule" it seems I don't need to set IN/OUT when setting rules on individual interfaces vs a floating rule.

So which interface should have the GW set? the WAN's? with ANY source to ANY dest?

I currently can ping from my computer the WiFi2 Interface, the WiFi router, but not the remote WiFi router or Interface. The pfSense box CAN.

Granted those are not critical to access, however if I cannot access those with the rules I have set, I'm skeptical of reaching the 2.0/24 network over the WiFi.

Derelict

No idea what you're talking about. See the attached. All you have to do is set the gateway group. If it's doing something else, fix your javascript/browser. All of those advanced sections are independent of each other.

afreaken

@Derelict:

No idea what you're talking about. See the attached. All you have to do is set the gateway group. If it's doing something else, fix your javascript/browser. All of those advanced sections are independent of each other.

Yeah I did that already, all I'm asking is which interface this needs to be done on, or set of interfaces. Earlier you said interfaces in question, I'm trying to figure out which ones. Maybe I have the right config and something is wrong with my wifi router, but I can't ping the remote one from my PC but I can from the pfsense box.

afreaken

Well I guess my rules are fine, I for some reason cannot ping the gateway on the remote side as well as the wifi router, however now that I have fixed the pfsense box rules, I can connect to the box from this side. cannot ping any of it's interfaces though, or the remote network

Derelict

You need static routes to the remote network. Did you allow any traffic or just TCP/UDP? If the latter, it will block ICMP (ping).

afreaken

@Derelict:

You need static routes to the remote network. Did you allow any traffic or just TCP/UDP? If the latter, it will block ICMP (ping).

Allow Any, Figured out the ping issue, there was an old rule set that I had to delete. Will set statics now. This is where I have been confused before, when I set the static for a remote network over the multiWAN WiFi backhaul, it wants to choose an interface, and I cannot replicate the rule over another interface. Does the multiWAN gateway handle the second route?

I noticed if I use an alias it seems to allow me to set statics for the same alias across multiple GW's, is this OK?

Derelict

You need static routes for the cisco to pfsense and pfsense to cisco paths. From pfSense over the wireless links, use policy routing on your LAN rules to the gateway group.

The Cisco1 needs to know how to get to 192.168.2.0/24. It needs a route for 192.168.2.0/24 to pfSense 1.
pfSense 1 needs to know how to get to 192.168.2.0/24. It needs a policy route on LAN source any dest 192.168.2.0/24 with the wireless gateway group set.
pfSense 2 needs to know how to get to 192.168.2.0/24. It needs a gateway created for Cisco2. and a static route for 192.168.2.0/24 to Cisco2.
Cisco2 does not need a route because 192.168.2.0/24 is a connected network.

And the reverse for the other direction. This is assuming the wireless links/pfSense is not the default gateway at either location. in that case just point the default to the next upstream device.

@Derelict:

My point is why not just failover on the Ciscos?

Ok this is true, but on the other hand why not laod balance using policy based routing
between the pfSense firewalls?

phil.davis

I am not sure why you are using floating rules - that is where the confusion about In/Out is coming from, floating rules let you apply the rule on traffic coming In or Out of an Interface, there is also the terminology of In/Out for limiters - a different place in the GUI and different thing.

I think you really want:

Static route on each pfSense pointing to its local Cisco for a route to the subnet behind the Cisco.
Do not do any NAT on the "WAN" backhaul interfaces - perhaps just make those ordinary LAN-style interfaces (no Upstream Gateway defined)
Gateway defined for the IP address on the other side of each "WAN" backhaul link.
Gateway group/s that include the 2 gateways with whatever tiers you want to make it load-balance or fail-over.
Pass rule/s on the LAN interface to pass traffic source: the subnet behind the Cisco (and the local pfSense LAN subnet for completeness), destination: the subnet behind remote end Cisco (and the remote end pfSense LAN subnet for completeness), gateway: the gateway group you made.
Pass rule/s on the WAN backhaul interfaces to allow incoming traffic from the other end (or just allow all to get it going).
(These "pass" rules on individual interfaces are "In" rules - you do not get a choice about that - they will apply to traffic flows initiated from the interface they are on)

It should all work in a conceptually similar way to having a couple of site-to-site VPN tunnels between pfSense systems and routing intranet traffic across them.

afreaken

@phil.davis:

I am not sure why you are using floating rules - that is where the confusion about In/Out is coming from, floating rules let you apply the rule on traffic coming In or Out of an Interface, there is also the terminology of In/Out for limiters - a different place in the GUI and different thing.

I think you really want:

Static route on each pfSense pointing to its local Cisco for a route to the subnet behind the Cisco.

Do not do any NAT on the "WAN" backhaul interfaces - perhaps just make those ordinary LAN-style interfaces (no Upstream Gateway defined)

Gateway defined for the IP address on the other side of each "WAN" backhaul link.

Gateway group/s that include the 2 gateways with whatever tiers you want to make it load-balance or fail-over.

Pass rule/s on the LAN interface to pass traffic source: the subnet behind the Cisco (and the local pfSense LAN subnet for completeness), destination: the subnet behind remote end Cisco (and the remote end pfSense LAN subnet for completeness), gateway: the gateway group you made.

Pass rule/s on the WAN backhaul interfaces to allow incoming traffic from the other end (or just allow all to get it going).
(These "pass" rules on individual interfaces are "In" rules - you do not get a choice about that - they will apply to traffic flows initiated from the interface they are on)

It should all work in a conceptually similar way to having a couple of site-to-site VPN tunnels between pfSense systems and routing intranet traffic across them.

Thanks, I will try this out. What Derelict recommended may work, I need to go to the remote site and fix/replace the remote pfsense box. It gets stuck in the shutdown process when restarting. I got sick last week and took time off so I haven't gotten around to it yet. I will try both and respond back with the results.

afreaken

@afreaken:

@phil.davis:

I am not sure why you are using floating rules - that is where the confusion about In/Out is coming from, floating rules let you apply the rule on traffic coming In or Out of an Interface, there is also the terminology of In/Out for limiters - a different place in the GUI and different thing.

I think you really want:

Static route on each pfSense pointing to its local Cisco for a route to the subnet behind the Cisco.

Do not do any NAT on the "WAN" backhaul interfaces - perhaps just make those ordinary LAN-style interfaces (no Upstream Gateway defined)

Gateway defined for the IP address on the other side of each "WAN" backhaul link.

Gateway group/s that include the 2 gateways with whatever tiers you want to make it load-balance or fail-over.

Pass rule/s on the LAN interface to pass traffic source: the subnet behind the Cisco (and the local pfSense LAN subnet for completeness), destination: the subnet behind remote end Cisco (and the remote end pfSense LAN subnet for completeness), gateway: the gateway group you made.

Pass rule/s on the WAN backhaul interfaces to allow incoming traffic from the other end (or just allow all to get it going).
(These "pass" rules on individual interfaces are "In" rules - you do not get a choice about that - they will apply to traffic flows initiated from the interface they are on)

It should all work in a conceptually similar way to having a couple of site-to-site VPN tunnels between pfSense systems and routing intranet traffic across them.

Thanks, I will try this out. What Derelict recommended may work, I need to go to the remote site and fix/replace the remote pfsense box. It gets stuck in the shutdown process when restarting. I got sick last week and took time off so I haven't gotten around to it yet. I will try both and respond back with the results.

After doing my testing, I got the setup working. Used this as the guide and it does work. The pfsense box is deployed on 1 end, the end which was in need of being replaced. Had a few issues which needed to be resolved as we have several networks to join up and allow traffic to, however they are not critical to business processes, only critical for IT and some small use cases for some users. They are resolved afaik, guess I'll find out when something doesn't work and someone complains about it to me.

A big thank you to phil.davis and Derelict for the helpful input. I learned a lot through the input from you two. Also thanks to the others who gave input, promoting useful discussion of this topic.

afreaken

One issue I've been seeing is that sometimes pings will not resolve going across the wifi link between the pfsense and opposite linux box. The ping is going from client to server or server to client, both do not resolve, and not all clients experience the problems. They are on the same network.

Derelict

What does "resolve" mean in the context of using ping/icmp?

afreaken

@Derelict:

What does "resolve" mean in the context of using ping/icmp?

requests time out. Also, if a gateway goes down in a group, clients that were once able to ping, can no longer ping.

afreaken

well this may be unnecessary. I replaced the box on this side with a pfsense box and the timeouts stopped. It may have been an issue with the linux box on this side. Will post with updates after it's been running overnight.

afreaken

One question I have is how do I give this box access to the internet? Normally you would have a wan connected to the internet, but this is routing for a private network. Do I need to create policy routing?

pfsense box [LAN port 10.2] –-- [10.1 WiFi backhaul port] Cisco [to external IP]

Derelict

NAT for 10.2 on the Cisco. Set pfSense's default gateway to 10.1 and configure DNS, etc.

Derelict

Also, if you're pinging across a load-balanced link and the link your state is on goes down, your ping will stop because the state doesn't move to the other circuit. Stopping and restarting the ping should result in pings again as a new state is created using the other link.

afreaken

@Derelict:

Also, if you're pinging across a load-balanced link and the link your state is on goes down, your ping will stop because the state doesn't move to the other circuit. Stopping and restarting the ping should result in pings again as a new state is created using the other link.

Which is what I tried to do, however the machiens would still not return a ping. I tried even resetting the states within the pfsense box, but nothing happened. There are some issues with the linux boxes we are replacing, and this seems to be one of them. After switching out both ends, compared to only 1 end, pings have been very stable. A lot of the routing it should be doing, doesn't seem to be done. The interface will say one thing, but it will do something else completely. We also have little control over the backend of the linux box.

afreaken

@afreaken:

well this may be unnecessary. I replaced the box on this side with a pfsense box and the timeouts stopped. It may have been an issue with the linux box on this side. Will post with updates after it's been running overnight.

Well things have been running smoothly for the past few days, things are looking good.