Routing issue with two LANs and external router

fabio.grasso.fy

Hello there,
I'm facing a routing issue with pfSense 2.4.5.

I have a configuration with a certain complexity: pfSense has an external interface (WAN) and two internal ones (LAN and OPT1).
Another router is then connected to the LAN and has its own dedicated subnet (let's call it EXT).

My problem is this: communication between LAN and OPT1 takes place without problems.
What is not working properly is the communication between the OPT1 network and the EXT network

Briefly summarize the networks:

LAN 172.16.0.0/16
OPT1 10.16.0.0/16
EXT 192.168.0.0/24

pfSense LAN 172.16.0.1 - OPT1 10.16.0.1
router LAN 172.16.0.10 - EXT 192.168.0.10
server2 EXT 192.168.0.11 (gateway 192.168.0.10)
server3 OPT1 10.16.0.11 (gateway 10.16.0.1)

On pfSense I configured a route for
192.168.0.0/24 via 172.16.0.10

and then I created the ACLs on LAN and OPT1 (I also tried to put an allow any any rule, to make sure there were no blockages by the firewall). I also activated the "Bypass firewall rules for traffic on the same interface" option.

Here are the results of my tests:

pfSense > Router.............OK
pfSense > EXT Server2........OK
pfSense > OPT1 (Server3).....OK

router > LAN pfSense.........OK
router > EXT Server2.........OK
router > OPT1 (Server3)......OK	hop 172.16.0.1 (pfSense)

EXT Server2 > EXT router.....OK
EXT Server2 > LAN pfSense....OK	hop 192.168.0.10 (EXT router)
EXT Server2 > OPT1 (Server3).NO	hops 192.168.0.10 (EXT router) > 172.16.0.1 (LAN pfSense)

OPT1 (Server3) > OPT1 pfSense.OK
OPT1 (Server3) > OPT1 router..OK hop 10.16.0.1 (pfSense)
OPT1 (Server3) > EXT Server2..NO hop 10.16.0.1 (pfSense)

I have done various tests, packet captures, etc. but I can't understand what I did wrong.
I therefore came to the doubt that pfSense is not able to do routing in this way (for example, there is a similar limitation on Cisco ASAs, whisch I've used in the past).

I did a simulation of a similar configuration using a Linux server (a trivial installation with two NICs and ip_forward active) as a router instead of the pfSense and it works correctly.

Do you have any suggestion?

Thanks

viragomann

@fabio-grasso-fy said in Routing issue with two LANs and external router:

Another router is then connected to the LAN and has its own dedicated subnet (let's call it EXT).

That's not a recommended way to connect two routers whos networks should be able to communicate.

@fabio-grasso-fy said in Routing issue with two LANs and external router:

On pfSense I configured a route for
192.168.0.0/24 via 172.16.0.10

So pfSense knows the route to the network behind the second router, but your LAN devices don't.

Add this route to each LAN device you want to talk with that network or remove the second router from LAN and connect it on a separate interface to pfSense.

fabio.grasso.fy

@viragomann said in Routing issue with two LANs and external router:

@fabio-grasso-fy said in Routing issue with two LANs and external router:

Another router is then connected to the LAN and has its own dedicated subnet (let's call it EXT).

That's not a recommended way to connect two routers whos networks should be able to communicate.

I was trying to use pfSense as "main" router, without adding another one. Since pfSense is basically a FreeBSD box I was supposed to use it for both routing and firewalling.

I don't understand why you say it's not the recommended way, there is an extra hop in the middle but it should work

@fabio-grasso-fy said in Routing issue with two LANs and external router:

On pfSense I configured a route for
192.168.0.0/24 via 172.16.0.10

So pfSense knows the route to the network behind the second router, but your LAN devices don't.

Add this route to each LAN device you want to talk with that network

Why I should do this? I have pfSense as default gateway in my LAN devices, so it's supposed that is pfSense that do the routing job.

or remove the second router from LAN and connect it on a separate interface to pfSense.

I cannot do this: If I do so, I will have two interface in the same subnet and this will not work.

As I said if I replace pfSense with a simple Linux with active ip_forward, everything works correctly, so I have the impression that something is missing in the pfsense configuration

Thanks

viragomann

Routing works properly only if the source and the destination are on different interfaces. If they are on the same, you will get an asymmetric routing issue.

@fabio-grasso-fy said in Routing issue with two LANs and external router:

or remove the second router from LAN and connect it on a separate interface to pfSense.

I cannot do this: If I do so, I will have two interface in the same subnet and this will not work.

Of course, the other interface has to be a unique subnet assigned.

@fabio-grasso-fy said in Routing issue with two LANs and external router:

As I said if I replace pfSense with a simple Linux with active ip_forward, everything works correctly, so I have the impression that something is missing in the pfsense configuration

Presumably the Linux box did masquerading on that traffic.
Yes, this can also be done on pfSense, but it's not an optimal setup though. By doing masqerading, all pakets seem to come from the router itself instead from the origin device.

However, if you want to set that up:
Firewall > NAT > Outbound
Switch into the hybrid mode and save.
Add a new rule:
Interface: LAN
source: LAN net
destination: 192.168.0.0/24
Translation: interface address (default)
Save

fabio.grasso.fy

@viragomann said in Routing issue with two LANs and external router:

Routing works properly only if the source and the destination are on different interfaces. If they are on the same, you will get an asymmetric routing issue.

Yes, that's the classic asymmetric routing. But what deceived me is what is written in the pfSense documentation:

In asymmetric routing scenarios, there is an option that may be used to prevent legitimate traffic from being dropped. The option adds firewall rules which allow all traffic between networks defined in static routes using a more permissive set of rule options and state handling. To activate this option:

Click System > Advanced
Click the Firewall/NAT tab
Check Bypass firewall rules for traffic on the same interface
Click Save

So I was thinking that the option "Bypass firewall rules for traffic on the same interface" was enough to make everything working

Yes, this can also be done on pfSense, but it's not an optimal setup though. By doing masqerading, all pakets seem to come from the router itself instead from the origin device.

However, if you want to set that up:
Firewall > NAT > Outbound
Switch into the hybrid mode and save.
Add a new rule:
Interface: LAN
source: LAN net
destination: 192.168.0.0/24
Translation: interface address (default)
Save

That's make sense and can be a solution. I will try it but I feel confident.

Thanks for your help