Problems setting up correct routing for different internal networks

[th]

Hi, I'm having troubles setting up routing in pfSense (as part of bigger problems...).

My pfSense has WAN (igb0) connected to my ISP, and LAN (igb1) (10.0.0.0/24) is connected to a switch. I also have a wireless card, OPT2 (ath0_wlan0) (10.1.0.0/24), and LAN and OPT2 are bridged. DHCP is running on LAN and OPT2.

I want to stop using the wireless card in the pfSense router, and instead use a Ubiquiti AP running OpenWrt which is connected to the switch. The Ubiquiti has two wireless networks configured on it, one for me (MYNET, also 10.0.0.0/24 by using the pfSense DHCP server) and one for guests (GUESTS, 192.168.3.0/24, using a DHCP server on the Ubiquiti).

I know it might seem weird, but I want to have separate private networks for MYNET and GUESTS. And since laptops connected to MYNET should have access to machines on my home network I figure it's best to use the pfSense DHCP server. But since devices connected to GUESTS should not have anything to do with my home network, I figured a DHCP server on the Ubiquiti putting them on a completely different network was a good idea.

When I wirelessly connect a laptop to MYNET everything works as expected. I can surf, and I can connect to machines on my home network.

But when I wirelessly connect a laptop to GUESTS nothing works. I can't surf and I can't connect to anything on my home network.

Here's a (simplified and slightly anonymized) tcpdump output from pfSense's igb0, when the laptop is connecting to a www server when using MYNET:

17:40:21.598915 IP AA.BB.CC.137.30945 > XX.YY.ZZ.35.80: Flags [S], seq 1761383132
17:40:21.600829 IP XX.YY.ZZ.35.80 > AA.BB.CC.137.30945: Flags [S.], seq 1236209384, ack 1761383133
17:40:21.604541 IP AA.BB.CC.137.30945 > XX.YY.ZZ.35.80: Flags [.], ack 1
17:40:21.604591 IP AA.BB.CC.137.30945 > XX.YY.ZZ.35.80: Flags [P.], seq 1:19, ack 1
17:40:21.606431 IP XX.YY.ZZ.35.80 > AA.BB.CC.137.30945: Flags [.], ack 19
17:40:21.635453 IP XX.YY.ZZ.35.80 > AA.BB.CC.137.30945: Flags [P.], seq 1:568, ack 19
17:40:21.635501 IP XX.YY.ZZ.35.80 > AA.BB.CC.137.30945: Flags [F.], seq 568, ack 19
17:40:21.639749 IP AA.BB.CC.137.30945 > XX.YY.ZZ.35.80: Flags [.], ack 568
17:40:21.640069 IP AA.BB.CC.137.30945 > XX.YY.ZZ.35.80: Flags [F.], seq 19, ack 569
17:40:21.642043 IP XX.YY.ZZ.35.80 > AA.BB.CC.137.30945: Flags [.], ack 20

And then trying the same thing when the laptop is connected to GUESTS:

17:41:13.668985 IP AA.BB.CC.137.23445 > XX.YY.ZZ.35.80: Flags [S], seq 2171037162
17:41:13.670965 IP XX.YY.ZZ.35.80 > AA.BB.CC.137.23445: Flags [S.], seq 3417218341, ack 2171037163
17:41:13.671012 IP XX.YY.ZZ.35.80 > 192.168.3.208.44424: Flags [S.], seq 3417218341, ack 2171037163
17:41:13.973662 IP XX.YY.ZZ.35.80 > AA.BB.CC.137.23445: Flags [S.], seq 3417218341, ack 2171037163
17:41:13.973707 IP XX.YY.ZZ.35.80 > 192.168.3.208.44424: Flags [S.], seq 3417218341, ack 2171037163
17:41:14.693800 IP AA.BB.CC.137.23445 > XX.YY.ZZ.35.80: Flags [S], seq 2171037162
17:41:14.695898 IP XX.YY.ZZ.35.80 > AA.BB.CC.137.23445: Flags [S.], seq 3417218341, ack 2171037163
17:41:14.695942 IP XX.YY.ZZ.35.80 > 192.168.3.208.44424: Flags [S.], seq 3417218341, ack 2171037163
17:41:16.733771 IP XX.YY.ZZ.35.80 > AA.BB.CC.137.23445: Flags [S.], seq 3417218341, ack 2171037163
17:41:16.733822 IP XX.YY.ZZ.35.80 > 192.168.3.208.44424: Flags [S.], seq 3417218341, ack 2171037163
17:41:16.773788 IP AA.BB.CC.137.23445 > XX.YY.ZZ.35.80: Flags [S], seq 2171037162
17:41:16.775771 IP XX.YY.ZZ.35.80 > AA.BB.CC.137.23445: Flags [S.], seq 3417218341, ack 2171037163
17:41:16.775813 IP XX.YY.ZZ.35.80 > 192.168.3.208.44424: Flags [S.], seq 3417218341, ack 2171037163
17:41:20.829694 IP XX.YY.ZZ.35.80 > AA.BB.CC.137.23445: Flags [S.], seq 3417218341, ack 2171037163
17:41:20.829748 IP XX.YY.ZZ.35.80 > 192.168.3.208.44424: Flags [S.], seq 3417218341, ack 2171037163

So pfSense tries sending packets addressed to 192.168.3.208 (my laptop) out on igb0 (WAN) instead of sending them on igb1 (LAN) where they would have a chance of reaching my Ubiquiti.

Running tcpdump on igb1 verifies that when the laptop is connected to MYNET, packets to 10.0.0.217 (my laptop) are sent on igb1 (and reaches my laptop), but when connected to GUESTS all I see are the SYN packets sent by the laptop, and nothing else.

As far as I can tell I need to set up policy based routing to make pfSense send the 192.168.3.208 packets out on igb1 (to the switch), but all search hits so far has been about policy based routing when the pfSense is connected to two WANs, and my problem is the opposite; one WAN, but two different internal networks, and I can't get it to work.

Now I know I'm doing weird stuff, but I still think it should be possible. Or is there a tutorial somewhere with a better setup, achieving the network separation I'm aiming for? Do I have to use a physical port on the pfSense machine and put the Ubiquiti there, instead of having it connected to the switch?

There might be other things I need to fix as well, but right now I just want to learn how to set up policy based routing to get this particular part of my problems solved. If possible...

Thanks,

[bingo600]

@th

OpenWRT on a UBI AP , i didn't know you could do that.

Well to me it seems like you should use multi vlans between the pfSense & the AP.

If your AP doesn't support that, you really don't want to try two different ip ranges on the AP.

/Bingo