[Solved] Can not route between Multi-Homed Clients Behind pfSense

helloadam · Oct 28, 2017, 7:57 PM

Hello,

I am trying to have two clients, which are multi-homed, access each other behind a private network managed by pfSense. When the clients are only using the WAN (not managed by pfSense) they are able to access each other. When the clients are only using the private network, managed by pfSense, they can access each other.

However, when the clients are multi-homed, configured to use WAN and private network, they can not access each other over the private network.

Attached is a layout of the environment and how the clients are connected. I am pretty sure I have to add some sort of static routes or and gateways configuration in pfSense. Is this right? What am I missing? Some more details below:

Thank you in advanced,
-Adam

pfSense Setup:
-Its not the firewall
I log all firewall rules, allow and deny, and I can see from the logs that traffic is not being blocked. As mentioned before, the setup works fine when clients are not mulit-homed. To further prove its not the firewall, I disabled all rules and created one rule in pfSense, on each nic, that allows traffic from anywhere to anywhere. Clients also have iptables rules flushed. Block private networks is only setup on WAN port on pfSense

-Interface: intra212 and TESTNET
IPv4 Upstream gateway: NONE
Static IPv4 setup with each interface taking the gateway IPv4 address within their respective subnet.
See picture for more details

Firewall NAT Mode
Automatic outbound NAT rule generation.

-DHCP Service on intra212 and TESTNET
Default configuration. Nothing changed on settings page besides: Enable DHCP, Change DHCP display lease time from UTC to local time and Enable RRD statistics graphs

System / Routing / Gateways
Only one gateway configured and that is pfSense WAN, public IPv4
System / Routing / Static Routes
Empty
Client 1 & Client 2 Setup:
iptables has been flushed
WAN nic (ens192) is configured with static IP information
WAN gateway is configured as default

private network nic (ens224) is configured as DHCP client

host information (client 2 has same setup, just different IPv4 numbers)


$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         XX.123.244.225  0.0.0.0         UG    0      0        0 ens192
0.0.0.0         10.42.0.1       0.0.0.0         UG    203    0        0 ens224
10.42.0.0       0.0.0.0         255.255.255.0   U     203    0        0 ens224
XX.123.244.224  0.0.0.0         255.255.255.224 U     0      0        0 ens192
$ ip route show table main
default via XX.123.244.225 dev ens192 
default via 10.42.0.1 dev ens224 src 10.42.0.65 metric 203 
10.42.0.0/24 dev ens224 proto kernel scope link src 10.42.0.65 metric 203 
XX.123.244.224/27 dev ens192 proto kernel scope link src XX.123.244.235


$ ip a
...
...
2: ens192: <broadcast,multicast,up,lower_up>mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:a7:07:41 brd ff:ff:ff:ff:ff:ff
    inet XX.123.244.235/27 brd XX.123.244.255 scope global ens192
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:fea7:741/64 scope link 
       valid_lft forever preferred_lft forever
3: ens224: <broadcast,multicast,up,lower_up>mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:8e:dd:58 brd ff:ff:ff:ff:ff:ff
    inet 10.42.0.65/24 brd 10.42.0.255 scope global ens224
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:fe8e:dd58/64 scope link 
       valid_lft forever preferred_lft forever</broadcast,multicast,up,lower_up></broadcast,multicast,up,lower_up>

johnpoz · Oct 26, 2017, 8:41 AM

0.0.0.0 XX.123.244.225 0.0.0.0 UG 0 0 0 ens192
0.0.0.0 10.42.0.1 0.0.0.0 UG 203 0 0 ens224

This is the problem.. you have 2 default routes.

If you want your clients to talk via a private networks.. Then the only route these host would have is to the other network for that interface - it would not have a default gateway set on it. You would just setup a specific route on the box.

So for 10.42.0.64 he would have a route for 10.212.228/24 talk to 10.42.0.1 (pfsense). This route is setup specifically on that host..

Other client 10.212.228.64 would not have gateway set on this interface either and would have a route on itself for 10.42.0/24 talk to 10.212.228.1 (pfsense). Again this route is on the 10.212.228.64 host

helloadam · Oct 28, 2017, 7:57 PM

@johnpoz:

0.0.0.0 XX.123.244.225 0.0.0.0 UG 0 0 0 ens192
0.0.0.0 10.42.0.1 0.0.0.0 UG 203 0 0 ens224

This is the problem.. you have 2 default routes.

If you want your clients to talk via a private networks.. Then the only route these host would have is to the other network for that interface - it would not have a default gateway set on it. You would just setup a specific route on the box.

So for 10.42.0.64 he would have a route for 10.212.228/24 talk to 10.42.0.1 (pfsense). This route is setup specifically on that host..

Other client 10.212.228.64 would not have gateway set on this interface either and would have a route on itself for 10.42.0/24 talk to 10.212.228.1 (pfsense). Again this route is on the 10.212.228.64 host

John,

Thank you for the information and yes, this fixed the issue.

Rather then going to each host and defining a static route for 10.42.0.0/24 and 10.212.228.0/24 I ended up creating one for the entire 10.0.0.0/8 range to the respected gateways. This will allow me flexibility in the future if I add a new network or have to re-ip things. I certainly do not want to login to each host and have to update their respective routes when the network changes.

Example:


ip route add 10.0.0.0/8 via 10.212.228.1 dev ens224

-Adam

johnpoz · Oct 29, 2017, 9:06 AM

Sure that works.. Another solution would not to multi home you boxes like that.. Seems kind of pointless if you ask me..

Also such a setup doesn't stop them from talking to each other on their other network.. Why would you not just put the clients behind pfsense for everything?

What is the point of the multihomed setup? That you want/need to firewall?