[Solved] Can not route between Multi-Homed Clients Behind pfSense



  • Hello,

    I am trying to have two clients, which are multi-homed, access each other behind a private network managed by pfSense. When the clients are only using the WAN (not managed by pfSense) they are able to access each other. When the clients are only using the private network, managed by pfSense, they can access each other.

    However, when the clients are multi-homed, configured to use WAN and private network, they can not access each other over the private network.

    Attached is a layout of the environment and how the clients are connected. I am pretty sure I have to add some sort of static routes or and gateways configuration in pfSense. Is this right? What am I missing? Some more details below:

    Thank you in advanced,
    -Adam

    pfSense Setup:
    -Its not the firewall
    I log all firewall rules, allow and deny, and I can see from the logs that traffic is not being blocked. As mentioned before, the setup works fine when clients are not mulit-homed. To further prove its not the firewall, I disabled all rules and created one rule in pfSense, on each nic, that allows traffic from anywhere to anywhere. Clients also have iptables rules flushed. Block private networks is only setup on WAN port on pfSense

    -Interface: intra212 and TESTNET
    IPv4 Upstream gateway: NONE
    Static IPv4 setup with each interface taking the gateway IPv4 address within their respective subnet.
    See picture for more details

    • Firewall NAT Mode
      Automatic outbound NAT rule generation.

    -DHCP Service on intra212 and TESTNET
    Default configuration. Nothing changed on settings page besides: Enable DHCP, Change DHCP display lease time from UTC to local time and Enable RRD statistics graphs

    • System / Routing / Gateways
      Only one gateway configured and that is pfSense WAN, public IPv4

    • System / Routing / Static Routes
      Empty

    • Client 1 & Client 2 Setup:
      iptables has been flushed
      WAN nic (ens192) is configured with static IP information
      WAN gateway is configured as default

    private network nic (ens224) is configured as DHCP client

    host information (client 2 has same setup, just different IPv4 numbers)

    
    $ route -n
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
    0.0.0.0         XX.123.244.225  0.0.0.0         UG    0      0        0 ens192
    0.0.0.0         10.42.0.1       0.0.0.0         UG    203    0        0 ens224
    10.42.0.0       0.0.0.0         255.255.255.0   U     203    0        0 ens224
    XX.123.244.224  0.0.0.0         255.255.255.224 U     0      0        0 ens192
    $ ip route show table main
    default via XX.123.244.225 dev ens192 
    default via 10.42.0.1 dev ens224 src 10.42.0.65 metric 203 
    10.42.0.0/24 dev ens224 proto kernel scope link src 10.42.0.65 metric 203 
    XX.123.244.224/27 dev ens192 proto kernel scope link src XX.123.244.235
    
    
    
    $ ip a
    ...
    ...
    2: ens192: <broadcast,multicast,up,lower_up>mtu 1500 qdisc mq state UP group default qlen 1000
        link/ether 00:50:56:a7:07:41 brd ff:ff:ff:ff:ff:ff
        inet XX.123.244.235/27 brd XX.123.244.255 scope global ens192
           valid_lft forever preferred_lft forever
        inet6 fe80::250:56ff:fea7:741/64 scope link 
           valid_lft forever preferred_lft forever
    3: ens224: <broadcast,multicast,up,lower_up>mtu 1500 qdisc mq state UP group default qlen 1000
        link/ether 00:50:56:8e:dd:58 brd ff:ff:ff:ff:ff:ff
        inet 10.42.0.65/24 brd 10.42.0.255 scope global ens224
           valid_lft forever preferred_lft forever
        inet6 fe80::250:56ff:fe8e:dd58/64 scope link 
           valid_lft forever preferred_lft forever</broadcast,multicast,up,lower_up></broadcast,multicast,up,lower_up> 
    



  • LAYER 8 Global Moderator

    0.0.0.0        XX.123.244.225  0.0.0.0        UG    0      0        0 ens192
    0.0.0.0        10.42.0.1      0.0.0.0        UG    203    0        0 ens224

    This is the problem.. you have 2 default routes.

    If you want your clients to talk via a private networks.. Then the only route these host would have is to the other network for that interface - it would not have a default gateway set on it.  You would just setup a specific route on the box.

    So for 10.42.0.64 he would have a route for 10.212.228/24 talk to 10.42.0.1 (pfsense).  This route is setup specifically on that host..

    Other client 10.212.228.64 would not have gateway set on this interface either and would have a route on itself for 10.42.0/24 talk to 10.212.228.1 (pfsense).  Again this route is on the 10.212.228.64 host



  • @johnpoz:

    0.0.0.0        XX.123.244.225  0.0.0.0        UG    0      0        0 ens192
    0.0.0.0        10.42.0.1      0.0.0.0        UG    203    0        0 ens224

    This is the problem.. you have 2 default routes.

    If you want your clients to talk via a private networks.. Then the only route these host would have is to the other network for that interface - it would not have a default gateway set on it.  You would just setup a specific route on the box.

    So for 10.42.0.64 he would have a route for 10.212.228/24 talk to 10.42.0.1 (pfsense).  This route is setup specifically on that host..

    Other client 10.212.228.64 would not have gateway set on this interface either and would have a route on itself for 10.42.0/24 talk to 10.212.228.1 (pfsense).  Again this route is on the 10.212.228.64 host

    John,

    Thank you for the information and yes, this fixed the issue.

    Rather then going to each host and defining a static route for 10.42.0.0/24 and 10.212.228.0/24 I ended up creating one for the entire 10.0.0.0/8 range to the respected gateways. This will allow me flexibility in the future if I add a new network or have to re-ip things. I certainly do not want to login to each host and have to update their respective routes when the network changes.

    Example:

    
    ip route add 10.0.0.0/8 via 10.212.228.1 dev ens224
    
    

    -Adam


  • LAYER 8 Global Moderator

    Sure that works.. Another solution would not to multi home you boxes like that.. Seems kind of pointless if you ask me..

    Also such a setup doesn't stop them from talking to each other on their other network..  Why would you not just put the clients behind pfsense for everything?

    What is the point of the multihomed setup?  That you want/need to firewall?


Log in to reply