Source Based Routing With FreeBSD Using Multiple Routing Tables
-
An interesting article about the setfib feature:
http://www.mmacleod.ca/blog/2011/06/source-based-routing-with-freebsd-using-multiple-routing-table/
Source Based Routing With FreeBSD Using Multiple Routing Tables
MIKE MACLEOD
Jun 21, 2011Something has been bugging me for several years now. In that time I have usually had access to multiple WAN connections, owing to my participation in the telecom industry. However, I’ve never been able to get SSHD to behave the way I wanted it to. I wanted to be able to connect to the SSH daemon on my (FreBSD) router from whichever WAN connection I wanted. Unfortunately, SSHD is stuborn about always routing its response to the default gateway of the router, which breaks an SSH connection coming in from the secondary WAN connection.
I have finally, at long last, found the solution.
The Problem: When describing this problem to other FreeBSD users, they pretty universally assumed that I was mistaken, and that the SSH daemon would by default route its responses out the interface that the initial request had been received on. Evidently, it is uncommon to run SSHD on more than one WAN interface. At its root, this problem just boils down to the routing table. SSHD doesn’t route it’s responses to the interface that they were received on, it uses the routing table to determine where to send it’s responses. If the request came from a local network, then it responds to the local network. If it originated from a non-local network, then it uses the default gateway. It’s really that simple. There is no logic for having multiple paths to non-local networks.
Frequently Offered Solution: Since I use pf for my firewall, I’m frequently told to use pf’s route-to and reply-to functionality to solve this problem. I have at times used route-to and reply-to extensively in my pf.conf. But route-to and reply-to do not trump the default routing table for traffic the originates or terminates on the router itself. They are useful only for traffic passing through the router. pf can only make routing decisions when a packet passes through an interface. It can try and set the reply-to interface to be the second WAN connection when an inbound SSH connection is made, but neither the SSH daemon nor the routing table on the host know or care about the routing preferences of pf.
The Real Solution: FreeBSD has support for multiple routing tables. It’s little known, and even less documented, but it does exist. Basically, you need to recompile your kernel with multiple routing table support (“options ROUTETABLES=2″), and then use the setfib program to set which routing table to use when starting another program. The syntax is similar to nice: setfib 1 route add default 192.168.1.1 would add a default route of 192.168.1.1 to the second routing table on the host. If not specified, the default routing table is 0. On FreeBSD, pf also has support for multiple routing tables with the little discussed rtable option.
So here are the steps to solving this problem:
Step 1: Rebuild your kernel with the ROUTETABLES option set to a non-zero integer. This is how many routing tables your host will support.
[root@router]~ $ cat /root/kernels/ROUTER | grep ROUTETABLES
options ROUTETABLES=6Step 2: Add rtable awareness to your pf.conf file:
[root@router]~ $ cat /etc/pf.conf | grep rtable
pass in log on tun0 inet proto icmp from any to (tun0) icmp-type rtable 0
pass in log on tun1 inet proto icmp from any to (tun1) icmp-type rtable 1
pass in log on tun0 inet proto tcp from any to (tun0) port ssh rtable 0
pass in log on tun1 inet proto tcp from any to (tun1) port ssh rtable 1
pass in log on em0 inet proto tcp from em0:network to (em0) port 22 rtable 0Step 3: Disable the SSH daemon in your rc.conf:
[root@router]~ $ cat /etc/rc.conf | grep ssh
#sshd_enable="YES" # This is now handled by /etc/rc.localStep 4: Create /etc/rc.local file to start multiple SSH daemons. To do this, copy the /etc/ssh/sshd_config file to several alternates, one per interface you want SSHD to listen to, and set the ListenAddress for each file to only the IP for that interface.
[root@router]~ $ cat /etc/rc.local
/etc/rc.local
Build my alternate routing tables
/usr/sbin/setfib 0 /sbin/route add default 20.0.0.1
/usr/sbin/setfib 1 /sbin/route add default 30.0.0.1Start SSH daemons for each interface
/usr/sbin/setfib 0 /usr/sbin/sshd -f /etc/ssh/sshd_config.lan
/usr/sbin/setfib 0 /usr/sbin/sshd -f /etc/ssh/sshd_config.tun0
/usr/sbin/setfib 1 /usr/sbin/sshd -f /etc/ssh/sshd_config.tun1Conclusion: Because the SSH daemon listening on tun1 is using a routing table that features the tun1 interface as the default gateway, the response will go out tun1. An inbound connection to tun0 will hit the SSH daemon listening on tun0 (which is an entirely separate process from the one listening on tun1) and uses the routing table associated with tun0, which features tun0 as the default gateway.
In my above config it’s worth pointing out that it doesn’t actually matter which routing table the SSH daemon listening on the LAN interface uses, because both routing tables see the LAN network as a local one. By default on FreeBSD with multiple routing tables enabled, all local networks will still appear in all the routing tables. There is a sysctl option to disable this behavior.
-
I've done some pretty fun things with setfib, but it's best to avoid it wherever possible IMO. It's not really a clean/neat solution, though sometimes it is necessary.
The main thing I've done with it is make a jail live in a different subnet and use a different default gateway than the main host.
-
When reading it, I wondered whether setfib could be used for the problem of traffic initiated by daemons running on pfsense itself:
Due to the way IPsec tunnels are kludged into the FreeBSD kernel, any traffic initiated by pfsense to go through an IPsec tunnel gets the wrong source IP (and typically doesn't go through the tunnel at all as a result).
http://doc.pfsense.org/index.php/Why_can't_I_query_SNMP,_use_syslog,_NTP,_or_other_services_initiated_by_the_firewall_itself_over_IPsec_VPN%3F
PS: BTW in the area of policy based routing stock FreeBSD seems to be less capable/flexible than Linux's iproute2.
-
It wouldn't help with that really, since that would break them trying to reach anything else to other subnets. The static route is the best way there.
