Source based routing
-
Hello,
We're having a routing issue with a pfSense machine that we can't quite seem to figure out.
The pfSense machine has two WAN interfaces, one of which is the default gateway, the other an OpenVPN tunnel endpoint.
When there is an incoming connection from the VPN tunnel to a machine in the LAN segment, the return packets are sent back over the other WAN interface. (The default gateway.)
As an example and to clarify a little more, an ICMP echo request and reply (it is not limited to ICMP, this also happens with TCP SYN and SYN/ACK packets):
- External host A sends ICMP request over VPN tunnel
- ICMP request arrives at pfSense machine
- ICMP request is forwarded to host B in the LAN
- Host B in the LAN send ICMP reply
- ICMP reply arrives at pfSense machine
- pfSense sends reply out trough default gw WAN interface, not the VPN tunnel the request came from. (ignoring source based routing rules)
We've tried to correct this behavior with a firewall rule that states packets with machine in the LAN as source and the dest address that is on the other side of the VPN tunnel should use the VPN interface as the gateway, however this rule seems to be entirely ignored.
We've reproduced the same problem in a lab setup with the VPN tunnel omitted and just an extra WAN interface in it's place so we're sure it's not an issue with the OpenVPN tunnel.
Suspecting there may be something going on in the state table we added a rule to disable state tracking which resulted in the return packet being dropped entirely. We're not sure at this time if the state table is involved in the problem or that this is a side-effect.
Is this a bug in pf? Is there anyone else who is having this problem?
Please feel free to contact me for further details. -
looks like missing routes
-
looks like missing routes
The routing table is in order.
Either way the default route is out the gateway it's going out, so that makes sense. But the whole point was to have a firewall rule that overwrites the default route for a specific source/dest. -
well it does or should.
there are only a couple of reasons when this does not happen (that i know of):
-missing routes (return routes & most likely)
-some weird floating rules
-weird NATif you want more help, you gonna have to provide more info. (screenshots of vpn config/routing table/firewall rules/ .. // …)
-
Routing table (IPv6 routes removed for brevity):
Destination Gateway Flags Refs Use Mtu Netif Expire default 192.168.17.1 UGS 0 131244 1500 bge0_vlan5 10.9.0.0/24 link#9 U 0 3264752 1500 bge1_vlan992 10.9.0.1 link#9 UHS 0 0 16384 lo0 127.0.0.1 link#5 UH 0 164 16384 lo0 192.168.1.0/24 link#8 U 0 10659 1500 bge1_vlan991 192.168.1.1 link#8 UHS 0 0 16384 lo0 192.168.17.0/24 link#7 U 0 695238 1500 bge0_vlan5 192.168.17.52 link#7 UHS 0 0 16384 lo0firewall rules (IPv6 rules removed for brevity):
scrub on bge0_vlan5 all fragment reassemble scrub on bge1_vlan991 all fragment reassemble scrub on bge1_vlan992 all fragment reassemble anchor "relayd/*" all anchor "openvpn/*" all anchor "ipsec/*" all block drop in log inet all label "Default deny rule IPv4" block drop out log inet all label "Default deny rule IPv4" block drop quick inet proto tcp from any port = 0 to any block drop quick inet proto tcp from any to any port = 0 block drop quick inet proto udp from any port = 0 to any block drop quick inet proto udp from any to any port = 0 block drop quick from <snort2c>to any label "Block snort2c hosts" block drop quick from any to <snort2c>label "Block snort2c hosts" block drop in log quick proto tcp from <sshlockout>to (self) port = ssh label "sshlockout" block drop in log quick proto tcp from <webconfiguratorlockout>to (self) port = https label "webConfiguratorlockout" block drop in quick from <virusprot>to any label "virusprot overload table" block drop in on ! bge0_vlan5 inet from 192.168.17.0/24 to any block drop in inet from 192.168.17.52 to any block drop in on bge0_vlan5 inet6 from fe80::215:c5ff:fefa:27b to any pass in on bge0_vlan5 proto udp from any port = bootps to any port = bootpc keep state label "allow dhcp client out WAN" pass out on bge0_vlan5 proto udp from any port = bootpc to any port = bootps keep state label "allow dhcp client out WAN" block drop in on ! bge1_vlan991 inet from 192.168.1.0/24 to any block drop in inet from 192.168.1.1 to any block drop in on bge1_vlan991 inet6 from fe80::1:1 to any pass in quick on bge1_vlan991 inet proto udp from any port = bootpc to 255.255.255.255 port = bootps keep state label "allow access to DHCP server" pass in quick on bge1_vlan991 inet proto udp from any port = bootpc to 192.168.1.1 port = bootps keep state label "allow access to DHCP server" pass out quick on bge1_vlan991 inet proto udp from 192.168.1.1 port = bootps to any port = bootpc keep state label "allow access to DHCP server" block drop in on ! bge1_vlan992 inet from 10.9.0.0/24 to any block drop in inet from 10.9.0.1 to any pass in on lo0 inet all flags S/SA keep state label "pass IPv4 loopback" pass out on lo0 inet all flags S/SA keep state label "pass IPv4 loopback" pass out inet all flags S/SA keep state allow-opts label "let out anything IPv4 from firewall host itself" pass out route-to (bge0_vlan5 192.168.17.1) inet from 192.168.17.52 to ! 192.168.17.0/24 flags S/SA keep state allow-opts label "let out anything from firewall host itself" pass in quick on bge1_vlan991 proto tcp from any to (bge1_vlan991) port = https flags S/SA keep state label "anti-lockout rule" pass in quick on bge1_vlan991 proto tcp from any to (bge1_vlan991) port = http flags S/SA keep state label "anti-lockout rule" pass in quick on bge1_vlan991 proto tcp from any to (bge1_vlan991) port = ssh flags S/SA keep state label "anti-lockout rule" anchor "userrules/*" all pass in quick on bge0_vlan5 reply-to (bge0_vlan5 192.168.17.1) inet proto tcp from any to any port = https flags S/SA keep state label "USER_RULE" pass in quick on bge0_vlan5 reply-to (bge0_vlan5 192.168.17.1) inet proto tcp from any to any port = ssh flags S/SA keep state label "USER_RULE" pass in quick on bge0_vlan5 reply-to (bge0_vlan5 192.168.17.1) inet proto tcp from any to 192.168.1.1 port = ssh flags S/SA keep state label "USER_RULE: NAT " pass in quick on bge0_vlan5 reply-to (bge0_vlan5 192.168.17.1) inet proto tcp from any to 192.168.1.1 port = https flags S/SA keep state label "USER_RULE: NAT " pass in log quick on bge1_vlan991 route-to (bge1_vlan992 10.9.0.2) inet from 192.168.1.10 to 192.168.21.0/24 no state label "USER_RULE: return path for VM host" pass in quick on bge1_vlan991 inet from 192.168.1.0/24 to any flags S/SA keep state label "USER_RULE: Default allow LAN to any rule" pass in quick on bge1_vlan992 inet all flags S/SA keep state label "USER_RULE" anchor "tftp-proxy/*" all</virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c>192.168.17.52 is the WAN interface (VLAN tag 5) and is the default gateway. The ICMP reply packet goes out over this interface.
192.168.1.1 is the LAN interface (VLAN tag 991) The target machine is on this interface, The ICMP request and response to and from the client machine go over this interface.
10.9.0.1 is the OPT1 interface (VLAN tag 992) and is the second WAN interface in the test setup or the OpenVPN endpoint in the production setup. The initial ICMP request comes in on this interface. -
so where is the route for the (lan)-subnet at the other end of the tunnel?
without a return route, it'll allways send it over the default gateway, that's how every router on the planet works.
-
In your OpenVPN server and client, put the relevant networks in local network/s and remote network/s boxes. Then routes to them will appear in the relevant routing tables.
We've tried to correct this behavior with a firewall rule that states packets with machine in the LAN as source and the dest address that is on the other side of the VPN tunnel should use the VPN interface as the gateway, however this rule seems to be entirely ignored.
The first packet coming in across the OpenVPN to LAN will have created a state. The reply packet matches that state, so it gets handled as per the settings that apply to that state. The pf rule set is not re-evaluated for return packets. So that is why the rule you tried to make seems to be ignored - it only gets processed for the initial packet that ends up creating a state.
-
In your OpenVPN server and client, put the relevant networks in local network/s and remote network/s boxes. Then routes to them will appear in the relevant routing tables.
We've tried to correct this behavior with a firewall rule that states packets with machine in the LAN as source and the dest address that is on the other side of the VPN tunnel should use the VPN interface as the gateway, however this rule seems to be entirely ignored.
The first packet coming in across the OpenVPN to LAN will have created a state. The reply packet matches that state, so it gets handled as per the settings that apply to that state. The pf rule set is not re-evaluated for return packets. So that is why the rule you tried to make seems to be ignored - it only gets processed for the initial packet that ends up creating a state.
Would it make sense to write rules to disable state tracking for this source-dest pair in order to make it use the source routing rule in the firewall?
so where is the route for the (lan)-subnet at the other end of the tunnel?
without a return route, it'll allways send it over the default gateway, that's how every router on the planet works.
As far as I'm aware it is not possible to specify routes that only apply to a certain source (range), which is why we are using a firewall rule in the first place. Or is this something that does exist? (If so, what is it called in pfSense?)
-
routes and firewall rules are two seperate things.
routes are used to inform a device what "highway' to use to get to a certain destination.
rules are borderpatrol checkpointsyou can not do routing without routes. so filling in the necessary remote/local networks in your openvpn configuration is required.
afterwards you can use rules to decide who can/cannot use the newly discovered highway. -
routes and firewall rules are two seperate things.
routes are used to inform a device what "highway' to use to get to a certain destination.
rules are borderpatrol checkpointsyou can not do routing without routes. so filling in the necessary remote/local networks in your openvpn configuration is required.
afterwards you can use rules to decide who can/cannot use the newly discovered highway.It is possible to modify such behavior by means of the firewall rules (redirecting to a specific gateway etc.)
But my question remains, can routes apply to a specific source only? Since not all traffic towards these hosts needs to go over the 2nd interface, so routing by destination only is not enough.
Besides that, the problem is not limited to OpenVPN since we reproduced the problem without OpenVPN. (Though the real world application for limiting by source address may become somewhat odd then, but that is besides the point.) -
Perhaps to clarify the situation just a little bit:
There is a destination subnet that needs to be routed trough a specific gateway, this cannot change until certain legacy applications are completely phased out.
The newer applications need to connect to hosts in the same destination subnet but use a new gateway.The only way we can distinguish the old and new application traffic is by their source addresses.
This is why we looked at a firewall rule to redirect packets with a specific source and destination to a gateway that is not the normal gateway for that destination subnet. This works fine when the first packet comes from the side where the source based routing firewall rule is in effect but is entirely broken when the other side initiates the connection. Then the first return packet is sent over the default gateway instead. (As per the routing config rather than the firewall rule.)
I'm also entirely open to alternative solutions to the problem, but it feels intuitively feels like it should work both ways. But as explained earlier in this thread, apparently states throw a spanner in the works.