[How to] pfSense Selective Routing via VPN and WAN Interfaces

Skepticatpf · Feb 2, 2016, 8:01 PM

Summary of the posts below

Background: Certain scenarios call for routing some of your LAN IPs via WAN interface vs VPN one. Example: gaming PC to avoid latency, Hulu media player as Hulu blocks PIA.
Goal:
1. Route my Roku player only through WAN as Hulu blocks PIA IPs
2. Route my gaming PC only through WAN as VPN kills latency
3. Make sure that one specific LAN IP can only use PIAVPN interface and blocked if PIA is down
4. Every other devices should go through VPN.
Special thank you goes to: kesawi, Derelict
Resources: https://www.infotechwerx.com/blog/Policy-Routing-Certain-Traffic-Through-OpenVPN-Client-Connection

STEP 1
Set-up your PIA VPN by following this thread https://forum.pfsense.org/index.php?topic=76015.0

STEP 2
Route Hulu player and PC only through WAN as Hulu blocks PIA IPs

1.Make sure you assign a static IP address to your Hulu player and PC.
2.Create a firewall alias for your Hulu player and PC (makes it easier to look at your list of rules and see what's going on. If you have multiple rules referring to the same IP, it allows you to change it in one place rather than having to change multiple rules).
3.Create a LAN firewall rule for your Hulu/PC and (important) move it up to the top:


Action: Allow
TCP/IP Version: IPv4
Protocol: Any
Source: Single Host or Alias - Alias you created for your Hulu/PC
Destination: Any
Destination Port Range: Any
Gateway (Under advanced features): Select the WAN Gateway

STEP 3
Make sure that one specific LAN IP can only use PIAVPN interface and blocked if PIA is down
Important: This device on your LAN will not be able to access the Internet if PIAVPN interface goes down. Exactly what I need, but make sure you understand this.

1.Create an alias for this LAN IP (eg SERVER1)
2.You could create a rule similar to the above for the server specifying the PIAVPN as the gateway, but that probably isn't necessary as from what you've written your default route appears to be through the VPN. I don't think that there would be any harm creating the rule anyway, as it will ensure this IP will continue to use the VPN if you change your default gateway to the WAN interface.
3. Create the following LAN rule and (important) move it to the second top spot


Action: Allow
TCP/IP Version: IPv4
Protocol: Any
Source: Single Host or Alias - Alias you created for your SERVER1
Destination: Any
Destination Port Range: Any
Advanced: Advanced - Mark packets matching this rule: NO_WAN_EGRESS
Gateway (Under advanced features): Select the PIAVPN Gateway

STEP 4
Why it works: In the above rule you tagged certain traffic with "NO_WAN_EGRESS" label. This rule works because you tell it to look for that marker

Create a floating rule to prevent SERVER1 egress via WAN


Action: Reject
Quick: Checked
Interface: WAN (you can also select multiple WAN interfaces or an interface group here)
Direction: out
Protocol: any
Source: any
Destination: any
Description: Reject outbound traffic marked NO_WAN_EGRESS
Advanced: You can match packet on a mark placed before on another rule: NO_WAN_EGRESS

kesawi · Jan 26, 2016, 12:05 PM

1. Route my Roku player only through WAN as Hulu blocks PIA IPs

Make sure you assign a static IP address to your Roku player.
Create a firewall alias for your Roku player (this isn't essential, but using alias with firewall rules makes it easier to look at your list of rules and see what's going on. If you have multiple rules referring to the same IP, it allows you to change it in one place rather than having to change multiple rules).
Create a LAN firewall rule for your roku:

TCP/IP Version: IPv4
Protocol: Any
Source: Single Host or Alias - Alias you created for your Roku
Destination: Any
Destination Port Range: Any
Gateway (Under advanced features): Select the WAN Gateway

2. Route my gaming PC only through WAN as VPN kills latency

Do the same as above for your gaming PC

3. Make sure that one specific LAN IP can only use PIAVPN interface and blocked if PIA is down

Create an alias for this LAN IP (eg SERVER1)
You could create a rule similar to the above for the server specifying the PIAVPN as the gateway, but that probably isn't necessary as from what you've written your default route appears to be through the VPN. I don't think that there would be any harm creating the rule anyway, as it will ensure this IP will continue to use the VPN if you change your default gateway to the WAN interface.
Create the following floating rules:

Action: Match
Interface: LAN
Direction: in
TCP/IP Version: IPv4 + IPv6
Protocol: Any
Source: Single Host or Alias - SERVER1
Destination: any
Advanced Options: Mark packets matching this rule: NO_WAN_EGRESS

Action: Reject
Interface: WAN
Quick: Enabled
Direction: out
TCP/IP Version: IPv4 + IPv6
Protocol: Any
Source: Any
Destination: any
Advanced Options: Match packet on a mark: NO_WAN_EGRESS

johnpoz · Jan 26, 2016, 1:49 PM

keep in mind when you want to use policy routing your going to most likely want to block getting the default route from your vpn provider. then route your traffic you want to go out your vpn via gateway in the rules.

Rules that you want to allow to go to your other networks you put above the rule that uses the vpn gateway.

Skepticatpf · Jan 27, 2016, 12:38 AM

EDIT: PC is also going through VPN instead of WAN… Attaching the NAT rules screen shot.

Thank you guys for the response... As suggested I think I need to reorder the rules as my Roku is still trying to use VPN interface.

A question: in the second floating rule, shouldn't I specify source as SERVER1? Otherwise how would it know? Adjusted below...

@kesawi:

Action: Match
Interface: LAN
Direction: in
TCP/IP Version: IPv4 + IPv6
Protocol: Any
Source: Single Host or Alias - SERVER1
Destination: any
Advanced Options: Mark packets matching this rule: NO_WAN_EGRESS

Action: Reject
Interface: WAN
Quick: Enabled
Direction: out
TCP/IP Version: IPv4 + IPv6
Protocol: Any
Source: Single Host or Alias - SERVER1
Destination: any
Advanced Options: Match packet on a mark: NO_WAN_EGRESS

Derelict · Jan 27, 2016, 12:37 AM

If you are using NAT you can't match traffic by inside local source address using floating rules on WAN out because NAT has already happened and the source address is now your mapped NAT address.

That's why I use the whole NO_WAN_EGRESS scheme. Mark it on LAN in and let that rule prevent it from going out WAN.

The reject rule should be protocol any source any dest any matching on NO_WAN_EGRESS. You don't care what it is or where it came from and just want it blocked from going out the clear WAN if it's marked.

I am not a fan of using floating rules for things that can be done on interface rule tabs. For instance, mark the traffic NO_WAN_EGRESS on the LAN rule that routes the traffic out the VPN.

Skepticatpf · Jan 27, 2016, 12:51 AM

Apologies for being a dense noob. I set-up the NAT per tutorial posted here. With my set-up, you mind typing up a step by step? Does you comment apply to kesawi's post as additional rules or replacement?

Thank you!

@Derelict:

If you are using NAT you can't match traffic by inside local source address using floating rules on WAN out because NAT has already happened and the source address is now your mapped NAT address.

That's why I use the whole NO_WAN_EGRESS scheme. Mark it on LAN in and let that rule prevent it from going out WAN.

The reject rule should be protocol any source any dest any matching on NO_WAN_EGRESS. You don't care what it is or where it came from and just want it blocked from going out the clear WAN if it's marked.

I am not a fan of using floating rules for things that can be done on interface rule tabs. For instance, mark the traffic NO_WAN_EGRESS on the LAN rule that routes the traffic out the VPN.

Derelict · Jan 27, 2016, 12:57 AM

https://www.infotechwerx.com/blog/Policy-Routing-Certain-Traffic-Through-OpenVPN-Client-Connection

kesawi · Jan 27, 2016, 12:53 PM

@Derelict:

I am not a fan of using floating rules for things that can be done on interface rule tabs. For instance, mark the traffic NO_WAN_EGRESS on the LAN rule that routes the traffic out the VPN.

My preference for using a LAN floating rule to mark traffic is that the floating rule will be processed before the individual LAN rule. If I misconfigure the LAN rules or get the order of my LAN rules wrong, then traffic may not be marked and it could leak out of the WAN interface. At least with the LAN floating rule the traffic will always be marked and blocked by the WAN floating rule, even if I stuff up my LAN interface rules (at least that's my theory :)).

@Skepticatpf:

Apologies for being a dense noob. I set-up the NAT per tutorial posted here. With my set-up, you mind typing up a step by step? Does you comment apply to kesawi's post as additional rules or replacement?

They are replacement rules. If you are using the LAN floating rule (ie as per my post) then there is no need for another LAN interface rule to mark the packets from the host. If you are using a LAN interface rule to mark the packets as per Derelict's suggestion, then you don't need the LAN floating rule.

@Derelict:

https://www.infotechwerx.com/blog/Policy-Routing-Certain-Traffic-Through-OpenVPN-Client-Connection

Nice write up, should probably be a sticky pointing to it.

Skepticatpf · Jan 27, 2016, 2:33 PM

@kesawi:

They are replacement rules. If you are using the LAN floating rule (ie as per my post) then there is no need for another LAN interface rule to mark the packets from the host. If you are using a LAN interface rule to mark the packets as per Derelict's suggestion, then you don't need the LAN floating rule.

OK, would my NAT rules interfere with LAN floating rules? I assume I need to remove NAT for PIAVPN and do everything either from floating or LAN tabs. I want to try one clean set-up to make sure it works and I understand what is being done.

Derelict · Jan 27, 2016, 3:42 PM

NAT rules do not route or pass/reject/deny traffic.

NAT rules simply determine what happens to the source address/port if the router has determined traffic should be sent out the interface identified in the rule.

Skepticatpf · Jan 28, 2016, 6:58 PM

OK, I think I got the point of NAT not routing, but per your post it seems it would prevent my floating rule from working as it translated LAN IPs…

@Derelict:

If you are using NAT you can't match traffic by inside local source address using floating rules on WAN out because NAT has already happened and the source address is now your mapped NAT address.

Reference also tells me that NAT gets processed first:

https://doc.pfsense.org/index.php/Firewall_Rule_Processing_Order
More accurately, the following order (still simplified) is found in the ruleset (Check /tmp/rules.debug):

Outbound NAT rules
Inbound NAT rules such as Port Forwards (including rdr pass and UPnP)
NAT rules for the Load Balancing daemon (relayd)
Rules dynamically received from RADIUS for OpenVPN and IPsec clients
Internal automatic rules (pass and block for various items like lockout, snort, DHCP, etc.)
User-defined rules:
Rules defined on the floating tab
Rules defined on interface group tabs (Including OpenVPN)
Rules defined on interface tabs (WAN, LAN, OPTx, etc)
Automatic VPN rules

So, per my attached NAT screen shot, do I need to remove them if I to use kesawi's recommended floating rule?

This is the instructions PIA listing for NAT set-up:

At this point, automatic rules that were in use in Automatic outbound NAT will be generated. It will look like the following, with the exception that the Source IP subnet will match your LAN IP subnet.
https://www.privateinternetaccess.com/images/tutorials/PfSense4.png?7cd5ee465d9c63d0bcf7bc759a274547

The next step is to duplicate each of these rules, but change the NAT Address from WAN to PIAVPN. Start with the first rule by clicking the Plus sign immediately to the right of the line "to add a new NAT based on this one." A new page will open. Configure as follows:

https://www.privateinternetaccess.com/images/tutorials/PfSense6.png?126ce9963a0353a47776de24cd40b752

Derelict · Jan 28, 2016, 7:56 PM

Yes, NAT happens before OUTBOUND floating rules are processed so you cannot identify traffic by the real source IP address.

What's the fascination with floating rules? They are great when match rules, "unquick" rules, or rules going out an interface are needed but not for simple in rules. It just adds complexity and makes you look in two different places to see what's really going on.

You have a host on LAN. Put the rules on LAN that forward traffic from it wherever you want. If you never want it to go out WAN mark it on those rules. Then with ONE floating rule on WAN out protocol any source any dest any and matching the mark you can block all traffic that should not go out WAN from all interfaces everywhere (Except the firewall itself, which is a different issue). You never have to worry about NAT because rules coming IN an interface are pre-NAT.

Skepticatpf · Jan 28, 2016, 8:26 PM

@Derelict:

Yes, NAT happens before OUTBOUND floating rules are processed so you cannot identify traffic by the real source IP address.

Thank you for sticking with me! OK, this getting clear to me, I promise.

@Derelict:

What's the fascination with floating rules? They are great when match rules, "unquick" rules, or rules going out an interface are needed but not for simple in rules. It just adds complexity and makes you look in two different places to see what's really going on.

No emotional attachment what so ever: pfSense user of 4 days here. I want to avoid complexity as much as possible.

@Derelict:

You have a host on LAN. Put the rules on LAN that forward traffic from it wherever you want. If you never want it to go out WAN mark it on those rules. Then with ONE floating rule on WAN out protocol any source any dest any and matching the mark you can block all traffic that should not go out WAN from all interfaces everywhere (Except the firewall itself, which is a different issue). You never have to worry about NAT because rules coming IN an interface are pre-NAT.

OK, when I am home I will try to clean up and do this via LAN rules…

Skepticatpf · Feb 2, 2016, 12:46 AM

Success!

I accomplished this with two LAN rules, which I forgot to move to the top, duh. One for Hulu/PC routing to WAN, another for Server routing to PIAVPN. Marked the latter one as NO_WAN_EGRESS. Created then a floating rule.

Question: PIA has a few US servers. Can I create multiple interfaces and use them for failover? E.g. PIA1 US-EAST, PIA2 US-NY, if PIA1 goes down, pfSense will try to bring up PIA2.

Action: Reject
Quick: Checked
Interface: WAN (you can also select multiple WAN interfaces or an interface group here)
Direction: out
Protocol: any
Source: any
Destination: any
Description: Reject outbound traffic marked NO_WAN_EGRESS
Advanced: You can match packet on a mark placed before on another rule: NO_WAN_EGRESS