Policy based routing and VPN again



  • I have been reading so many articles and so many posts here about how to set up VPN and direct a set of LAN clients to use only that single VPN connection.

    I've tried the tag method, I've tried the 2 rule method; one where you enforce the gateway, and the one after it to reject in case the gateway goes down.

    I've tried articles from PIA, These Forums, this other one that was very education: https://www.infotechwerx.com/blog/Prevent-Any-Traffic-VPN-Hosts-Egressing-WAN

    I'm at my wits end, it has been 2 weeks, and about 12 hours of trying to figure this out.  Port Forwarding on the VPN link to the 2 LAN clients.  Currently outbound works on the 2 clients, they go out the correct VPN interface, their IP resolves, I am able to do speed tests.  But the Port forwarding is all sorts of wacked.  I originally configured my interfaces, but toyed with the OpenVPN rule set, not realizing that I had named my interface OPENVPN.  After this realization I removed my rules from the incorrect instance, renamed the interfaces, and Applied it to AirVPN.

    When I have an allow default on the OpenVPN interface, I see traffic hit my 2 clients, but the clients response goes out the WAN (igb2), no bueno.  If I remove the any to any rule on OpenVPN I get the AirVPN IP and the Redirected to IP on the AirVPN interface (ovpnc3)

    09:02:17.224896 IP 198.199.98.246.38628 > 10.4.10.131.58910: Flags [s], seq 3287419867, win 14600, options [mss 1288,sackOK,TS val 1799929115 ecr 0,nop,wscale 8], length 0
    09:02:17.224939 IP 198.199.98.246.38628 > 192.168.0.47.58910: Flags [s], seq 3287419867, win 14600, options [mss 1288,sackOK,TS val 1799929115 ecr 0,nop,wscale 8], length 0
    09:02:18.220811 IP 198.199.98.246.38628 > 10.4.10.131.58910: Flags [s], seq 3287419867, win 14600, options [mss 1288,sackOK,TS val 1799929365 ecr 0,nop,wscale 8], length 0
    09:02:18.220835 IP 198.199.98.246.38628 > 192.168.0.47.58910: Flags [s], seq 3287419867, win 14600, options [mss 1288,sackOK,TS val 1799929365 ecr 0,nop,wscale 8], length 0
    
    To me seems mind boggling if it was doing the translation and dropping the translation in AirVPN interfaces. I've toyed with floating rules, but since removed them, because I think it's dirty to apply rules specific to an interface on something so generic.
    
    If I change my default route to AirVPN through the System -> Routing, Selecting AirVPN and assign it as Default, magically everything is functional, but I don't want all my clients going over the VPN.
    
    I've posted my rules.debug for posterity, and edited masked my PUBLIC_IP
    
    [code]
    set optimization normal
    set limit states 1634000
    set limit src-nodes 1634000
    
    #System aliases
    
    loopback = "{ lo0 }"
    SPECTRUM = "{ igb2 }"
    LAN = "{ ix0 }"
    WORKVPN = "{ ovpnc2 }"
    AIRVPN = "{ ovpnc3 }"
    OpenVPN = "{ openvpn }"
    
    #SSH Lockout Table
    table <sshlockout> persist
    table <webconfiguratorlockout> persist
    #Snort tables
    table <snort2c>
    table <virusprot>
    table <bogons> persist file "/etc/bogons"
    table <negate_networks> 
    
    # User Aliases 
    revo_cameras = "{   8200  8016  10019 }"
    table <transmission> {   192.168.0.44  192.168.0.47 } 
    transmission = "<transmission>"
    Webconsole = "{   22  1337 }"
    
    # Gateways
    GWSPECTRUM_DHCP = " route-to ( igb2 70.125.128.1 ) "
    GWAirVPN = " route-to ( ovpnc3 10.4.10.131 ) "
    GWOPENVPN_DHCP = " route-to ( ovpnc2 10.0.130.1 ) "
    
    set loginterface ix0
    
    set skip on pfsync0
    
    scrub on $SPECTRUM all    fragment reassemble
    scrub on $LAN all    fragment reassemble
    scrub on $WORKVPN all    fragment reassemble
    scrub on $AIRVPN all    fragment reassemble
    
    no nat proto carp
    no rdr proto carp
    nat-anchor "natearly/*"
    nat-anchor "natrules/*"
    
    # Outbound NAT rules (manual)
    nat on $AIRVPN  from 192.168.0.0/23 to any -> 10.4.10.131/32 port 1024:65535  
    nat on $WORKVPN  from 192.168.0.0/23 to any -> 10.0.130.11/32 port 1024:65535  
    nat on $SPECTRUM  from 192.168.0.0/23 to any -> PUBLIC_IP/32 port 1024:65535  
    
    # Load balancing anchor
    rdr-anchor "relayd/*"
    # TFTP proxy
    rdr-anchor "tftp-proxy/*"
    # NAT Inbound Redirects
    rdr on ovpnc3 proto { tcp udp } from any to 10.4.10.131 port 57695 -> 192.168.0.44
    # Reflection redirects
    rdr on { ix0 openvpn } proto { tcp udp } from any to 10.4.10.131 port 57695 tag PFREFLECT -> 127.0.0.1 port 19000
    
    rdr on ovpnc3 proto { tcp udp } from any to 10.4.10.131 port 58910 -> 192.168.0.47
    # Reflection redirects
    rdr on { ix0 openvpn } proto { tcp udp } from any to 10.4.10.131 port 58910 tag PFREFLECT -> 127.0.0.1 port 19001
    
    rdr on ix0 proto tcp from any to (self) port 80 -> 192.168.0.1 port 8080
    # Reflection redirects
    rdr on openvpn proto tcp from any to (self) port 80 tag PFREFLECT -> 127.0.0.1 port 19002
    
    rdr on igb2 proto { tcp udp } from any to 70.125.154.54 port 34197 -> 192.168.1.183
    # Reflection redirects
    rdr on { ix0 openvpn } proto { tcp udp } from any to PUBLIC_IP port 34197 tag PFREFLECT -> 127.0.0.1 port 19003
    
    rdr on igb2 proto tcp from any to PUBLIC_IP port $revo_cameras -> 192.168.0.98
    # Reflection redirects
    rdr on { ix0 openvpn } proto tcp from any to PUBLIC_IP port 8200 tag PFREFLECT -> 127.0.0.1 port 19004
    rdr on { ix0 openvpn } proto tcp from any to PUBLIC_IP port 8016 tag PFREFLECT -> 127.0.0.1 port 19005
    rdr on { ix0 openvpn } proto tcp from any to PUBLIC_IP port 10019 tag PFREFLECT -> 127.0.0.1 port 19006
    
    rdr on igb2 proto { tcp udp } from any to PUBLIC_IP port 25565 -> 192.168.1.156
    # Reflection redirects
    rdr on { ix0 openvpn } proto { tcp udp } from any to PUBLIC_IP port 25565 tag PFREFLECT -> 127.0.0.1 port 19007
    
    # UPnPd rdr anchor
    rdr-anchor "miniupnpd"
    
    anchor "relayd/*"
    anchor "openvpn/*"
    anchor "ipsec/*"
    # Allow IPv6 on loopback
    pass in  quick on $loopback inet6 all tracker 1000000001 label "pass IPv6 loopback"
    pass out  quick on $loopback inet6 all tracker 1000000002 label "pass IPv6 loopback"
    # Block all IPv6
    block in log quick inet6 all tracker 1000000003 label "Block all IPv6"
    block out log quick inet6 all tracker 1000000004 label "Block all IPv6"
    # block IPv4 link-local. Per RFC 3927, link local "MUST NOT" be forwarded by a routing device,
    # and clients "MUST NOT" send such packets to a router. FreeBSD won't route 169.254./16, but
    # route-to can override that, causing problems such as in redmine #2073
    block in log quick from 169.254.0.0/16 to any tracker 1000000101 label "Block IPv4 link-local"
    block in log quick from any to 169.254.0.0/16 tracker 1000000102 label "Block IPv4 link-local"
    #---------------------------------------------------------------------------
    # default deny rules
    #---------------------------------------------------------------------------
    block in log inet all tracker 1000000103 label "Default deny rule IPv4"
    block out log inet all tracker 1000000104 label "Default deny rule IPv4"
    block in log inet6 all tracker 1000000105 label "Default deny rule IPv6"
    block out log inet6 all tracker 1000000106 label "Default deny rule IPv6"
    
    # IPv6 ICMP is not auxilary, it is required for operation
    # See man icmp6(4)
    # 1    unreach         Destination unreachable
    # 2    toobig          Packet too big
    # 128  echoreq         Echo service request
    # 129  echorep         Echo service reply
    # 133  routersol       Router solicitation
    # 134  routeradv       Router advertisement
    # 135  neighbrsol      Neighbor solicitation
    # 136  neighbradv      Neighbor advertisement
    pass  quick inet6 proto ipv6-icmp from any to any icmp6-type {1,2,135,136} tracker 1000000107 keep state
    
    # Allow only bare essential icmpv6 packets (NS, NA, and RA, echoreq, echorep)
    pass out  quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {129,133,134,135,136} tracker 1000000108 keep state
    pass out  quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {129,133,134,135,136} tracker 1000000109 keep state
    pass in  quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000110 keep state
    pass in  quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000111 keep state
    pass in  quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {128,133,134,135,136} tracker 1000000112 keep state
    
    # We use the mighty pf, we cannot be fooled.
    block log quick inet proto { tcp, udp } from any port = 0 to any tracker 1000000113 label "Block traffic from port 0"
    block log quick inet proto { tcp, udp } from any to any port = 0 tracker 1000000114 label "Block traffic to port 0"
    block log quick inet6 proto { tcp, udp } from any port = 0 to any tracker 1000000115 label "Block traffic from port 0"
    block log quick inet6 proto { tcp, udp } from any to any port = 0 tracker 1000000116 label "Block traffic to port 0"
    
    # Snort package
    block log quick from <snort2c> to any tracker 1000000117 label "Block snort2c hosts"
    block log quick from any to <snort2c> tracker 1000000118 label "Block snort2c hosts"
    
    # SSH lockout
    block in log quick proto tcp from <sshlockout> to (self) port 22 tracker 1000000301 label "sshlockout"
    
    # webConfigurator lockout
    block in log quick proto tcp from <webconfiguratorlockout> to (self) port 1337 tracker 1000000351 label "webConfiguratorlockout"
    block in log quick from <virusprot> to any tracker 1000000400 label "virusprot overload table"
    antispoof log for $SPECTRUM tracker 1000001570
    # allow our DHCP client out to the SPECTRUM
    pass in  on $SPECTRUM proto udp from any port = 67 to any port = 68 tracker 1000001591 label "allow dhcp client out SPECTRUM"
    pass out  on $SPECTRUM proto udp from any port = 68 to any port = 67 tracker 1000001592 label "allow dhcp client out SPECTRUM"
    # Not installing DHCP server firewall rules for SPECTRUM which is configured for DHCP.
    antispoof log for $LAN tracker 1000002620
    # allow access to DHCP server on LAN
    pass in  quick on $LAN proto udp from any port = 68 to 255.255.255.255 port = 67 tracker 1000002641 label "allow access to DHCP server"
    pass in  quick on $LAN proto udp from any port = 68 to 192.168.0.1 port = 67 tracker 1000002642 label "allow access to DHCP server"
    pass out  quick on $LAN proto udp from 192.168.0.1 port = 67 to any port = 68 tracker 1000002643 label "allow access to DHCP server"
    antispoof log for $WORKVPN tracker 1000003670
    antispoof log for $AIRVPN tracker 1000004720
    
    # loopback
    pass in  on $loopback inet all tracker 1000005811 label "pass IPv4 loopback"
    pass out  on $loopback inet all tracker 1000005812 label "pass IPv4 loopback"
    pass in  on $loopback inet6 all tracker 1000005813 label "pass IPv6 loopback"
    pass out  on $loopback inet6 all tracker 1000005814 label "pass IPv6 loopback"
    # let out anything from the firewall host itself and decrypted IPsec traffic
    pass out  inet all keep state allow-opts tracker 1000005815 label "let out anything IPv4 from firewall host itself"
    pass out  inet6 all keep state allow-opts tracker 1000005816 label "let out anything IPv6 from firewall host itself"
    pass out  route-to ( igb2 70.125.128.1 ) from PUBLIC_IP to !70.125.128.0/19 tracker 1000005911 keep state allow-opts label "let out anything from firewall host itself"
    pass out  route-to ( ovpnc2 10.0.130.1 ) from 10.0.130.11 to !10.0.130.11/32 tracker 1000005912 keep state allow-opts label "let out anything from firewall host itself"
    pass out  route-to ( ovpnc3 10.4.10.131 ) from 10.4.10.131 to !10.4.0.0/16 tracker 1000005913 keep state allow-opts label "let out anything from firewall host itself"
    # make sure the user cannot lock himself out of the webConfigurator or SSH
    pass in  quick on ix0 proto tcp from any to (ix0) port { 1337 22 } tracker 10000 keep state label "anti-lockout rule"
    # NAT Reflection rules
    pass in  inet tagged PFREFLECT tracker 1000006231 keep state label "NAT REFLECT: Allow traffic to localhost"
    
    # User-defined rules follow
    
    anchor "userrules/*"
    # array key "enc0" does not exist for "" in array: {SPECTRUM LAN WORKVPN AIRVPN OpenVPN } label "USER_RULE"
    pass  in  quick  on $SPECTRUM reply-to ( igb2 70.125.128.1 ) inet proto icmp  from any to any tracker 1485194335 keep state  label "USER_RULE: Allow ping"
    pass  in  quick  on $SPECTRUM reply-to ( igb2 70.125.128.1 ) inet proto tcp  from 216.69.255.55 to PUBLIC_IP port $Webconsole tracker 1435184149 flags S/SA keep state  label "USER_RULE: Remote Admin"
    pass  in  quick  on $SPECTRUM reply-to ( igb2 70.125.128.1 ) inet proto tcp  from any to 192.168.0.98 port $revo_cameras tracker 1460580583 flags S/SA keep state  label "USER_RULE: NAT Revo Cameras"
    pass  in  quick  on $SPECTRUM reply-to ( igb2 70.125.128.1 ) inet proto { tcp udp }  from any to 192.168.1.156 port 25565 tracker 1460580584 keep state  label "USER_RULE: NAT  Minecraft"
    pass  in  quick  on $SPECTRUM reply-to ( igb2 70.125.128.1 ) inet proto { tcp udp }  from any to 192.168.1.183 port 34197 tracker 1466545887 keep state  label "USER_RULE: NAT Factorio headless"
    pass  in  quick  on $LAN inet from $transmission  to <negate_networks> tracker 10000001 keep state  label "NEGATE_ROUTE: Negate policy routing for destination"
    pass  in  quick  on $LAN  $GWAirVPN inet from $transmission to any tracker 1493862877 keep state  label "USER_RULE"
    block  in log  quick  on $LAN inet from $transmission to any tracker 1493864751  label "USER_RULE"
    pass  in  quick  on $LAN inet proto tcp  from any to 192.168.0.1 port 8080 tracker 1478203523 flags S/SA keep state  label "USER_RULE: NAT "
    pass  in  quick  on $LAN inet from any to any tracker 0100000101 keep state  label "USER_RULE: Default allow LAN to any rule"
    pass  in  quick  on $AIRVPN  $GWAirVPN inet proto { tcp udp }  from any to 192.168.0.47 port 58910 tracker 1493878657 keep state  label "USER_RULE: NAT Transmission The Mixing Bowl"
    pass  in  quick  on $AIRVPN  $GWAirVPN inet proto { tcp udp }  from any to 192.168.0.44 port 57695 tracker 1493878050 keep state  label "USER_RULE: NAT Transmission Radarr/Sonarr"
    pass  in  quick  on $AIRVPN reply-to ( ovpnc3 10.4.10.131 ) inet from any to any tracker 1493886673 keep state  label "USER_RULE"
    
    # VPN Rules
    
    anchor "tftp-proxy/*"
    anchor "miniupnpd"
    [/code]
    
    Any possible insight on why/how I've messed this up?
    
    I'm on 2.3.4[/s][/s][/s][/s]</negate_networks></virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c></transmission></transmission></negate_networks></bogons></virusprot></snort2c></webconfiguratorlockout></sshlockout>
    

  • Banned

    All you do is set the gateway in advanced settings on your firewall rules.



  • That's what I thought, but what about port forwarding, everything coming in through the vpn is trying to go back out the WAN, but the device can ping out through the VPN without a problem, all traffic originating from the device is going through the VPN tunnel as expected, but if the traffic is initiated from VPN, it tries to go back out the WAN.

    There are no entries in the negate_networks.

    In the rules.debug gateways are set, and assigned to my two rules in in the $LAN, and the 2 rules:  Everything for the 2 devices is configured for going out the vpn.  Ideas?

    
    pass  in  quick  on $LAN  $GWAirVPN inet from $transmission to any tracker 1493862877 keep state  label "USER_RULE"
    block  in log  quick  on $LAN inet from $transmission to any tracker 1493864751  label "USER_RULE"
    pass  in  quick  on $LAN inet proto tcp  from any to 192.168.0.1 port 8080 tracker 1478203523 flags S/SA keep state  label "USER_RULE: NAT "
    pass  in  quick  on $LAN inet from any to any tracker 0100000101 keep state  label "USER_RULE: Default allow LAN to any rule"
    pass  in  quick  on $AIRVPN  $GWAirVPN inet proto { tcp udp }  from any to 192.168.0.47 port 58910 tracker 1493878657 keep state  label "USER_RULE: NAT Transmission The Mixing Bowl"
    pass  in  quick  on $AIRVPN  $GWAirVPN inet proto { tcp udp }  from any to 192.168.0.44 port 57695 tracker 1493878050 keep state  label "USER_RULE: NAT Transmission Radarr/Sonarr"
    pass  in  quick  on $AIRVPN reply-to ( ovpnc3 10.4.10.131 ) inet from any to any tracker 1493886673 keep state  label "USER_RULE"
    
    






  • tl;dr.

    To me, best option is to create an alias with machine's ips that should use VPN. Then create the rules in lan interface and set gateway on that rules. Take care about the order. Don't forget that as soon as a rule matches, it will be applied.

    Connections initiated from outside, will not pass to the network as long as you don't nat them. Connections initiated from inside will go thru that gateway as long as it's initiated from one ip in that alias.


  • Rebel Alliance Global Moderator

    "everything coming in through the vpn is trying to go back out the WAN"

    Well then sounds like your issue is on the remote side..



  • @Hugovsky:

    tl;dr.

    To me, best option is to create an alias with machine's ips that should use VPN. Then create the rules in lan interface and set gateway on that rules. Take care about the order. Don't forget that as soon as a rule matches, it will be applied.

    Connections initiated from outside, will not pass to the network as long as you don't nat them. Connections initiated from inside will go thru that gateway as long as it's initiated from one ip in that alias.

    Word for word, my config says that's what I'm doing

    pass  in  quick  on $LAN  $GWAirVPN inet from $transmission to any tracker 1493862877 keep state  label "USER_RULE"
    

    "$transmission" is myAlias table list of hosts that are supposed to use the "$GWAirVPN" gateway on "$LAN" as the FIRST rule.

    When someone says

    To me, best option is to create an alias with machine's ips that should use VPN. Then create the rules in lan interface and set gateway on that rules. Take care about the order. Don't forget that as soon as a rule matches, it will be applied.

    Am I not doing that?  I'm confused, as I thought that's what I was doing.  I must be missing something.

    @johnpoz:

    "everything coming in through the vpn is trying to go back out the WAN"

    Well then sounds like your issue is on the remote side..

    How could this be on the Remote side?  Just like the other topic with a similar issue, this is a VPN service.  If I set my default gateway to that of the VPN service, all port forwarding responses and initiated traffic goes through the VPN as one would assume should work.



  • @CuteBoi:

    @Hugovsky:

    tl;dr.

    To me, best option is to create an alias with machine's ips that should use VPN. Then create the rules in lan interface and set gateway on that rules. Take care about the order. Don't forget that as soon as a rule matches, it will be applied.

    Connections initiated from outside, will not pass to the network as long as you don't nat them. Connections initiated from inside will go thru that gateway as long as it's initiated from one ip in that alias.

    Word for word, my config says that's what I'm doing

    pass  in  quick  on $LAN  $GWAirVPN inet from $transmission to any tracker 1493862877 keep state  label "USER_RULE"
    

    "$transmission" is myAlias table list of hosts that are supposed to use the "$GWAirVPN" gateway on "$LAN" as the FIRST rule.

    When someone says

    To me, best option is to create an alias with machine's ips that should use VPN. Then create the rules in lan interface and set gateway on that rules. Take care about the order. Don't forget that as soon as a rule matches, it will be applied.

    Am I not doing that?  I'm confused, as I thought that's what I was doing.  I must be missing something.

    Well… to be fair, I did put tl;dr there didn't I?  ;)

    Can you post the GUI rule?



  • that Tl;dr was too far up, I typically skip to the bottom to see if it was there. bleh.

    ![Screenshot from 2017-05-26 09-07-54.png](/public/imported_attachments/1/Screenshot from 2017-05-26 09-07-54.png)
    ![Screenshot from 2017-05-26 09-07-54.png_thumb](/public/imported_attachments/1/Screenshot from 2017-05-26 09-07-54.png_thumb)



  • Seems ok to me. Do you have NAT rules (port forward to inside ip) also?



  • Added an allow rule to see if that would help, not like the results make sense anyways.

    ![Screenshot from 2017-05-26 09-48-46.png](/public/imported_attachments/1/Screenshot from 2017-05-26 09-48-46.png)
    ![Screenshot from 2017-05-26 09-48-46.png_thumb](/public/imported_attachments/1/Screenshot from 2017-05-26 09-48-46.png_thumb)


  • Banned

    Post a pic of your NAT rules.



  • Simple stuff

    ![Screenshot from 2017-05-26 13-11-18.png](/public/imported_attachments/1/Screenshot from 2017-05-26 13-11-18.png)
    ![Screenshot from 2017-05-26 13-11-18.png_thumb](/public/imported_attachments/1/Screenshot from 2017-05-26 13-11-18.png_thumb)


  • Banned

    I'm no pro, and maybe it's different for your config. But based on my working config, I think the "simple stuff" is where you messed up.

    What about localhost?

    My outbound NAT looks more like this.

    VPN 127.0.0.0/8 * * 500 VPN Address * static=yes

    VPN 127.0.0.0/8 * * * VPN Address * static=no

    VPN (subnet a,b,c, etc) * * 500 VPN Address * static=yes

    VPN (subnet a,b,c, etc) * * * VPN Address * static=no

    That's with Hybrid outbound.



  • I checked this, added in the nat rule.

    Still no work.  Nat rules are working as far as I can tell.

    ![Screenshot from 2017-05-26 14-54-48.png](/public/imported_attachments/1/Screenshot from 2017-05-26 14-54-48.png)
    ![Screenshot from 2017-05-26 14-54-48.png_thumb](/public/imported_attachments/1/Screenshot from 2017-05-26 14-54-48.png_thumb)



  • For giggles, I added in a am AIRVPN_TAG parameter on the port forwarded rules in the AIRVPN ruleset, and a floating rule of "any any any with tagged AIRVPN_TAG" to set the gateway to GW AIRVPN, and still, traffic is trying to go out the WAN.