Asymmetric routing problem? pictures and config attached



  • After initialy haveing some problems getting my VPN tunnel to work, it's now OK. But looking over the Firewall logs I see signs of asymmetric routing issues, which I can't seem to find the solution for, so I'm reaching out to you guys for some input and hopefully an helping hand.

    I tried looking over various forum threads and other sites for a possible solution, but I can't seem to find the issue.

    Logs:

    The entry comeing in at VPN2VLAN where Destination is 10.0.10.2:59990 is me checking if port is open from an external source.

    I've attached images of my configuration along with the entire config of the router

    Network setup (the WAN/LAN VLAN setup is due to 1 physical NIC on the PFSENSE machine)

    Interfaces - (assign)

    Interfaces - (assign) - VLANs

    Firewall - NAT - Port forwards

    Firewall - NAT - Outbound NAT

    Firewall - Rules - Floating (reject marked packets, they get marked in the rule @ VPN2VLAN)

    Firewall - Rules - WAN

    Firewall - Rules - LAN

    Firewall - Rules - VPN_WAN

    Firewall - Rules - VPN2VLAN

    Firewall - Rules - OpenVPN (this is rules for a OpenVPN server)

    And finaly, the config file:

    set optimization normal
    set limit states 1623000
    set limit src-nodes 1623000
    
    #System aliases
    
    loopback = "{ lo0 }"
    WAN = "{ em0_vlan10 }"
    LAN = "{ em0_vlan20 }"
    VPN_WAN = "{ ovpnc2 }"
    VPN2VLAN = "{ em0_vlan30 }"
    OpenVPN = "{ openvpn }"
    
    #SSH Lockout Table
    table <sshlockout>persist
    table <webconfiguratorlockout>persist
    #Snort tables
    table <snort2c>table <virusprot>table <bogons>persist file "/etc/bogons"
    table <bogonsv6>persist file "/etc/bogonsv6"
    table <vpn_networks>{ 192.168.2.0/24 }
    table <negate_networks>{ 192.168.2.0/24 }
    
    # User Aliases
    
    # Gateways
    GWWAN_DHCP = " route-to ( em0_vlan10 xxx.xxx.234.1 ) "
    GWVPN_WAN_VPNV4 = " route-to ( ovpnc2 xxx.xxx.42.111 ) "
    GWWAN_DHCP6 = " route-to ( em0_vlan10 xxx.xxx.234.1 ) "
    GWVPN_WAN_VPNV6 = " route-to ( ovpnc2 xxx.xxx.42.111 ) "
    
    set loginterface em0_vlan20
    
    set skip on pfsync0
    
    scrub on $WAN all    fragment reassemble
    scrub on $LAN all    fragment reassemble
    scrub on $VPN_WAN all    fragment reassemble
    scrub on $VPN2VLAN all    fragment reassemble
    
    no nat proto carp
    no rdr proto carp
    nat-anchor "natearly/*"
    nat-anchor "natrules/*"
    
    # Outbound NAT rules (manual)
    nat on $WAN  from 127.0.0.0/8 to any port 500 -> xxx.xxx.235.198/32  static-port
    nat on $WAN  from 127.0.0.0/8 to any -> xxx.xxx.235.198/32 port 1024:65535
    nat on $WAN  from 192.168.1.0/24 to any port 500 -> xxx.xxx.235.198/32  static-port
    nat on $WAN  from 192.168.1.0/24 to any -> xxx.xxx.235.198/32 port 1024:65535
    nat on $WAN  from 192.168.2.0/24 to any port 500 -> xxx.xxx.235.198/32  static-port
    nat on $WAN  from 192.168.2.0/24 to any -> xxx.xxx.235.198/32 port 1024:65535
    nat on $VPN_WAN  from 10.0.10.0/24 to any port 500 -> xxx.xxx.42.111/32  static-port
    nat on $VPN_WAN  from 10.0.10.0/24 to any -> xxx.xxx.42.111/32 port 1024:65535
    
    # Load balancing anchor
    rdr-anchor "relayd/*"
    # TFTP proxy
    rdr-anchor "tftp-proxy/*"
    # NAT Inbound Redirects
    rdr on em0_vlan10 proto tcp from any to xxx.xxx.235.198 port 80 -> 192.168.1.11
    # Reflection redirect
    rdr on { em0_vlan20 em0_vlan30 openvpn } proto tcp from any to xxx.xxx.235.198 port 80 -> 192.168.1.11
    no nat on em0_vlan20 proto tcp from em0_vlan20 to 192.168.1.11 port 80
    nat on em0_vlan20 proto tcp from 192.168.1.0/24 to 192.168.1.11 port 80 -> 192.168.1.1 port 1024:65535
    
    rdr on em0_vlan10 proto tcp from any to xxx.xxx.235.198 port 443 -> 192.168.1.11
    # Reflection redirect
    rdr on { em0_vlan20 em0_vlan30 openvpn } proto tcp from any to xxx.xxx.235.198 port 443 -> 192.168.1.11
    no nat on em0_vlan20 proto tcp from em0_vlan20 to 192.168.1.11 port 443
    nat on em0_vlan20 proto tcp from 192.168.1.0/24 to 192.168.1.11 port 443 -> 192.168.1.1 port 1024:65535
    
    rdr on em0_vlan10 proto tcp from any to xxx.xxx.235.198 port 32400 -> 192.168.1.10
    # Reflection redirect
    rdr on { em0_vlan20 em0_vlan30 openvpn } proto tcp from any to xxx.xxx.235.198 port 32400 -> 192.168.1.10
    no nat on em0_vlan20 proto tcp from em0_vlan20 to 192.168.1.10 port 32400
    nat on em0_vlan20 proto tcp from 192.168.1.0/24 to 192.168.1.10 port 32400 -> 192.168.1.1 port 1024:65535
    
    rdr on ovpnc2 proto { tcp udp } from any to xxx.xxx.42.111 port 59990:60000 -> 10.0.10.2
    # Reflection redirect
    rdr on { em0_vlan20 em0_vlan30 openvpn } proto { tcp udp } from any to xxx.xxx.42.111 port 59990:60000 -> 10.0.10.2
    no nat on em0_vlan30 proto { tcp udp } from em0_vlan30 to 10.0.10.2 port 59990:60000
    nat on em0_vlan30 proto { tcp udp } from 10.0.10.0/24 to 10.0.10.2 port 59990:60000 -> 10.0.10.1 port 1024:65535
    
    # UPnPd rdr anchor
    rdr-anchor "miniupnpd"
    
    anchor "relayd/*"
    anchor "openvpn/*"
    anchor "ipsec/*"
    # block IPv4 link-local. Per RFC 3927, link local "MUST NOT" be forwarded by a routing device,
    # and clients "MUST NOT" send such packets to a router. FreeBSD won't route 169.254./16, but
    # route-to can override that, causing problems such as in redmine #2073
    block in log quick from 169.254.0.0/16 to any tracker 1000000101 label "Block IPv4 link-local"
    block in log quick from any to 169.254.0.0/16 tracker 1000000102 label "Block IPv4 link-local"
    #---------------------------------------------------------------------------
    # default deny rules
    #---------------------------------------------------------------------------
    block in log inet all tracker 1000000103 label "Default deny rule IPv4"
    block out log inet all tracker 1000000104 label "Default deny rule IPv4"
    block in log inet6 all tracker 1000000105 label "Default deny rule IPv6"
    block out log inet6 all tracker 1000000106 label "Default deny rule IPv6"
    
    # IPv6 ICMP is not auxilary, it is required for operation
    # See man icmp6(4)
    # 1    unreach         Destination unreachable
    # 2    toobig          Packet too big
    # 128  echoreq         Echo service request
    # 129  echorep         Echo service reply
    # 133  routersol       Router solicitation
    # 134  routeradv       Router advertisement
    # 135  neighbrsol      Neighbor solicitation
    # 136  neighbradv      Neighbor advertisement
    pass log quick inet6 proto ipv6-icmp from any to any icmp6-type {1,2,135,136} tracker 1000000107 keep state
    
    # Allow only bare essential icmpv6 packets (NS, NA, and RA, echoreq, echorep)
    pass out log quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {129,133,134,135,136} tracker 1000000108 keep state
    pass out log quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {129,133,134,135,136} tracker 1000000109 keep state
    pass in log quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000110 keep state
    pass in log quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000111 keep state
    pass in log quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {128,133,134,135,136} tracker 1000000112 keep state
    
    # We use the mighty pf, we cannot be fooled.
    block log quick inet proto { tcp, udp } from any port = 0 to any tracker 1000000113 label "Block traffic from port 0"
    block log quick inet proto { tcp, udp } from any to any port = 0 tracker 1000000114 label "Block traffic to port 0"
    block log quick inet6 proto { tcp, udp } from any port = 0 to any tracker 1000000115 label "Block traffic from port 0"
    block log quick inet6 proto { tcp, udp } from any to any port = 0 tracker 1000000116 label "Block traffic to port 0"
    
    # Snort package
    block log quick from <snort2c>to any tracker 1000000117 label "Block snort2c hosts"
    block log quick from any to <snort2c>tracker 1000000118 label "Block snort2c hosts"
    
    # SSH lockout
    block in log quick proto tcp from <sshlockout>to (self) port 22 tracker 1000000301 label "sshlockout"
    
    # webConfigurator lockout
    block in log quick proto tcp from <webconfiguratorlockout>to (self) port 80 tracker 1000000351 label "webConfiguratorlockout"
    block in log quick from <virusprot>to any tracker 1000000400 label "virusprot overload table"
    # allow our DHCPv6 client out to the WAN
    pass in log quick on $WAN proto udp from fe80::/10 port = 546 to fe80::/10 port = 546 tracker 1000000561 label "allow dhcpv6 client in WAN"
    pass in log quick on $WAN proto udp from any port = 547 to any port = 546 tracker 1000000562 label "allow dhcpv6 client in WAN"
    pass out log quick on $WAN proto udp from any port = 546 to any port = 547 tracker 1000000563 label "allow dhcpv6 client out WAN"
    # block bogon networks (IPv4)
    # http://www.cymru.com/Documents/bogon-bn-nonagg.txt
    block in  quick on $WAN from <bogons>to any tracker 11000 label "block bogon IPv4 networks from WAN"
    # block bogon networks (IPv6)
    # http://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt
    block in  quick on $WAN from <bogonsv6>to any tracker 11000 label "block bogon IPv6 networks from WAN"
    antispoof log for $WAN tracker 1000001570
    # block anything from private networks on interfaces with the option set
    block in  quick on $WAN from 10.0.0.0/8 to any tracker 12000 label "Block private networks from WAN block 10/8"
    block in  quick on $WAN from 127.0.0.0/8 to any tracker 12000 label "Block private networks from WAN block 127/8"
    b[b]lock in  quick on $WAN from 172.16.0.0/12 to any tracker 12000 label "Block private networks from WAN block 172.16/12"
    block in  quick on $WAN from 192.168.0.0/16 to any tracker 12000 label "Block private networks from WAN block 192.168/16"
    block in  quick on $WAN from fc00::/7 to any tracker 12000 label "Block ULA networks from WAN block fc00::/7"
    # allow our DHCP client out to the WAN
    pass in log on $WAN proto udp from any port = 67 to any port = 68 tracker 1000001591 label "allow dhcp client out WAN"
    pass out log on $WAN proto udp from any port = 68 to any port = 67 tracker 1000001592 label "allow dhcp client out WAN"
    # Not installing DHCP server firewall rules for WAN which is configured for DHCP.
    antispoof log for $LAN tracker 1000002620
    # allow access to DHCP server on LAN
    pass in log quick on $LAN proto udp from any port = 68 to 255.255.255.255 port = 67 tracker 1000002641 label "allow access to DHCP server"
    pass in log quick on $LAN proto udp from any port = 68 to 192.168.1.1 port = 67 tracker 1000002642 label "allow access to DHCP server"
    pass out log quick on $LAN proto udp from 192.168.1.1 port = 67 to any port = 68 tracker 1000002643 label "allow access to DHCP server"
    antispoof log for $VPN_WAN tracker 1000003670
    antispoof log for $VPN2VLAN tracker 1000004720
    
    # loopback
    pass in log on $loopback inet all tracker 1000005811 label "pass IPv4 loopback"
    pass out log on $loopback inet all tracker 1000005812 label "pass IPv4 loopback"
    pass in log on $loopback inet6 all tracker 1000005813 label "pass IPv6 loopback"
    pass out log on $loopback inet6 all tracker 1000005814 label "pass IPv6 loopback"
    # let out anything from the firewall host itself and decrypted IPsec traffic
    pass out log inet all keep state allow-opts tracker 1000005815 label "let out anything IPv4 from firewall host itself"
    pass out log inet6 all keep state allow-opts tracker 1000005816 label "let out anything IPv6 from firewall host itself"
    pass out log route-to ( em0_vlan10 xxx.xxx.234.1 ) from xxx.xxx.235.198 to !xxx.xxx.234.0/23 tracker 1000005911 keep state allow-opts label "let out anything from firewall host itself"
    pass out log route-to ( ovpnc2 xxx.xxx.42.111 ) from xxx.xxx.42.111 to !xxx.xxx.42.96/27 tracker 1000005912 keep state allow-opts label "let out anything from firewall host itself"
    # make sure the user cannot lock himself out of the webConfigurator or SSH
    pass in log quick on em0_vlan20 proto tcp from any to (em0_vlan20) port { 80 22 } tracker 10000 keep state label "anti-lockout rule"
    # NAT Reflection rules
    pass in log inet tagged PFREFLECT tracker 1000006231 keep state label "NAT REFLECT: Allow traffic to localhost"
    
    # User-defined rules follow
    
    anchor "userrules/*"
    block return  out  on {  em0_vlan10  em0_vlan20  } inet from any to any  tagged "vlan_vpn" tracker 1480374277  label "USER_RULE"
    block return  out  on {  em0_vlan10  em0_vlan20  } inet6 from any to any  tagged "vlan_vpn" tracker 1480374277  label "USER_RULE"
    pass  in  quick  on $OpenVPN  from any to any keep state  label "USER_RULE: OpenVPN  wizard"
    pass  in  quick  on $WAN reply-to ( em0_vlan10 xxx.xxx.234.1 )  proto tcp  from any to 192.168.1.11 port 80 tracker 1480465724 flags S/SA keep state  label "USER_RULE: NAT "
    pass  in  quick  on $WAN reply-to ( em0_vlan10 xxx.xxx.234.1 )  proto tcp  from any to 192.168.1.11 port 443 tracker 1480465740 flags S/SA keep state  label "USER_RULE: NAT "
    pass  in  quick  on $WAN reply-to ( em0_vlan10 xxx.xxx.234.1 ) inet proto tcp  from any to 192.168.1.10 port 32400 tracker 1469306509 flags S/SA keep state  label "USER_RULE: NAT"
    pass  in  quick  on $WAN reply-to ( em0_vlan10 xxx.xxx.234.1 )  proto udp  from any to xxx.xxx.235.198 port 1194 keep state  label "USER_RULE: OpenVPN  wizard"
    pass  in  quick  on $LAN inet from 192.168.1.0/24 to any tracker 0100000101 keep state  label "USER_RULE: Default allow LAN to any rule"
    # at the break! label "USER_RULE: Default allow LAN IPv6 to any rule"
    pass  in  quick  on $VPN_WAN reply-to ( ovpnc2 xxx.xxx.42.111 ) inet proto { tcp udp }  from any to 10.0.10.2 port 59989 >< 60001 tracker 1480558625 keep state  label "USER_RULE: NAT "
    pass  in  quick  on $VPN2VLAN inet from 10.0.10.0/24  to <negate_networks>tag "vlan_vpn" tracker 10000001 keep state  label "NEGATE_ROUTE: Negate policy routing for destination"
    pass  in  quick  on $VPN2VLAN  $GWVPN_WAN_VPNV4 inet from 10.0.10.0/24 to any  tag "vlan_vpn" tracker 1480442441 keep state  label "USER_RULE"
    
    # VPN Rules
    
    anchor "tftp-proxy/*"
    anchor "miniupnpd"
    [/b]
    
    I've also tried checking, "Static route filtering - Bypass firewall rules for traffic on the same interface" in Sysem->Advanced->Firewall & NAT as someone suggested in another thread (diden't help). 
    
    I'm really hopeing for some help here as this is quite out of my depth.</negate_networks></bogonsv6></bogons></virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c></negate_networks></vpn_networks></bogonsv6></bogons></virusprot></snort2c></webconfiguratorlockout></sshlockout> 
    


  • So after intense debugging and countless hours (and a great deal of help from Piab-NL@IRC) I decided to reinstall the pfsense box, setting it up as shown above. Now it works as expected, and I have no idea as to why. Im happy it works, but quite confused as to what the problem was. I did however upgrade to 2.3.2_1 (from 2.3.2).

    I'll do a diff on the config file later today, just to see if I spot anything strange.



  • Very strange.  Glad you got it working.