Asymmetric routing problem? pictures and config attached
-
After initialy haveing some problems getting my VPN tunnel to work, it's now OK. But looking over the Firewall logs I see signs of asymmetric routing issues, which I can't seem to find the solution for, so I'm reaching out to you guys for some input and hopefully an helping hand.
I tried looking over various forum threads and other sites for a possible solution, but I can't seem to find the issue.
Logs:
The entry comeing in at VPN2VLAN where Destination is 10.0.10.2:59990 is me checking if port is open from an external source.I've attached images of my configuration along with the entire config of the router
Network setup (the WAN/LAN VLAN setup is due to 1 physical NIC on the PFSENSE machine)
Interfaces - (assign)
Interfaces - (assign) - VLANs
Firewall - NAT - Port forwards
Firewall - NAT - Outbound NAT
Firewall - Rules - Floating (reject marked packets, they get marked in the rule @ VPN2VLAN)
Firewall - Rules - WAN
Firewall - Rules - LAN
Firewall - Rules - VPN_WAN
Firewall - Rules - VPN2VLAN
Firewall - Rules - OpenVPN (this is rules for a OpenVPN server)
And finaly, the config file:
set optimization normal set limit states 1623000 set limit src-nodes 1623000 #System aliases loopback = "{ lo0 }" WAN = "{ em0_vlan10 }" LAN = "{ em0_vlan20 }" VPN_WAN = "{ ovpnc2 }" VPN2VLAN = "{ em0_vlan30 }" OpenVPN = "{ openvpn }" #SSH Lockout Table table <sshlockout>persist table <webconfiguratorlockout>persist #Snort tables table <snort2c>table <virusprot>table <bogons>persist file "/etc/bogons" table <bogonsv6>persist file "/etc/bogonsv6" table <vpn_networks>{ 192.168.2.0/24 } table <negate_networks>{ 192.168.2.0/24 } # User Aliases # Gateways GWWAN_DHCP = " route-to ( em0_vlan10 xxx.xxx.234.1 ) " GWVPN_WAN_VPNV4 = " route-to ( ovpnc2 xxx.xxx.42.111 ) " GWWAN_DHCP6 = " route-to ( em0_vlan10 xxx.xxx.234.1 ) " GWVPN_WAN_VPNV6 = " route-to ( ovpnc2 xxx.xxx.42.111 ) " set loginterface em0_vlan20 set skip on pfsync0 scrub on $WAN all fragment reassemble scrub on $LAN all fragment reassemble scrub on $VPN_WAN all fragment reassemble scrub on $VPN2VLAN all fragment reassemble no nat proto carp no rdr proto carp nat-anchor "natearly/*" nat-anchor "natrules/*" # Outbound NAT rules (manual) nat on $WAN from 127.0.0.0/8 to any port 500 -> xxx.xxx.235.198/32 static-port nat on $WAN from 127.0.0.0/8 to any -> xxx.xxx.235.198/32 port 1024:65535 nat on $WAN from 192.168.1.0/24 to any port 500 -> xxx.xxx.235.198/32 static-port nat on $WAN from 192.168.1.0/24 to any -> xxx.xxx.235.198/32 port 1024:65535 nat on $WAN from 192.168.2.0/24 to any port 500 -> xxx.xxx.235.198/32 static-port nat on $WAN from 192.168.2.0/24 to any -> xxx.xxx.235.198/32 port 1024:65535 nat on $VPN_WAN from 10.0.10.0/24 to any port 500 -> xxx.xxx.42.111/32 static-port nat on $VPN_WAN from 10.0.10.0/24 to any -> xxx.xxx.42.111/32 port 1024:65535 # Load balancing anchor rdr-anchor "relayd/*" # TFTP proxy rdr-anchor "tftp-proxy/*" # NAT Inbound Redirects rdr on em0_vlan10 proto tcp from any to xxx.xxx.235.198 port 80 -> 192.168.1.11 # Reflection redirect rdr on { em0_vlan20 em0_vlan30 openvpn } proto tcp from any to xxx.xxx.235.198 port 80 -> 192.168.1.11 no nat on em0_vlan20 proto tcp from em0_vlan20 to 192.168.1.11 port 80 nat on em0_vlan20 proto tcp from 192.168.1.0/24 to 192.168.1.11 port 80 -> 192.168.1.1 port 1024:65535 rdr on em0_vlan10 proto tcp from any to xxx.xxx.235.198 port 443 -> 192.168.1.11 # Reflection redirect rdr on { em0_vlan20 em0_vlan30 openvpn } proto tcp from any to xxx.xxx.235.198 port 443 -> 192.168.1.11 no nat on em0_vlan20 proto tcp from em0_vlan20 to 192.168.1.11 port 443 nat on em0_vlan20 proto tcp from 192.168.1.0/24 to 192.168.1.11 port 443 -> 192.168.1.1 port 1024:65535 rdr on em0_vlan10 proto tcp from any to xxx.xxx.235.198 port 32400 -> 192.168.1.10 # Reflection redirect rdr on { em0_vlan20 em0_vlan30 openvpn } proto tcp from any to xxx.xxx.235.198 port 32400 -> 192.168.1.10 no nat on em0_vlan20 proto tcp from em0_vlan20 to 192.168.1.10 port 32400 nat on em0_vlan20 proto tcp from 192.168.1.0/24 to 192.168.1.10 port 32400 -> 192.168.1.1 port 1024:65535 rdr on ovpnc2 proto { tcp udp } from any to xxx.xxx.42.111 port 59990:60000 -> 10.0.10.2 # Reflection redirect rdr on { em0_vlan20 em0_vlan30 openvpn } proto { tcp udp } from any to xxx.xxx.42.111 port 59990:60000 -> 10.0.10.2 no nat on em0_vlan30 proto { tcp udp } from em0_vlan30 to 10.0.10.2 port 59990:60000 nat on em0_vlan30 proto { tcp udp } from 10.0.10.0/24 to 10.0.10.2 port 59990:60000 -> 10.0.10.1 port 1024:65535 # UPnPd rdr anchor rdr-anchor "miniupnpd" anchor "relayd/*" anchor "openvpn/*" anchor "ipsec/*" # block IPv4 link-local. Per RFC 3927, link local "MUST NOT" be forwarded by a routing device, # and clients "MUST NOT" send such packets to a router. FreeBSD won't route 169.254./16, but # route-to can override that, causing problems such as in redmine #2073 block in log quick from 169.254.0.0/16 to any tracker 1000000101 label "Block IPv4 link-local" block in log quick from any to 169.254.0.0/16 tracker 1000000102 label "Block IPv4 link-local" #--------------------------------------------------------------------------- # default deny rules #--------------------------------------------------------------------------- block in log inet all tracker 1000000103 label "Default deny rule IPv4" block out log inet all tracker 1000000104 label "Default deny rule IPv4" block in log inet6 all tracker 1000000105 label "Default deny rule IPv6" block out log inet6 all tracker 1000000106 label "Default deny rule IPv6" # IPv6 ICMP is not auxilary, it is required for operation # See man icmp6(4) # 1 unreach Destination unreachable # 2 toobig Packet too big # 128 echoreq Echo service request # 129 echorep Echo service reply # 133 routersol Router solicitation # 134 routeradv Router advertisement # 135 neighbrsol Neighbor solicitation # 136 neighbradv Neighbor advertisement pass log quick inet6 proto ipv6-icmp from any to any icmp6-type {1,2,135,136} tracker 1000000107 keep state # Allow only bare essential icmpv6 packets (NS, NA, and RA, echoreq, echorep) pass out log quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {129,133,134,135,136} tracker 1000000108 keep state pass out log quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {129,133,134,135,136} tracker 1000000109 keep state pass in log quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000110 keep state pass in log quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000111 keep state pass in log quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {128,133,134,135,136} tracker 1000000112 keep state # We use the mighty pf, we cannot be fooled. block log quick inet proto { tcp, udp } from any port = 0 to any tracker 1000000113 label "Block traffic from port 0" block log quick inet proto { tcp, udp } from any to any port = 0 tracker 1000000114 label "Block traffic to port 0" block log quick inet6 proto { tcp, udp } from any port = 0 to any tracker 1000000115 label "Block traffic from port 0" block log quick inet6 proto { tcp, udp } from any to any port = 0 tracker 1000000116 label "Block traffic to port 0" # Snort package block log quick from <snort2c>to any tracker 1000000117 label "Block snort2c hosts" block log quick from any to <snort2c>tracker 1000000118 label "Block snort2c hosts" # SSH lockout block in log quick proto tcp from <sshlockout>to (self) port 22 tracker 1000000301 label "sshlockout" # webConfigurator lockout block in log quick proto tcp from <webconfiguratorlockout>to (self) port 80 tracker 1000000351 label "webConfiguratorlockout" block in log quick from <virusprot>to any tracker 1000000400 label "virusprot overload table" # allow our DHCPv6 client out to the WAN pass in log quick on $WAN proto udp from fe80::/10 port = 546 to fe80::/10 port = 546 tracker 1000000561 label "allow dhcpv6 client in WAN" pass in log quick on $WAN proto udp from any port = 547 to any port = 546 tracker 1000000562 label "allow dhcpv6 client in WAN" pass out log quick on $WAN proto udp from any port = 546 to any port = 547 tracker 1000000563 label "allow dhcpv6 client out WAN" # block bogon networks (IPv4) # http://www.cymru.com/Documents/bogon-bn-nonagg.txt block in quick on $WAN from <bogons>to any tracker 11000 label "block bogon IPv4 networks from WAN" # block bogon networks (IPv6) # http://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt block in quick on $WAN from <bogonsv6>to any tracker 11000 label "block bogon IPv6 networks from WAN" antispoof log for $WAN tracker 1000001570 # block anything from private networks on interfaces with the option set block in quick on $WAN from 10.0.0.0/8 to any tracker 12000 label "Block private networks from WAN block 10/8" block in quick on $WAN from 127.0.0.0/8 to any tracker 12000 label "Block private networks from WAN block 127/8" b[b]lock in quick on $WAN from 172.16.0.0/12 to any tracker 12000 label "Block private networks from WAN block 172.16/12" block in quick on $WAN from 192.168.0.0/16 to any tracker 12000 label "Block private networks from WAN block 192.168/16" block in quick on $WAN from fc00::/7 to any tracker 12000 label "Block ULA networks from WAN block fc00::/7" # allow our DHCP client out to the WAN pass in log on $WAN proto udp from any port = 67 to any port = 68 tracker 1000001591 label "allow dhcp client out WAN" pass out log on $WAN proto udp from any port = 68 to any port = 67 tracker 1000001592 label "allow dhcp client out WAN" # Not installing DHCP server firewall rules for WAN which is configured for DHCP. antispoof log for $LAN tracker 1000002620 # allow access to DHCP server on LAN pass in log quick on $LAN proto udp from any port = 68 to 255.255.255.255 port = 67 tracker 1000002641 label "allow access to DHCP server" pass in log quick on $LAN proto udp from any port = 68 to 192.168.1.1 port = 67 tracker 1000002642 label "allow access to DHCP server" pass out log quick on $LAN proto udp from 192.168.1.1 port = 67 to any port = 68 tracker 1000002643 label "allow access to DHCP server" antispoof log for $VPN_WAN tracker 1000003670 antispoof log for $VPN2VLAN tracker 1000004720 # loopback pass in log on $loopback inet all tracker 1000005811 label "pass IPv4 loopback" pass out log on $loopback inet all tracker 1000005812 label "pass IPv4 loopback" pass in log on $loopback inet6 all tracker 1000005813 label "pass IPv6 loopback" pass out log on $loopback inet6 all tracker 1000005814 label "pass IPv6 loopback" # let out anything from the firewall host itself and decrypted IPsec traffic pass out log inet all keep state allow-opts tracker 1000005815 label "let out anything IPv4 from firewall host itself" pass out log inet6 all keep state allow-opts tracker 1000005816 label "let out anything IPv6 from firewall host itself" pass out log route-to ( em0_vlan10 xxx.xxx.234.1 ) from xxx.xxx.235.198 to !xxx.xxx.234.0/23 tracker 1000005911 keep state allow-opts label "let out anything from firewall host itself" pass out log route-to ( ovpnc2 xxx.xxx.42.111 ) from xxx.xxx.42.111 to !xxx.xxx.42.96/27 tracker 1000005912 keep state allow-opts label "let out anything from firewall host itself" # make sure the user cannot lock himself out of the webConfigurator or SSH pass in log quick on em0_vlan20 proto tcp from any to (em0_vlan20) port { 80 22 } tracker 10000 keep state label "anti-lockout rule" # NAT Reflection rules pass in log inet tagged PFREFLECT tracker 1000006231 keep state label "NAT REFLECT: Allow traffic to localhost" # User-defined rules follow anchor "userrules/*" block return out on { em0_vlan10 em0_vlan20 } inet from any to any tagged "vlan_vpn" tracker 1480374277 label "USER_RULE" block return out on { em0_vlan10 em0_vlan20 } inet6 from any to any tagged "vlan_vpn" tracker 1480374277 label "USER_RULE" pass in quick on $OpenVPN from any to any keep state label "USER_RULE: OpenVPN wizard" pass in quick on $WAN reply-to ( em0_vlan10 xxx.xxx.234.1 ) proto tcp from any to 192.168.1.11 port 80 tracker 1480465724 flags S/SA keep state label "USER_RULE: NAT " pass in quick on $WAN reply-to ( em0_vlan10 xxx.xxx.234.1 ) proto tcp from any to 192.168.1.11 port 443 tracker 1480465740 flags S/SA keep state label "USER_RULE: NAT " pass in quick on $WAN reply-to ( em0_vlan10 xxx.xxx.234.1 ) inet proto tcp from any to 192.168.1.10 port 32400 tracker 1469306509 flags S/SA keep state label "USER_RULE: NAT" pass in quick on $WAN reply-to ( em0_vlan10 xxx.xxx.234.1 ) proto udp from any to xxx.xxx.235.198 port 1194 keep state label "USER_RULE: OpenVPN wizard" pass in quick on $LAN inet from 192.168.1.0/24 to any tracker 0100000101 keep state label "USER_RULE: Default allow LAN to any rule" # at the break! label "USER_RULE: Default allow LAN IPv6 to any rule" pass in quick on $VPN_WAN reply-to ( ovpnc2 xxx.xxx.42.111 ) inet proto { tcp udp } from any to 10.0.10.2 port 59989 >< 60001 tracker 1480558625 keep state label "USER_RULE: NAT " pass in quick on $VPN2VLAN inet from 10.0.10.0/24 to <negate_networks>tag "vlan_vpn" tracker 10000001 keep state label "NEGATE_ROUTE: Negate policy routing for destination" pass in quick on $VPN2VLAN $GWVPN_WAN_VPNV4 inet from 10.0.10.0/24 to any tag "vlan_vpn" tracker 1480442441 keep state label "USER_RULE" # VPN Rules anchor "tftp-proxy/*" anchor "miniupnpd" [/b] I've also tried checking, "Static route filtering - Bypass firewall rules for traffic on the same interface" in Sysem->Advanced->Firewall & NAT as someone suggested in another thread (diden't help). I'm really hopeing for some help here as this is quite out of my depth.</negate_networks></bogonsv6></bogons></virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c></negate_networks></vpn_networks></bogonsv6></bogons></virusprot></snort2c></webconfiguratorlockout></sshlockout> -
So after intense debugging and countless hours (and a great deal of help from Piab-NL@IRC) I decided to reinstall the pfsense box, setting it up as shown above. Now it works as expected, and I have no idea as to why. Im happy it works, but quite confused as to what the problem was. I did however upgrade to 2.3.2_1 (from 2.3.2).
I'll do a diff on the config file later today, just to see if I spot anything strange.
-
Very strange. Glad you got it working.
Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.