Reflective routing issues w/ 2.0
-
A while ago I posted this under the "general" routing subheading. Since then this issue has been fixed on 1.2.1, but I still see this issue plaguing 2.0.
==original post==
I have been trying to figure this out for days now ….
Here's the setup:
2 networks (production and virtual) w/ 2 pfSense boxes (prod = 1.2 & virt = 1.3/2.0) as their GW to the internet.
2 ISP connections w/ static addressing.
1 router in between the 2 networks [tried Cisco & Vyatta] (no default route, just statics that point to the pfSense boxes for the connected networks).2 hosts on each network (host V1-win32, V2-*NIX on the virtual network & hosts P1-win32, P2-*NIX on the production network).
So .. now the issue:
I have been able to (when the virtual pfSense box was 1.2) point all traffic on either network to the GW (pfSense) and then add static routes to either pfSense box for the other network (prod or virt) via the "common router" (Cisco or Vyatta) and communicate with no issues.
I have since been testing on the virt network w/ 1.3/2.0 and am no longer able to reflective route. I tried downgrading the 1.3/2.0 box to a 1.2.1 release and noticed the same results.
I see traffic entering the network via the intermediate router … from the prod network going in the prod > virt, but it never gets reflected back out to the prod network.
From the virt network ... I see no traffic going to the prod network at all (again via the intermediate router).
I'm wondering if there is a known reason why reflective routing is borked on the "newest" pfSense releases as it worked seamlessly on the older stable 1.2 release?? ??? ???
I know this may be tough to follow and I HAVE provided a picture if it helps ... ANYONE!!!
Thanks!!!
== further findings ==
I've been doing some testing/troubleshooting and see some different reactions based on the SNAP that I upgrade to (for testing I generally use the latest SNAP).
Things to bear in mind:
The Vyatta/Cisco router is connected to the same segment as the LAN interface of BOTH pfSense boxes.
The default route for each segment on the Vyatta/Cisco router points to pfSense as next hop.
There is NO security on the Vyatta/Cisco router (i.e. no packet filtering).
All hosts on either network (PE or VE) have a default gateway of pfSense.
Both pfSense boxes have a static route for the other network/segment via the Vyatta/Cisco intermediate router.The attached diagram will help to figure out what I am doing here ....
6 Simple test scenario(s):
NOTE: each test is preformed bi-directionally (i.e from PE > VE and then from VE > PE).1.) ping hosts VE1&2 & pfS2.0 from hosts PE1&2 and STABLE pfS1.2
2.) traceroute to hosts VE1&2 & pfS2.0 from hosts PE1&2 and STABLE pfS1.2
3.) SSH from VE1&2 to PE1&2 (and vice-versa)
4.) VNC from PE2 to VE2
5.) examine routes on both pfSense platforms and compare.
6.) examine logs (firewall) and see if anything is getting blocked.Sooooo ... with that all said ... 1.2 & 1.2.1 version all work without issue.
I notice that there are several issues with 2.0 "seemingly" on the return traffic .. i.e. the traffic makes it into the VE segment, but can't get back out. 1 & 2 test scenarios pass and I fail on 3 & 4 ....
Here's a clip of the "blocking rules" according to the 2.0 pfSense in the VE segment.
On a SNAP this morning (11.16.2008 @ 8:00+/-)
@5 block drop quick proto tcp from any port = 0 to any
And on the SNP this afternoon (11.16.2008 @ 13:00+/-)
@3 block drop in log all label "Default deny rule"
@30 pass in on le0 inet proto udp from any port = bootpc to 192.168.22.2 port = bootps keep state label "allow access to DHCP server"
@31 pass out on le0 inet proto udp from 192.168.22.2 port = bootps to any port = bootpc keep state label "allow access to DHCP server"
@32 anchor "spoofing" all
@33 anchor "loopback" all
@34 pass in on lo0 all flags S/SA keep state label "pass loopback"
@35 pass out on lo0 all flags S/SA keep state label "pass loopback"
@36 anchor "firewallout" all
@37 pass out all flags S/SA keep state label "let out anything from firewall host itself"
@38 anchor "anti-lockout" all
@39 pass in quick on le0 from any to (le0:2) flags S/SA keep state label "anti-lockout rule"Sorry for the length everyone.
-
can you retry next snapshot.
-
@ermal:
can you retry next snapshot.
I am testing SNAP:
http://snapshots.pfsense.org/FreeBSD7/RELENG_1/pfSense-Full-Update-2.0-ALPHA-ALPHA-20081116-1850.tgz
Will be back with an update shortly.
-
@ermal:
can you retry next snapshot.
No luck. It passes on test 1&2 (ping and traceroute), but I cannot VNC or SSH into the VE segment to either of the hosts VE1 or 2.
I see the traffic passing the intermediate router (Vyatta) and is see the host getting the packets (via windump on ve2 and tcpdump on VE1) and I see the traffic getting to the default gateway (via pfTop) in the VE segment (pfSense 2.0).
Then it seem to trigger blocking on the pfSense box and I see the denial of traffic as follows :
dst(target-VE1)/dst-port(VNC-tcp5900/SSH-tcp22) > src(originator-PE1)/random-port(>1023) == block
or if you prefer …
Nov 16 19:46:15 LAN 192.168.22.22:5900 192.168.1.20:33722 TCPAnd when I show the "triggering" rule (click on the "X") .. here's what I get :
The rule that triggered this action is:
@3 block drop in log all label "Default deny rule"
@30 pass in on le0 inet proto udp from any port = bootpc to 192.168.22.2 port = bootps keep state label "allow access to DHCP server"
@31 pass out on le0 inet proto udp from 192.168.22.2 port = bootps to any port = bootpc keep state label "allow access to DHCP server"
@32 anchor "spoofing" all
@33 anchor "loopback" all
@34 pass in on lo0 all flags S/SA keep state label "pass loopback"
@35 pass out on lo0 all flags S/SA keep state label "pass loopback"
@36 anchor "firewallout" all
@37 pass out all flags S/SA keep state label "let out anything from firewall host itself"
@38 anchor "anti-lockout" all
@39 pass in quick on le0 from any to (le0:2) flags S/SA keep state label "anti-lockout rule" -
can you send me your ruleset and config.xml to ermal @ pfsense nospam .org
-
@ermal:
can you send me your ruleset and config.xml to ermal @ pfsense nospam .org
I have no issue with posting here either, but can't post a pdf ???, as this environment is torn down and completely segmented from the enterprise facets.
For all intents and purposes .. the test platform (pfSense 2.0) is basically a "default setup". The only real configuration data I keep between SNAP updates and configuration restores is the WAN IP (obviously :o)) and the cert and ssh port setup. The rest of the config is pretty plain jane and default.
There are two pdf files on their way to you …. and thanks for your assistance on this!!!!!!
;D ;D -
Try next snapshot and tell me how it went.
-
@ermal:
Try next snapshot and tell me how it went.
I tried the below version when I saw your post and it has the same results …
Latest Version : Wed Nov 19 12:52:13 EST 2008
Sorry .. I forgot to post the results ...
Nov 20 06:23:56 LAN 192.168.22.22:5900 192.168.1.20:18340 TCP
The rule that triggered this action is:
@3 block drop in log all label "Default deny rule"
@30 anchor "spoofing" all
@31 anchor "loopback" all
@32 pass in on lo0 all flags S/SA keep state label "pass loopback"
@33 pass out on lo0 all flags S/SA keep state label "pass loopback"
@34 anchor "firewallout" all
@35 pass out all flags S/SA keep state label "let out anything from firewall host itself"
@36 anchor "anti-lockout" all
@37 pass in quick on le0 from any to (le0:2) flags S/SA keep state label "anti-lockout rule"
@38 anchor "packagelate" all
@39 block drop in log quick proto tcp from sshlockout:0to any port = rsh-spx label "sshlockout"</sshlockout:0> -
Not sure which if that update has the change in.
Can you check you have rules with 'no state' in them and verify that they have quick in them? -
@ermal:
Can you check you have rules with 'no state' in them and verify that they have quick in them?
Are you talking about all of the rules or just the ones that are getting hit on when I try the test??
I will upgrade to SNAP - New version: Thu Nov 20 19:48:23 EST 2008 … and try this one.
-
@ermal:
Can you check you have rules with 'no state' in them and verify that they have quick in them?
It is still not working with the latest SNAP - Thu Nov 20 19:19:44 EST 2008
Here are the rules:
cat /tmp/rules.debug
#System aliases
loopback = "{ lo0 }"
WAN = "{ le1 }"
LAN = "{ le0 }"User Aliases
netbios = "{ 135 137 138 139 445 }"
set loginterface le1
set loginterface le0
set optimization aggressive
set limit states 51000scrub on $WAN all fragment reassemble
scrub on $LAN all fragment reassemblenat-anchor "ftp-proxy/"
nat-anchor "natearly/"
nat-anchor "natrules/*"Outbound NAT rules
Subnets to NAT
tonatsubnets = "{ 192.168.168.0/24 192.168.10.0/26 192.168.22.0/24 }"
no nat on $WAN to port tftp
nat on $WAN from $tonatsubnets port 500 to any port 500 -> x.x.x.134/32 port 500
nat on $WAN from $tonatsubnets port 4500 to any port 4500 -> x.x.x.134/32 port 4500
nat on $WAN from $tonatsubnets port 5060 to any port 5060 -> x.x.x.134/32 port 5060
nat on $WAN from $tonatsubnets to any -> x.x.x.134/32#SSH Lockout Table
table <sshlockout>persistLoad balancing anchor
rdr-anchor "relayd/*"
FTP proxy
rdr-anchor "ftp-proxy/"
rdr-anchor "tftp-proxy/"IMSpector rdr anchor
rdr-anchor "imspector"
UPnPd rdr anchor
rdr-anchor "miniupnpd"
anchor "ftpsesame/"
anchor "relayd/"
anchor "firewallrules"
#–-------------------------------------------------------------------------default deny rules
#---------------------------------------------------------------------------
block in log all label "Default deny rule"
block out log all label "Default deny rule"We use the mighty pf, we cannot be fooled.
block quick proto { tcp, udp } from any port = 0 to any
block quick proto { tcp, udp } from any to any port = 0snort2c
table <snort2c>persist
block quick from <snort2c>to any label "Block snort2c hosts"
block quick from any to <snort2c>label "Block snort2c hosts"package manager early specific hook
anchor "packageearly"
carp
anchor "carp"
table <virusprot>block in quick from <virusprot>to any label "virusprot overload table"
table <bogons>persist file "/etc/bogons"block bogon networks
http://www.cymru.com/Documents/bogon-bn-nonagg.txt
anchor "wanbogons"
block in log quick on $WAN from <bogons>to any label "block bogon networks from WAN"
antispoof for le1block anything from private networks on interfaces with the option set
antispoof for $WAN
block in log quick on $WAN from 10.0.0.0/8 to any label "block private networks from wan block 10/8"
block in log quick on $WAN from 127.0.0.0/8 to any label "block private networks from wan block 127/8"
block in log quick on $WAN from 172.16.0.0/12 to any label "block private networks from wan block 172.16/12"
block in log quick on $WAN from 192.168.0.0/16 to any label "block private networks from wan block 192.168/16"
antispoof for le0allow access to DHCP server on LAN
anchor "dhcpserverLAN"
pass in on $LAN proto udp from any port = 68 to 255.255.255.255 port = 67 label "allow access to DHCP server"
pass in on $LAN proto udp from any port = 68 to 192.168.2.2 port = 67 label "allow access to DHCP server"
pass out on $LAN proto udp from 192.168.2.2 port = 67 to any port = 68 label "allow access to DHCP server"
anchor "spoofing"loopback
anchor "loopback"
pass in on $loopback all label "pass loopback"
pass out on $loopback all label "pass loopback"anchor "firewallout"
let out anything from the firewall host itself and decrypted IPsec traffic
pass out all keep state label "let out anything from firewall host itself"
make sure the user cannot lock himself out of the webConfigurator or SSH
anchor "anti-lockout"
pass in quick on le0 from any to (le0) keep state label "anti-lockout rule"NAT Reflection rules
package manager late specific hook
anchor "packagelate"
SSH lockout
block in log quick proto tcp from <sshlockout>to any port 222 label "sshlockout"
anchor "ftp-proxy/*"enable ftp-proxy
User-defined aliases follow
User-defined rules follow
block in quick on $LAN proto { tcp udp } from 192.168.2.0/24 to any port $netbios label "USER_RULE: block netbios"
pass in quick on $LAN from 192.168.2.0/24 to any keep state label "USER_RULE: Default allow LAN to any rule"
pass in quick on $LAN from { 192.168.1.0/24 } to 192.168.2.2 keep state label "USER_RULE"VPN Rules
anchor "limitingesr"
IMSpector
anchor "imspector"
uPnPd
anchor "miniupnpd"</sshlockout></bogons></bogons></virusprot></virusprot></snort2c></snort2c></snort2c></sshlockout>
-
Not sure what ermal is after but no state only appears on rules that are added when "Bypass firewall rules for traffic on the same interface" is checked in System -> Advanced.
-
the bypass rules aren't even being added anymore in 2.0.
-
Not sure what ermal is after but no state only appears on rules that are added when "Bypass firewall rules for traffic on the same interface" is checked in System -> Advanced.
That is what I understood to be the case, and for these tests I don't have that option checked.
-
They should be added again after checking it.
-
I just updated to the latest SNAP:
2.0-ALPHA-ALPHA
built on Tue Nov 25 14:59:09 EST 2008
FreeBSD 7.1-PRERELEASEand the issue still exists. As before tests one and two pass fine but then I get stopped dead at test three (SSH) and four (VNC test).
Here's the logging output …..
Nov 27 10:11:56 LAN 192.168.22.22:5900 192.168.1.20:18340 TCP
The rule that triggered this action is:
@3 block drop in log all label "Default deny rule"
@30 anchor "spoofing" all
@31 anchor "loopback" all
@32 pass in on lo0 all flags S/SA keep state label "pass loopback"
@33 pass out on lo0 all flags S/SA keep state label "pass loopback"
@34 anchor "firewallout" all
@35 pass out all flags S/SA keep state label "let out anything from firewall host itself"
@36 anchor "anti-lockout" all
@37 pass in quick on le0 from any to (le0:2) flags S/SA keep state label "anti-lockout rule"
@38 anchor "packagelate" all
@39 block drop in log quick proto tcp from sshlockout:0to any port = rsh-spx label "sshlockout"</sshlockout:0>