Reflective routing issues w/ 2.0



  • A while ago I posted this under the "general" routing subheading.  Since then this issue has been fixed on 1.2.1, but I still see this issue plaguing 2.0.

    ==original post==

    I have been trying to figure this out for days now ….

    Here's the setup:

    2 networks (production and virtual) w/ 2 pfSense boxes (prod = 1.2 & virt = 1.3/2.0) as their GW to the internet.
    2 ISP connections w/ static addressing.
    1 router in between the 2 networks [tried Cisco & Vyatta] (no default route, just statics that point to the pfSense boxes for the connected networks).

    2 hosts on each network (host V1-win32, V2-*NIX on the virtual network & hosts P1-win32, P2-*NIX on the production network).

    So .. now the issue:

    I have been able to (when the virtual pfSense box was 1.2) point all traffic on either network to the GW (pfSense) and then add static routes to either pfSense box for the other network (prod or virt) via the "common router" (Cisco or Vyatta) and communicate with no issues.

    I have since been testing on the virt network w/ 1.3/2.0 and am no longer able to reflective route.  I tried downgrading the 1.3/2.0 box to a 1.2.1 release and noticed the same results.

    I see traffic entering the network via the intermediate router … from the prod network going in the prod > virt, but it never gets reflected back out to the prod network.

    From the virt network ... I see no traffic going to the prod network at all (again via the intermediate router).

    I'm wondering if there is a known reason why reflective routing is borked on the "newest" pfSense releases as it worked seamlessly on the older stable 1.2 release?? ??? ???

    I know this may be tough to follow and I HAVE provided a picture if it helps ... ANYONE!!!

    Thanks!!!

    == further findings ==

    I've been doing some testing/troubleshooting and see some different reactions based on the SNAP that I upgrade to (for testing I generally use the latest SNAP).

    Things to bear in mind:
    The Vyatta/Cisco router is connected to the same segment as the LAN interface of BOTH pfSense boxes.
    The default route for each segment on the Vyatta/Cisco router points to pfSense as next hop.
    There is NO security on the Vyatta/Cisco router (i.e. no packet filtering).
    All hosts on either network (PE or VE) have a default gateway of pfSense.
    Both pfSense boxes have a static route for the other network/segment via the Vyatta/Cisco intermediate router.

    The attached diagram will help to figure out what I am doing here ....

    6 Simple test scenario(s):
    NOTE: each test is preformed bi-directionally (i.e from PE > VE and then from VE > PE).

    1.) ping hosts VE1&2 & pfS2.0 from hosts PE1&2 and STABLE pfS1.2
    2.) traceroute to hosts VE1&2 & pfS2.0 from hosts PE1&2 and STABLE pfS1.2
    3.) SSH from VE1&2 to PE1&2 (and vice-versa)
    4.) VNC from PE2 to VE2
    5.) examine routes on both pfSense platforms and compare.
    6.) examine logs (firewall) and see if anything is getting blocked.

    Sooooo ... with that all said ... 1.2 & 1.2.1 version all work without issue.

    I notice that there are several issues with 2.0 "seemingly" on the return traffic .. i.e. the traffic makes it into the VE segment, but can't get back out.  1 & 2 test scenarios pass and I fail on 3 & 4 ....

    Here's a clip of the "blocking rules" according to the 2.0 pfSense in the VE segment.

    On a SNAP this morning (11.16.2008 @ 8:00+/-)

    @5 block drop quick proto tcp from any port = 0 to any

    And on the SNP this afternoon (11.16.2008 @ 13:00+/-)

    @3 block drop in log all label "Default deny rule"
    @30 pass in on le0 inet proto udp from any port = bootpc to 192.168.22.2 port = bootps keep state label "allow access to DHCP server"
    @31 pass out on le0 inet proto udp from 192.168.22.2 port = bootps to any port = bootpc keep state label "allow access to DHCP server"
    @32 anchor "spoofing" all
    @33 anchor "loopback" all
    @34 pass in on lo0 all flags S/SA keep state label "pass loopback"
    @35 pass out on lo0 all flags S/SA keep state label "pass loopback"
    @36 anchor "firewallout" all
    @37 pass out all flags S/SA keep state label "let out anything from firewall host itself"
    @38 anchor "anti-lockout" all
    @39 pass in quick on le0 from any to (le0:2) flags S/SA keep state label "anti-lockout rule"

    Sorry for the length everyone.



  • can you retry next snapshot.



  • @ermal:

    can you retry next snapshot.

    I am testing SNAP:

    http://snapshots.pfsense.org/FreeBSD7/RELENG_1/pfSense-Full-Update-2.0-ALPHA-ALPHA-20081116-1850.tgz

    Will be back with an update shortly.



  • @ermal:

    can you retry next snapshot.

    No luck.  It passes on test 1&2 (ping and traceroute), but I cannot VNC or SSH into the VE segment to either of the hosts VE1 or 2.

    I see the traffic passing the intermediate router (Vyatta) and is see the host getting the packets (via windump on ve2 and tcpdump on VE1) and I see the traffic getting to the default gateway (via pfTop) in the VE segment (pfSense 2.0).

    Then it seem to trigger blocking on the pfSense box and I see the denial of traffic as follows :

    dst(target-VE1)/dst-port(VNC-tcp5900/SSH-tcp22) > src(originator-PE1)/random-port(>1023) == block
    or if you prefer …
    Nov 16 19:46:15  LAN  192.168.22.22:5900  192.168.1.20:33722  TCP

    And when I show the "triggering" rule (click on the "X") .. here's what I get :

    The rule that triggered this action is:

    @3 block drop in log all label "Default deny rule"
    @30 pass in on le0 inet proto udp from any port = bootpc to 192.168.22.2 port = bootps keep state label "allow access to DHCP server"
    @31 pass out on le0 inet proto udp from 192.168.22.2 port = bootps to any port = bootpc keep state label "allow access to DHCP server"
    @32 anchor "spoofing" all
    @33 anchor "loopback" all
    @34 pass in on lo0 all flags S/SA keep state label "pass loopback"
    @35 pass out on lo0 all flags S/SA keep state label "pass loopback"
    @36 anchor "firewallout" all
    @37 pass out all flags S/SA keep state label "let out anything from firewall host itself"
    @38 anchor "anti-lockout" all
    @39 pass in quick on le0 from any to (le0:2) flags S/SA keep state label "anti-lockout rule"



  • can you send me your ruleset and config.xml to ermal @ pfsense nospam .org



  • @ermal:

    can you send me your ruleset and config.xml to ermal @ pfsense nospam .org

    I have no issue with posting here either, but can't post a pdf ???, as this environment is torn down and completely segmented from the enterprise facets.

    For all intents and purposes .. the test platform (pfSense 2.0) is basically a "default setup".  The only real configuration data I keep between SNAP updates and configuration restores is the WAN IP (obviously :o)) and the cert and ssh port setup.  The rest of the config is pretty plain jane and default.

    There are two pdf files on their way to you …. and thanks for your assistance on this!!!!!!
    ;D ;D



  • Try next snapshot and tell me how it went.



  • @ermal:

    Try next snapshot and tell me how it went.

    I tried the below version when I saw your post and it has the same results …

    Latest Version  : Wed Nov 19 12:52:13 EST 2008

    Sorry .. I forgot to post the results ...

    Nov 20 06:23:56  LAN  192.168.22.22:5900  192.168.1.20:18340  TCP

    The rule that triggered this action is:

    @3 block drop in log all label "Default deny rule"
    @30 anchor "spoofing" all
    @31 anchor "loopback" all
    @32 pass in on lo0 all flags S/SA keep state label "pass loopback"
    @33 pass out on lo0 all flags S/SA keep state label "pass loopback"
    @34 anchor "firewallout" all
    @35 pass out all flags S/SA keep state label "let out anything from firewall host itself"
    @36 anchor "anti-lockout" all
    @37 pass in quick on le0 from any to (le0:2) flags S/SA keep state label "anti-lockout rule"
    @38 anchor "packagelate" all
    @39 block drop in log quick proto tcp from sshlockout:0to any port = rsh-spx label "sshlockout"</sshlockout:0>



  • Not sure which if that update has the change in.
    Can you check you have rules with 'no state' in them and verify that they have quick in them?



  • @ermal:

    Can you check you have rules with 'no state' in them and verify that they have quick in them?

    Are you talking about all of the rules or just the ones that are getting hit on when I try the test??

    I will upgrade to SNAP - New version: Thu Nov 20 19:48:23 EST 2008 … and try this one.



  • @ermal:

    Can you check you have rules with 'no state' in them and verify that they have quick in them?

    It is still not working with the latest SNAP - Thu Nov 20 19:19:44 EST 2008

    Here are the rules:

    cat /tmp/rules.debug

    #System aliases

    loopback = "{ lo0 }"
    WAN = "{ le1 }"
    LAN = "{ le0 }"

    User Aliases

    netbios = "{ 135 137 138 139 445 }"

    set loginterface le1
    set loginterface le0
    set optimization aggressive
    set limit states 51000

    scrub on $WAN all    fragment reassemble
    scrub on $LAN all    fragment reassemble

    nat-anchor "ftp-proxy/"
    nat-anchor "natearly/
    "
    nat-anchor "natrules/*"

    Outbound NAT rules

    Subnets to NAT

    tonatsubnets  = "{ 192.168.168.0/24 192.168.10.0/26 192.168.22.0/24  }"
    no nat on $WAN to port tftp
    nat on $WAN from $tonatsubnets port 500 to any port 500 -> x.x.x.134/32 port 500
    nat on $WAN from $tonatsubnets port 4500 to any port 4500 -> x.x.x.134/32 port 4500
    nat on $WAN from $tonatsubnets port 5060 to any port 5060 -> x.x.x.134/32 port 5060
    nat on $WAN from $tonatsubnets to any -> x.x.x.134/32

    #SSH Lockout Table
    table <sshlockout>persist

    Load balancing anchor

    rdr-anchor "relayd/*"

    FTP proxy

    rdr-anchor "ftp-proxy/"
    rdr-anchor "tftp-proxy/
    "

    IMSpector rdr anchor

    rdr-anchor "imspector"

    UPnPd rdr anchor

    rdr-anchor "miniupnpd"

    anchor "ftpsesame/"
    anchor "relayd/
    "
    anchor "firewallrules"
    #–-------------------------------------------------------------------------

    default deny rules

    #---------------------------------------------------------------------------
    block in log all label "Default deny rule"
    block out log all label "Default deny rule"

    We use the mighty pf, we cannot be fooled.

    block quick proto { tcp, udp } from any port = 0 to any
    block quick proto { tcp, udp } from any to any port = 0

    snort2c

    table <snort2c>persist
    block quick from <snort2c>to any label "Block snort2c hosts"
    block quick from any to <snort2c>label "Block snort2c hosts"

    package manager early specific hook

    anchor "packageearly"

    carp

    anchor "carp"
    table <virusprot>block in quick from <virusprot>to any label "virusprot overload table"
    table <bogons>persist file "/etc/bogons"

    block bogon networks

    http://www.cymru.com/Documents/bogon-bn-nonagg.txt

    anchor "wanbogons"
    block in log quick on $WAN from <bogons>to any label "block bogon networks from WAN"
    antispoof for le1

    block anything from private networks on interfaces with the option set

    antispoof for $WAN
    block in log quick on $WAN from 10.0.0.0/8 to any label "block private networks from wan block 10/8"
    block in log quick on $WAN from 127.0.0.0/8 to any label "block private networks from wan block 127/8"
    block in log quick on $WAN from 172.16.0.0/12 to any label "block private networks from wan block 172.16/12"
    block in log quick on $WAN from 192.168.0.0/16 to any label "block private networks from wan block 192.168/16"
    antispoof for le0

    allow access to DHCP server on LAN

    anchor "dhcpserverLAN"
    pass in on $LAN proto udp from any port = 68 to 255.255.255.255 port = 67 label "allow access to DHCP server"
    pass in on $LAN proto udp from any port = 68 to 192.168.2.2 port = 67 label "allow access to DHCP server"
    pass out on $LAN proto udp from 192.168.2.2 port = 67 to any port = 68 label "allow access to DHCP server"
    anchor "spoofing"

    loopback

    anchor "loopback"
    pass in on $loopback all label "pass loopback"
    pass out on $loopback all label "pass loopback"

    anchor "firewallout"

    let out anything from the firewall host itself and decrypted IPsec traffic

    pass out all keep state label "let out anything from firewall host itself"

    make sure the user cannot lock himself out of the webConfigurator or SSH

    anchor "anti-lockout"
    pass in quick on le0 from any to (le0) keep state label "anti-lockout rule"

    NAT Reflection rules

    package manager late specific hook

    anchor "packagelate"

    SSH lockout

    block in log quick proto tcp from <sshlockout>to any port 222 label "sshlockout"
    anchor "ftp-proxy/*"

    enable ftp-proxy

    User-defined aliases follow

    User-defined rules follow

    block  in  quick  on $LAN  proto { tcp udp }  from 192.168.2.0/24 to any  port $netbios  label "USER_RULE: block netbios"
    pass  in  quick  on $LAN  from 192.168.2.0/24 to any keep state  label "USER_RULE: Default allow LAN to any rule"
    pass  in  quick  on $LAN  from {  192.168.1.0/24 } to 192.168.2.2 keep state  label "USER_RULE"

    VPN Rules

    anchor "limitingesr"

    IMSpector

    anchor "imspector"

    uPnPd

    anchor "miniupnpd"</sshlockout></bogons></bogons></virusprot></virusprot></snort2c></snort2c></snort2c></sshlockout>



  • Not sure what ermal is after but no state only appears on rules that are added when "Bypass firewall rules for traffic on the same interface" is checked in System -> Advanced.



  • the bypass rules aren't even being added anymore in 2.0.



  • @sullrich:

    Not sure what ermal is after but no state only appears on rules that are added when "Bypass firewall rules for traffic on the same interface" is checked in System -> Advanced.

    That is what I understood to be the case, and for these tests I don't have that option checked.



  • They should be added again after checking it.



  • I just updated to the latest SNAP:

    2.0-ALPHA-ALPHA
    built on Tue Nov 25 14:59:09 EST 2008
    FreeBSD 7.1-PRERELEASE

    and the issue still exists.  As before tests one and two pass fine but then I get stopped dead at test three (SSH) and four (VNC test).

    Here's the logging output …..

    Nov 27 10:11:56     LAN     192.168.22.22:5900     192.168.1.20:18340     TCP

    The rule that triggered this action is:

    @3 block drop in log all label "Default deny rule"
    @30 anchor "spoofing" all
    @31 anchor "loopback" all
    @32 pass in on lo0 all flags S/SA keep state label "pass loopback"
    @33 pass out on lo0 all flags S/SA keep state label "pass loopback"
    @34 anchor "firewallout" all
    @35 pass out all flags S/SA keep state label "let out anything from firewall host itself"
    @36 anchor "anti-lockout" all
    @37 pass in quick on le0 from any to (le0:2) flags S/SA keep state label "anti-lockout rule"
    @38 anchor "packagelate" all
    @39 block drop in log quick proto tcp from sshlockout:0to any port = rsh-spx label "sshlockout"</sshlockout:0>


Log in to reply