MultiWAN / DMZ routing problem



  • My setup - overview
    –-----------------

    Multi Wan - On a failover group for outbound from LAN - normally using WAN3 if up [ wan1 / 2 / 3 DSL with seperate routers and private xover lans to pfsense ]
    DMZ - with public IP's arriving down WAN1 (cisco) then routed to pfsense and into DMZ ( based on default routing [ no pfsense fixed route ] )
    LAN - ~20 users, outbound traffic generally routed through WAN group

    The problem I have is this…

    Mail server A - on the internet [ smtp heavily used from our LAN ]
    Mail server B - in DMZ

    If Mail server A tries to open an SMTP connection to Mail server B the following occurs…

    As expected the connection (SYN) comes in on WAN1 and is routed to pfsense, which then [ based on WAN1 firewall rules ] allows it port 25 access to Mail server B

    Mail server B then responds (SYN ACK) but the packet travels up WAN3 [getting natted on way out on adsl router] so upon arrival to Mail server A is ignored

    As far as I can guess there is one of 3 problems here…

    a - that due to all the existing connections TO port 25 on Mail server A (via wan3) pfsense routes the connection up wan3

    • OR -
      b - the inbound path for some reason is not recorded [ the firewall policy that allowed the request ] so the return path is generated as if it is not an existing connection
    • OR -
      c - I have misjudged what I should be able to do

    I am currently running on 2 Beta / 2010-01-25 [i386]

    I should also point out that from any other server the smtp connection is fine
    [ which is what makes me think this is problem a ]
    In which case i would suggest that maybe the sticky connections are causing the issue - and maybe the rule is transposing teh source and destination ports ?

    Any help would be appriciated

    TIA

    pugelarouge

    PS : Am aware this is BETA, and not suggested for production



  • OK - so heres me from one of my live internet [as in colo - no dmz, no nat or anything] web servers

    AAA.BBB.CCC.200 - my personal server in DMZ [behind pfsense - public IP routed - no natting - routed in via cisco sdsl router - then next fixed route to pfsense box on WAN1 ]
    ( AAA.BBB.CCC.200 has a gateway (DMZ1 on pfsense) at AAA.BBB.CCC.193 and a mask of 255.255.255.224 - so in range AAA.BBB.CCC.192 -> AAA.BBB.CCC.223 [ a /27 ] )

    AAA.BBB.CCC.228 - a company server, on standard public IP in a data center
    ( AAA.BBB.CCC.228 has a gateway at AAA.BBB.CCC.225 and a mask of 255.255.255.240 - so in range AAA.BBB.CCC.224 -> AAA.BBB.CCC.239 [ a /28 ] )

    DDD.EEE.FF.219 - the public IP of the adsl connected to wan3 on my pfsense box

    From AAA.BBB.CCC.228 :

    • I fired off a 'telnet AAA.BBB.CCC.200 80'
    • captured with 'tcpdump -n' [ diag_packet_capture.php doesn't want to show any capture at the mo]

    10:19:37.476767 IP AAA.BBB.CCC.228.49940 > AAA.BBB.CCC.200.80: S 2457225471:2457225471(0) win 5840 <mss 1460,sackok,timestamp="" 1076751111[|tcp]="">10:19:37.479894 IP DDD.EEE.FF.219.53622 > AAA.BBB.CCC.228.49940: S 712187932:712187932(0) ack 2457225472 win 5792 <mss 1460,sackok,timestamp="" 318338740[|tcp]="">10:19:37.479904 IP AAA.BBB.CCC.228.49940 > DDD.EEE.FF.219.53622: R 2457225472:2457225472(0) win 0

    And again - From AAA.BBB.CCC.228 :

    • I fired off a 'telnet AAA.BBB.CCC.200 25'
    • captured with 'tcpdump -n' [ diag_packet_capture.php doesn't want to show any capture at the mo]

    09:53:35.427471 IP AAA.BBB.CCC.228.49843 > AAA.BBB.CCC.200.25: S 2411757476:2411757476(0) win 5840 <mss 1460,sackok,timestamp="" 1076360598[|tcp]="">09:53:35.499524 IP DDD.EEE.FF.219.64428 > AAA.BBB.CCC.228.49843: S 2037136731:2037136731(0) ack 2411757477 win 5792 <mss 1460,sackok,timestamp="" 318182522[|tcp]="">09:53:35.499536 IP AAA.BBB.CCC.228.49843 > DDD.EEE.FF.219.64428: R 2411757477:2411757477(0) win 0

    From any other machine i have tested from thge telnet request connects
    [ I think what makes this a special case is that AAA.BBB.CCC.228 is a server we connect to (almost constantly) from our LAN, via WAN3]

    I have included the network setups as these ranges are neighbours - which throws up the possibility of me doing something wrong in the basic mask setup etc

    As far as I can see,

    AAA.BBB.CCC.228 connects to AAA.BBB.CCC.200 [ comes into pfsense on wan1 and goes to AAA.BBB.CCC.200 out of dmz1 ]
    AAA.BBB.CCC.200 replies to AAA.BBB.CCC.228 [ comes into pfsense on dmz1 and exits pfsense on wan3 [ why? ] getting natted on the adsl router attached to wan3  ]
    so AAA.BBB.CCC.228 doesn't see it as an ack from AAA.BBB.CCC.200

    I cant understand why the return path is through wan3 not wan1

    Am I right in thinking that the firewall rule allowing the inbound connection should route the reply back out the way it came in ??
    I am therefore presuming that this is taken care of by the rules on wan1 and the rules on dmz1 dont come in to it ??

    I am at a bit of a loss :

    I still think that there is a possibility that a (routing/firewall) rule matching a.b.c.d:m -> e.f.g.h:n maybe catching a.b.c.d:n -> e.f.g.h:m
    [ transposing the ports - as if the check is are do the ips match && do the ports match, rather than does the ip/port combo match ]
    however I imagine that this would cause more problems and would have been noticed : but then that is the point of BETA - so maybe thats the issue

    The only other thing that occurs is maybe i should be aliasing AAA.BBB.CCC.192 ( the network address for teh DMZ if you will ) onto WAN1 ??
    [ as it stands it just has 10.0.13.2 with the sdsl router on 10.0.13.1]

    Any thought would be appriciated,
    Sorry for the mammoth post

    TIA

    puge / DennisBagley</mss></mss></mss></mss>


Locked