MultiWAN / DMZ routing problem

DennisBagley

My setup - overview
–-----------------

Multi Wan - On a failover group for outbound from LAN - normally using WAN3 if up [ wan1 / 2 / 3 DSL with seperate routers and private xover lans to pfsense ]
DMZ - with public IP's arriving down WAN1 (cisco) then routed to pfsense and into DMZ ( based on default routing [ no pfsense fixed route ] )
LAN - ~20 users, outbound traffic generally routed through WAN group

The problem I have is this…

Mail server A - on the internet [ smtp heavily used from our LAN ]
Mail server B - in DMZ

If Mail server A tries to open an SMTP connection to Mail server B the following occurs…

As expected the connection (SYN) comes in on WAN1 and is routed to pfsense, which then [ based on WAN1 firewall rules ] allows it port 25 access to Mail server B

Mail server B then responds (SYN ACK) but the packet travels up WAN3 [getting natted on way out on adsl router] so upon arrival to Mail server A is ignored

As far as I can guess there is one of 3 problems here…

a - that due to all the existing connections TO port 25 on Mail server A (via wan3) pfsense routes the connection up wan3

OR -
b - the inbound path for some reason is not recorded [ the firewall policy that allowed the request ] so the return path is generated as if it is not an existing connection
OR -
c - I have misjudged what I should be able to do

I am currently running on 2 Beta / 2010-01-25 [i386]

I should also point out that from any other server the smtp connection is fine
[ which is what makes me think this is problem a ]
In which case i would suggest that maybe the sticky connections are causing the issue - and maybe the rule is transposing teh source and destination ports ?

Any help would be appriciated

TIA

pugelarouge

PS : Am aware this is BETA, and not suggested for production

DennisBagley

OK - so heres me from one of my live internet [as in colo - no dmz, no nat or anything] web servers

AAA.BBB.CCC.200 - my personal server in DMZ [behind pfsense - public IP routed - no natting - routed in via cisco sdsl router - then next fixed route to pfsense box on WAN1 ]
( AAA.BBB.CCC.200 has a gateway (DMZ1 on pfsense) at AAA.BBB.CCC.193 and a mask of 255.255.255.224 - so in range AAA.BBB.CCC.192 -> AAA.BBB.CCC.223 [ a /27 ] )

AAA.BBB.CCC.228 - a company server, on standard public IP in a data center
( AAA.BBB.CCC.228 has a gateway at AAA.BBB.CCC.225 and a mask of 255.255.255.240 - so in range AAA.BBB.CCC.224 -> AAA.BBB.CCC.239 [ a /28 ] )

DDD.EEE.FF.219 - the public IP of the adsl connected to wan3 on my pfsense box

From AAA.BBB.CCC.228 :

I fired off a 'telnet AAA.BBB.CCC.200 80'
captured with 'tcpdump -n' [ diag_packet_capture.php doesn't want to show any capture at the mo]

10:19:37.476767 IP AAA.BBB.CCC.228.49940 > AAA.BBB.CCC.200.80: S 2457225471:2457225471(0) win 5840 <mss 1460,sackok,timestamp="" 1076751111[|tcp]="">10:19:37.479894 IP DDD.EEE.FF.219.53622 > AAA.BBB.CCC.228.49940: S 712187932:712187932(0) ack 2457225472 win 5792 <mss 1460,sackok,timestamp="" 318338740[|tcp]="">10:19:37.479904 IP AAA.BBB.CCC.228.49940 > DDD.EEE.FF.219.53622: R 2457225472:2457225472(0) win 0

And again - From AAA.BBB.CCC.228 :

I fired off a 'telnet AAA.BBB.CCC.200 25'
captured with 'tcpdump -n' [ diag_packet_capture.php doesn't want to show any capture at the mo]

09:53:35.427471 IP AAA.BBB.CCC.228.49843 > AAA.BBB.CCC.200.25: S 2411757476:2411757476(0) win 5840 <mss 1460,sackok,timestamp="" 1076360598[|tcp]="">09:53:35.499524 IP DDD.EEE.FF.219.64428 > AAA.BBB.CCC.228.49843: S 2037136731:2037136731(0) ack 2411757477 win 5792 <mss 1460,sackok,timestamp="" 318182522[|tcp]="">09:53:35.499536 IP AAA.BBB.CCC.228.49843 > DDD.EEE.FF.219.64428: R 2411757477:2411757477(0) win 0

From any other machine i have tested from thge telnet request connects
[ I think what makes this a special case is that AAA.BBB.CCC.228 is a server we connect to (almost constantly) from our LAN, via WAN3]

I have included the network setups as these ranges are neighbours - which throws up the possibility of me doing something wrong in the basic mask setup etc

As far as I can see,

AAA.BBB.CCC.228 connects to AAA.BBB.CCC.200 [ comes into pfsense on wan1 and goes to AAA.BBB.CCC.200 out of dmz1 ]
AAA.BBB.CCC.200 replies to AAA.BBB.CCC.228 [ comes into pfsense on dmz1 and exits pfsense on wan3 [ why? ] getting natted on the adsl router attached to wan3 ]
so AAA.BBB.CCC.228 doesn't see it as an ack from AAA.BBB.CCC.200

I cant understand why the return path is through wan3 not wan1

Am I right in thinking that the firewall rule allowing the inbound connection should route the reply back out the way it came in ??
I am therefore presuming that this is taken care of by the rules on wan1 and the rules on dmz1 dont come in to it ??

I am at a bit of a loss :

I still think that there is a possibility that a (routing/firewall) rule matching a.b.c.d:m -> e.f.g.h:n maybe catching a.b.c.d:n -> e.f.g.h:m
[ transposing the ports - as if the check is are do the ips match && do the ports match, rather than does the ip/port combo match ]
however I imagine that this would cause more problems and would have been noticed : but then that is the point of BETA - so maybe thats the issue

The only other thing that occurs is maybe i should be aliasing AAA.BBB.CCC.192 ( the network address for teh DMZ if you will ) onto WAN1 ??
[ as it stands it just has 10.0.13.2 with the sdsl router on 10.0.13.1]

Any thought would be appriciated,
Sorry for the mammoth post

TIA

puge / DennisBagley</mss></mss></mss></mss>