[SOLVED]Multi WAN, Single LAN,cant split webserver on WAN1, email server on WAN2

[gb]

[SOLVED] See last post.  Summary:  Make sure the WAN interface configurations have a gateway set.

Hi Everyone,
I thought I had a simple design.  I have 2 wan interfaces, lets call them WAN1 and WAN2, and single LAN.
On the LAN, I have a webserver and an email server.

I have a default gateway out WAN1.
I have advanced outbound NAT enabled.

I am publishing TCP port 80 on a virtaul IP on the WAN1 subnet range using a port-forward.  Works great.  All web server traffic comes and goes on WAN1.
I am publishing TCP port 25 on a virtual IP on the WAN2 subnet range using a port-forward (also tried 1:1) expecting all email server traffic to come and go on the WAN2 interface and corresponding link - but the response goes out the WAN1 interface using the WAN2 Virtual IP as a source ???  (And consequently doesn't work at all).

More detail:
I have packet captured the WAN2 conversation on each interface of the PF host, and can clearly see that internal email server has responded to the intial SYN but the response can only be seen egressing PF on the WAN1 interface, not the originating inbound WAN2 interface.  Furthermore,  the response that goes out the WAN1 interface has the WAN2 Virtual IP as the source.     The ethernet frame for this response that egresses WAN1 has the destination MAC address correct for WAN1 - i.e At layer 3 the packet is correct if it was sent out WAN2, but PF has chosen WAN1 and the WAN1 gateway for some reason.

If I monitor the state table it eventually has a state of SYN_SENT:ESTABLISHED appear, so it seems fairly happy to build this (broken) state using this asymetric path. It never progresses past this state.

I have rebooted, cleared states, started over, but I can't make it work.

Why would PF chose to send the packet out the WAN1 interface when the inbound IP alias and portforward was on WAN2?

Any ideas welcome - I'm sure I have missed something pretty basic  :)

Thanks,

GB

[heper]

so you have disable automatic rule generation for NAT ?
what problems did you have with letting pfsense manage NAT automagically ?

can you post a screenshot of your NAT rules ? This looks like a NAT issue to me.

mvg

Jeroen

[gb]

Thanks heper for having a look.

I have opted for Advanced NAT because I need to in the future have different outbound IP mappings for different services.  A generic 'everything out one interface' wasn't going to work.

Screenshots are attached.

Thanks again,

GB.

[heper]

personally, i can not figure out what you are trying to accomplish with your gateway & static routes without further details.
do you know that static routes shouldn't be used for any directly attached network right ? (only in very special cases this is not true)
could you sketch us a basic network diagram that could explain what the static routes & gateways do and 'where' they are located?

Also if you could provide screenshots of the interface config,NAT,VIP, Firewall rules  - it would make it easier to figure out the problem.

kind regards

[gb]

Heh - sorry about that - I had included both images to upload but looks like only the last one did..

I will upload a network diagram now and the NAT config as well.

[gb]

here is the NAT config

[gb]

..and some interface config and aliases..

Thanks for any help!

[gb]

Solved!

Comparing the output of pfctl -sr ,  I notice that my other PFSense deployments have the 'reply-to' function set for similar inbound NATs.

In this deploment my rules don't have it, despite the System->Advanced->Firewall/NAT->"Disable reply-to" being unticked.

A quick search of this forum and I found the fix here:
http://forum.pfsense.org/index.php/topic,35758.0.html

It was certainly true that for the newest WAN link (WAN2) I didn't have a gateway set.  After setting the gateway the pfctl -sr | grep (rule) changed

from:

[2.0.1-RELEASE][admin@firewall.local]/root(78): pfctl -sr | grep 192.168.0.40
pass in log quick on em6 inet proto tcp from any to 192.168.0.40 port = https flags S/SA keep state label "USER_RULE: Allow world to Webmail"

to

[2.0.1-RELEASE][admin@firewall.local]/root(79): pfctl -sr | grep 192.168.0.40
pass in log quick on em6 reply-to (em6 2.2.2.2) inet proto tcp from any to 192.168.0.40 port = https flags S/SA keep state label "USER_RULE: Allow world to Webmail"

…and bingo,  all good!

Thanks for your help heper!

GB

[heper]

i didn't actually do anything … glad to be of service anyways :D