Active/active inbound routing – return path blocked ?

[bEsTiAn]

Hello,

Not sure I should put this here or in CARP sub-forum area…

Quick summary :
-- two WANs
-- two pfsense boxes (1.2.3) with one IP per ISP each
-- two servers behind the firewalls
-- CARP interface behind the two pfsense boxes so that my servers can have outbound connectivity

basic usage : public DNS for some names I host, or inbound SMTP traffic or web sites.

--> I would like to have inbound traffic on both pfsense, on both ISP, at same time.

First example :
Internet user does DNS lookup on WAN#1,IP#1 it comes to pfsense #1, NIC #1, and is routed to server #1
Return traffic is going to CARP master, which is pfsense #1, so return path is OK and DNS query is responded.

Second example :
Internet user does DNS lookup on WAN#2,IP#2 it comes to pfsense #2, NIC #2, and is router to server #2
Return traffic is going to CARP master, which is pfsense #1, so return path looks like being blocked, as DNS query is not responded ?

Third example :
Internet user does DNS lookup on WAN#2,IP#2 it comes to pfsense #2, NIC #2, and is router to server #2
I enabled outbound NAT on the internal interface, therefore the server behind the firewall does not use its gateway to respond.
Return traffic is going to pfsense #2, so return path is OK and DNS query is responded.

I have this using DNS, HTTP, SMTP, etc...
The most visible one is when I open SSH through Internet to the servers.
The session opens, stays opened for a couple of seconds, then is cut.

So, am I missing anything ? CARP sync is enabled (not the rules etc, but plain firewall table state), so I would have assumed that TCP sessions and therefore current communications are opened in both firewalls. Is there something I am doing wrong or is this a "expected behavior" with pfsense 1.2.3 ? Or is the issue at another level, on the ISP's router in front of the pfsense boxes ? Could it be solved somehow  ? Would using OpenBGP of any use in this, avoiding using CARP for my servers to reach the internet ? Or any other routing means for them ?

Thanks a lot for your time,

Guillaume

[bEsTiAn]

Replying to myself…
After having thought a bit more on how I wish my design to function, I realized that I need vIP (CARP) on each interface. That's fine.
Quite normal : internet client wants to talk to IP #1, it's not expecting a response from IP #2.

So, design v2 :
vIP#1 --> pfsense #1, ISP#1 master
vIP#2 --> pfsense #1, ISP#2 master
vIP#3 --> pfsense #2, ISP#1 master
vIP#4 --> pfsense #2, ISP#2 master
(the other box being the passive of each master vice-versa)

inbound nat is (sample):
vIP#1 TCP 80 (dst) --> server #1
vIP#2 TCP 80 (dst) --> server #1
vIP#3 TCP 80 (dst) --> server #2
vIP#4 TCP 80 (dst) --> server #2

outbound nat is (following same sample):
server #1 TCP 80 (src) --> WAN NIC #1 --> vIP#1
server #1 TCP 80 (src) --> WAN NIC #2 --> vIP#2
server #2 TCP 80 (src) --> WAN NIC #1 --> vIP#3
server #2 TCP 80 (src) --> WAN NIC #2 --> vIP#4
this being the same rules on both boxes

So, to give a practical example :

• client wants to browse to vIP#3
• reaches pfsense box #2 on WAN#1
• translated to server #2
• server #2 replies through pfsense #1 (master of LAN vIP)
• server #2's reply is through WAN NIC#1 as it's a TCP state already set in the state table (am I right here ?)
• outbound NAT as vIP#3 since it's server #2 and it is on WAN NIC#1

However I need to add a reverse rule on the LAN interface, allowing traffic originating from the server on the TCP 80 as src.

I need to test this further later on when I have enough vIP available on my secondary ISP (and there is another problem there, as they use static ARP entries in their systems... But that's another story), as currently I test on my primary ISP which is the default WAN for my pfsense firewalls. I will reply back here once it is confirmed as working on both WANs.

In the meantime, a question :
is this normal that the reverse rule is needed to be set up in the firewalls ? pfsync does not sync that info on the other nodes ?

Guillaume