Active/active inbound routing – return path blocked ?



  • Hello,

    Not sure I should put this here or in CARP sub-forum area…

    Quick summary :
    -- two WANs
    -- two pfsense boxes (1.2.3) with one IP per ISP each
    -- two servers behind the firewalls
    -- CARP interface behind the two pfsense boxes so that my servers can have outbound connectivity

    basic usage : public DNS for some names I host, or inbound SMTP traffic or web sites.

    --> I would like to have inbound traffic on both pfsense, on both ISP, at same time.

    First example :
    Internet user does DNS lookup on WAN#1,IP#1 it comes to pfsense #1, NIC #1, and is routed to server #1
    Return traffic is going to CARP master, which is pfsense #1, so return path is OK and DNS query is responded.

    Second example :
    Internet user does DNS lookup on WAN#2,IP#2 it comes to pfsense #2, NIC #2, and is router to server #2
    Return traffic is going to CARP master, which is pfsense #1, so return path looks like being blocked, as DNS query is not responded ?

    Third example :
    Internet user does DNS lookup on WAN#2,IP#2 it comes to pfsense #2, NIC #2, and is router to server #2
    I enabled outbound NAT on the internal interface, therefore the server behind the firewall does not use its gateway to respond.
    Return traffic is going to pfsense #2, so return path is OK and DNS query is responded.

    I have this using DNS, HTTP, SMTP, etc...
    The most visible one is when I open SSH through Internet to the servers.
    The session opens, stays opened for a couple of seconds, then is cut.

    So, am I missing anything ? CARP sync is enabled (not the rules etc, but plain firewall table state), so I would have assumed that TCP sessions and therefore current communications are opened in both firewalls. Is there something I am doing wrong or is this a "expected behavior" with pfsense 1.2.3 ? Or is the issue at another level, on the ISP's router in front of the pfsense boxes ? Could it be solved somehow  ? Would using OpenBGP of any use in this, avoiding using CARP for my servers to reach the internet ? Or any other routing means for them ?

    Thanks a lot for your time,

    Guillaume



  • Replying to myself…
    After having thought a bit more on how I wish my design to function, I realized that I need vIP (CARP) on each interface. That's fine.
    Quite normal : internet client wants to talk to IP #1, it's not expecting a response from IP #2.

    So, design v2 :
    vIP#1 --> pfsense #1, ISP#1 master
    vIP#2 --> pfsense #1, ISP#2 master
    vIP#3 --> pfsense #2, ISP#1 master
    vIP#4 --> pfsense #2, ISP#2 master
    (the other box being the passive of each master vice-versa)

    inbound nat is (sample):
    vIP#1 TCP 80 (dst) --> server #1
    vIP#2 TCP 80 (dst) --> server #1
    vIP#3 TCP 80 (dst) --> server #2
    vIP#4 TCP 80 (dst) --> server #2

    outbound nat is (following same sample):
    server #1 TCP 80 (src) --> WAN NIC #1 --> vIP#1
    server #1 TCP 80 (src) --> WAN NIC #2 --> vIP#2
    server #2 TCP 80 (src) --> WAN NIC #1 --> vIP#3
    server #2 TCP 80 (src) --> WAN NIC #2 --> vIP#4
    this being the same rules on both boxes

    So, to give a practical example :

    • client wants to browse to vIP#3
    • reaches pfsense box #2 on WAN#1
    • translated to server #2
    • server #2 replies through pfsense #1 (master of LAN vIP)
    • server #2's reply is through WAN NIC#1 as it's a TCP state already set in the state table (am I right here ?)
    • outbound NAT as vIP#3 since it's server #2 and it is on WAN NIC#1

    However I need to add a reverse rule on the LAN interface, allowing traffic originating from the server on the TCP 80 as src.

    I need to test this further later on when I have enough vIP available on my secondary ISP (and there is another problem there, as they use static ARP entries in their systems... But that's another story), as currently I test on my primary ISP which is the default WAN for my pfsense firewalls. I will reply back here once it is confirmed as working on both WANs.

    In the meantime, a question :
    is this normal that the reverse rule is needed to be set up in the firewalls ? pfsync does not sync that info on the other nodes ?

    Guillaume


Log in to reply