Multi WAN - 2 sites - packets routing



  • ( Already posted on the forum FR)

    Hello everyone and thank you for your interest,

    I am trying to join two sites via pfSense , used as the main router, directing packets over 4 VPNs based on remote port requested.
    I use firewall rules to re route the packets on the various gateways.

    See images here attached.

    My question:
    A packet enters in the pfSense via the LAN on the SITE1, and seeks to achieve the network SITE2 on an HTTPS server. A rule says:
    From SITE1, anything that seeks to reach SITE2 on port 443 pass through the VPN1 .
    The packet goes well on VPN1 to the remote site (SITE2). (packet capture)

    BUT: the packet does not return via VPN1 to the SITE1, he always take the default gateway of the remote pfSense (SITE 2).

    It's the same in the other direction, whatever the chosen VPN.

    Then : If the link is behind the default gateway is down, nothing works. This does not at all with the goal is to better manage traffic and links it to the backup .

    Do you know the problem ? How to solve it ?

    I did that with packet tagging and debian before …

    Thank you for your expertise and help.





  • At the client end you are saying something like:
    source LAN1net, destination LAN2 port 443 gateway Link1

    Then on the pfSense at the other end, where the "servers" are with port 443 you will have to reverse the rule:
    source LAN2net port 443, destination LAN1, gateway Link1

    The client end that starts the connection to port 443 will have some "random" ephemeral port number.



  • Thank you for your answer!
    Of course, I did it. But here are the results.
    The packets still pass over the WAN by default …
    I do not understand why!

    Packet capture SITE 1 LINKSYS2 :
    18:48:46.091734 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 86: (tos 0x0, ttl 122, id 4478, offset 0, flags [DF], proto TCP (6), length 72)
        16.1.7.240.5900 > 172.31.100.196.49331: Flags [P.], cksum 0xc85e (correct), seq 3706019731:3706019763, ack 897777932, win 16251, length 32
    18:48:46.263957 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 122, id 4479, offset 0, flags [DF], proto TCP (6), length 40)
        16.1.7.240.5900 > 172.31.100.196.49331: Flags [.], cksum 0xfc9c (correct), seq 32, ack 11, win 16241, length 0
    18:48:48.092669 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 97: (tos 0x0, ttl 122, id 4480, offset 0, flags [DF], proto TCP (6), length 83)
        16.1.7.240.5900 > 172.31.100.196.49331: Flags [P.], cksum 0x10d5 (correct), seq 32:75, ack 11, win 16241, length 43
    18:48:48.275501 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 122, id 4481, offset 0, flags [DF], proto TCP (6), length 40)
        16.1.7.240.5900 > 172.31.100.196.49331: Flags [.], cksum 0xfc71 (correct), seq 75, ack 21, win 16231, length 0

    Packet capture SITE 2 LINKSYS2 :
    18:48:53.131360 00:22:6b:3a:7e:a4 > 00:1b:21:9e:d7:a1, ethertype IPv4 (0x0800), length 64: (tos 0x0, ttl 126, id 148, offset 0, flags [DF], proto TCP (6), length 50)
        172.31.100.196.49331 > 16.1.7.240.5900: Flags [P.], cksum 0x2e05 (correct), seq 897777982:897777992, ack 3706019967, win 258, length 10
    18:48:54.129169 00:22:6b:3a:7e:a4 > 00:1b:21:9e:d7:a1, ethertype IPv4 (0x0800), length 64: (tos 0x0, ttl 126, id 149, offset 0, flags [DF], proto TCP (6), length 50)
        172.31.100.196.49331 > 16.1.7.240.5900: Flags [P.], cksum 0x2ddb (correct), seq 10:20, ack 33, win 258, length 10

    BUT !!!!! :

    PACKET CAPTURE WAN DEFAUT GATEWAY SITE1 :
    18:48:46.091734 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 86: (tos 0x0, ttl 122, id 4478, offset 0, flags [DF], proto TCP (6), length 72)
        16.1.7.240.5900 > 172.31.100.196.49331: Flags [P.], cksum 0xc85e (correct), seq 3706019731:3706019763, ack 897777932, win 16251, length 32
    18:48:46.263957 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 122, id 4479, offset 0, flags [DF], proto TCP (6), length 40)
        16.1.7.240.5900 > 172.31.100.196.49331: Flags [.], cksum 0xfc9c (correct), seq 32, ack 11, win 16241, length 0
    18:48:48.092669 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 97: (tos 0x0, ttl 122, id 4480, offset 0, flags [DF], proto TCP (6), length 83)
        16.1.7.240.5900 > 172.31.100.196.49331: Flags [P.], cksum 0x10d5 (correct), seq 32:75, ack 11, win 16241, length 43
    18:48:48.275501 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 122, id 4481, offset 0, flags [DF], proto TCP (6), length 40)
        16.1.7.240.5900 > 172.31.100.196.49331: Flags [.], cksum 0xfc71 (correct), seq 75, ack 21, win 16231, length 0

    PACKET CAPTURE WAN DEFAUT GATEWAY SITE2 :
    18:48:53.131360 00:22:6b:3a:7e:a4 > 00:1b:21:9e:d7:a1, ethertype IPv4 (0x0800), length 64: (tos 0x0, ttl 126, id 148, offset 0, flags [DF], proto TCP (6), length 50)
        172.31.100.196.49331 > 16.1.7.240.5900: Flags [P.], cksum 0x2e05 (correct), seq 897777982:897777992, ack 3706019967, win 258, length 10
    18:48:54.129169 00:22:6b:3a:7e:a4 > 00:1b:21:9e:d7:a1, ethertype IPv4 (0x0800), length 64: (tos 0x0, ttl 126, id 149, offset 0, flags [DF], proto TCP (6), length 50)
        172.31.100.196.49331 > 16.1.7.240.5900: Flags [P.], cksum 0x2ddb (correct), seq 10:20, ack 33, win 258, length 10

    The rules are the picture …




  • The source and destination look around the wrong way. In the capture at site 1 the packet has source 16.1.7.240.5900 so the rule on LAN at site 1 has to have source IP 16.1.7.240 port 5900, destination other LAN port any.



  • HOOOO !!! Sorry for the mistake …
    I inverted the site in my resume ...

    Please read :
    Packet capture SITE 2 LINKSYS2 :
    18:48:46.091734 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 86: (tos 0x0, ttl 122, id 4478, offset 0, flags [DF], proto TCP (6), length 72)
        16.1.7.240.5900 > 172.31.100.196.49331: Flags [P.], cksum 0xc85e (correct), seq 3706019731:3706019763, ack 897777932, win 16251, length 32
    18:48:46.263957 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 122, id 4479, offset 0, flags [DF], proto TCP (6), length 40)
        16.1.7.240.5900 > 172.31.100.196.49331: Flags [.], cksum 0xfc9c (correct), seq 32, ack 11, win 16241, length 0
    18:48:48.092669 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 97: (tos 0x0, ttl 122, id 4480, offset 0, flags [DF], proto TCP (6), length 83)
        16.1.7.240.5900 > 172.31.100.196.49331: Flags [P.], cksum 0x10d5 (correct), seq 32:75, ack 11, win 16241, length 43
    18:48:48.275501 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 122, id 4481, offset 0, flags [DF], proto TCP (6), length 40)
        16.1.7.240.5900 > 172.31.100.196.49331: Flags [.], cksum 0xfc71 (correct), seq 75, ack 21, win 16231, length 0

    Packet capture SITE 1 LINKSYS2 :
    18:48:53.131360 00:22:6b:3a:7e:a4 > 00:1b:21:9e:d7:a1, ethertype IPv4 (0x0800), length 64: (tos 0x0, ttl 126, id 148, offset 0, flags [DF], proto TCP (6), length 50)
        172.31.100.196.49331 > 16.1.7.240.5900: Flags [P.], cksum 0x2e05 (correct), seq 897777982:897777992, ack 3706019967, win 258, length 10
    18:48:54.129169 00:22:6b:3a:7e:a4 > 00:1b:21:9e:d7:a1, ethertype IPv4 (0x0800), length 64: (tos 0x0, ttl 126, id 149, offset 0, flags [DF], proto TCP (6), length 50)
        172.31.100.196.49331 > 16.1.7.240.5900: Flags [P.], cksum 0x2ddb (correct), seq 10:20, ack 33, win 258, length 10

    BUT !!!!! :

    PACKET CAPTURE WAN DEFAUT GATEWAY SITE2 :
    18:48:46.091734 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 86: (tos 0x0, ttl 122, id 4478, offset 0, flags [DF], proto TCP (6), length 72)
        16.1.7.240.5900 > 172.31.100.196.49331: Flags [P.], cksum 0xc85e (correct), seq 3706019731:3706019763, ack 897777932, win 16251, length 32
    18:48:46.263957 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 122, id 4479, offset 0, flags [DF], proto TCP (6), length 40)
        16.1.7.240.5900 > 172.31.100.196.49331: Flags [.], cksum 0xfc9c (correct), seq 32, ack 11, win 16241, length 0
    18:48:48.092669 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 97: (tos 0x0, ttl 122, id 4480, offset 0, flags [DF], proto TCP (6), length 83)
        16.1.7.240.5900 > 172.31.100.196.49331: Flags [P.], cksum 0x10d5 (correct), seq 32:75, ack 11, win 16241, length 43
    18:48:48.275501 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 122, id 4481, offset 0, flags [DF], proto TCP (6), length 40)
        16.1.7.240.5900 > 172.31.100.196.49331: Flags [.], cksum 0xfc71 (correct), seq 75, ack 21, win 16231, length 0

    PACKET CAPTURE WAN DEFAUT GATEWAY SITE1 :
    18:48:53.131360 00:22:6b:3a:7e:a4 > 00:1b:21:9e:d7:a1, ethertype IPv4 (0x0800), length 64: (tos 0x0, ttl 126, id 148, offset 0, flags [DF], proto TCP (6), length 50)
        172.31.100.196.49331 > 16.1.7.240.5900: Flags [P.], cksum 0x2e05 (correct), seq 897777982:897777992, ack 3706019967, win 258, length 10
    18:48:54.129169 00:22:6b:3a:7e:a4 > 00:1b:21:9e:d7:a1, ethertype IPv4 (0x0800), length 64: (tos 0x0, ttl 126, id 149, offset 0, flags [DF], proto TCP (6), length 50)
        172.31.100.196.49331 > 16.1.7.240.5900: Flags [P.], cksum 0x2ddb (correct), seq 10:20, ack 33, win 258, length 10



  • The rules should be on Site1 LAN and Site2 LAN - are they?
    Are there other rules above those?
    If so, what are they? They might be matching first.



  • Thank's for your question.

    You'll find attach all the rules on SITE1 and SITE2

    On SITE1 PFSENSE is in PRODUCTION. On SITE2 PFSENSE is in TEST mode …

    Many thanks for you help and interest.

    Regards.






  • Hi all,

    Alway the same pb.

    I made a change : no nat when packet use the ADSL VPN.

    The systeme worked during an half day. This moning after the reboot of the PFSENSE, with no change … it don't work ...

    I joint here a schem of the install ...




  • do you "own" the 16.1.7.0/24 subnet? (it's not in private address space)

    CDIR
    16.0.0.0/8
    Country Code
    US
    Registry
    arin
    Registration Date
    18/05/1989
    ASN Name
    HP-INTERNET-AS Hewlett-Packard Company



  • Hi,

    Thank's for your question.
    No, it's only an OOOOLLLLDDDD (1985) lan. And there is many old services on it (as400 for ex).
    I know i'm not in the rfc … but at this time it's not possible to change that.

    Regards.



  • Thank for interest,
    No , not possible to change that … at this time.
    The problem remains unsolved for me, and becomes critical (meaning death) : 3 months late ...
    but I do not have much time right now (and it takes time to test all this ... ).
    But i've read many and many docs about pfsense and route ... it appear that i'm not isolated. The defaut route for return packet seem to be discus in vpn and group gateway ...

    Actualy, I'm looking around NAT (apparently, for the return packet, pfsense take the first NAT rule that match -> defaut interface ...).
    Too : I'm looking around Floating Rules, with the Quick option ...
    And too : grouping my ADSL gateways and mount a vpn on it between my to sites ...

    If anyone has a similar architecture to that I want to put in place, thank you to tell me, it works with me ...



  • Hello everyone ,

    Yep , I am clinging to my topic !!!  :P

    My latest tests and my conclusion :

    I have been testing various protocol and it turns out they do not react the same way .

    As I have said above , ICMP is left of those below :

    With a ping, ICMP packets do well by the designated interface on the firewall 1 ICMP rule to the firewall 2 via the well VPN link and come on site 2, then arrive at the destination machine , leave and resume … well take the route specified in the rule of pfSense 2 Site 2 and finally arrive destination on the site 1 has taking good VPN link !

    (remember, for all that is not ICMP, the return is ALWAYS ON DEFAULT GATEWAY on the distant PFSENSE !).
    ICMP -> Layer 3 of the OSI model max -> no no FLAG TCP ACK !!!!

    In fact : pfSense returns the default interface ALL PACKTES marked ACK ! ( when a packet traverses a rule it is tagged by pfSense ACK).

    So new question for advanced users : how to solve my problem knowing that, without mounting a gas plant with floatings rules and proxy ... ???

    Thank's for your help and interest.

    Hope some people will be interest by the challenge !


Log in to reply