Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Multi WAN - 2 sites - packets routing

    Scheduled Pinned Locked Moved Routing and Multi WAN
    12 Posts 3 Posters 1.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P Offline
      phil.davis
      last edited by

      At the client end you are saying something like:
      source LAN1net, destination LAN2 port 443 gateway Link1

      Then on the pfSense at the other end, where the "servers" are with port 443 you will have to reverse the rule:
      source LAN2net port 443, destination LAN1, gateway Link1

      The client end that starts the connection to port 443 will have some "random" ephemeral port number.

      As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
      If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

      1 Reply Last reply Reply Quote 0
      • Z Offline
        zoummuoz
        last edited by

        Thank you for your answer!
        Of course, I did it. But here are the results.
        The packets still pass over the WAN by default …
        I do not understand why!

        Packet capture SITE 1 LINKSYS2 :
        18:48:46.091734 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 86: (tos 0x0, ttl 122, id 4478, offset 0, flags [DF], proto TCP (6), length 72)
            16.1.7.240.5900 > 172.31.100.196.49331: Flags [P.], cksum 0xc85e (correct), seq 3706019731:3706019763, ack 897777932, win 16251, length 32
        18:48:46.263957 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 122, id 4479, offset 0, flags [DF], proto TCP (6), length 40)
            16.1.7.240.5900 > 172.31.100.196.49331: Flags [.], cksum 0xfc9c (correct), seq 32, ack 11, win 16241, length 0
        18:48:48.092669 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 97: (tos 0x0, ttl 122, id 4480, offset 0, flags [DF], proto TCP (6), length 83)
            16.1.7.240.5900 > 172.31.100.196.49331: Flags [P.], cksum 0x10d5 (correct), seq 32:75, ack 11, win 16241, length 43
        18:48:48.275501 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 122, id 4481, offset 0, flags [DF], proto TCP (6), length 40)
            16.1.7.240.5900 > 172.31.100.196.49331: Flags [.], cksum 0xfc71 (correct), seq 75, ack 21, win 16231, length 0

        Packet capture SITE 2 LINKSYS2 :
        18:48:53.131360 00:22:6b:3a:7e:a4 > 00:1b:21:9e:d7:a1, ethertype IPv4 (0x0800), length 64: (tos 0x0, ttl 126, id 148, offset 0, flags [DF], proto TCP (6), length 50)
            172.31.100.196.49331 > 16.1.7.240.5900: Flags [P.], cksum 0x2e05 (correct), seq 897777982:897777992, ack 3706019967, win 258, length 10
        18:48:54.129169 00:22:6b:3a:7e:a4 > 00:1b:21:9e:d7:a1, ethertype IPv4 (0x0800), length 64: (tos 0x0, ttl 126, id 149, offset 0, flags [DF], proto TCP (6), length 50)
            172.31.100.196.49331 > 16.1.7.240.5900: Flags [P.], cksum 0x2ddb (correct), seq 10:20, ack 33, win 258, length 10

        BUT !!!!! :

        PACKET CAPTURE WAN DEFAUT GATEWAY SITE1 :
        18:48:46.091734 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 86: (tos 0x0, ttl 122, id 4478, offset 0, flags [DF], proto TCP (6), length 72)
            16.1.7.240.5900 > 172.31.100.196.49331: Flags [P.], cksum 0xc85e (correct), seq 3706019731:3706019763, ack 897777932, win 16251, length 32
        18:48:46.263957 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 122, id 4479, offset 0, flags [DF], proto TCP (6), length 40)
            16.1.7.240.5900 > 172.31.100.196.49331: Flags [.], cksum 0xfc9c (correct), seq 32, ack 11, win 16241, length 0
        18:48:48.092669 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 97: (tos 0x0, ttl 122, id 4480, offset 0, flags [DF], proto TCP (6), length 83)
            16.1.7.240.5900 > 172.31.100.196.49331: Flags [P.], cksum 0x10d5 (correct), seq 32:75, ack 11, win 16241, length 43
        18:48:48.275501 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 122, id 4481, offset 0, flags [DF], proto TCP (6), length 40)
            16.1.7.240.5900 > 172.31.100.196.49331: Flags [.], cksum 0xfc71 (correct), seq 75, ack 21, win 16231, length 0

        PACKET CAPTURE WAN DEFAUT GATEWAY SITE2 :
        18:48:53.131360 00:22:6b:3a:7e:a4 > 00:1b:21:9e:d7:a1, ethertype IPv4 (0x0800), length 64: (tos 0x0, ttl 126, id 148, offset 0, flags [DF], proto TCP (6), length 50)
            172.31.100.196.49331 > 16.1.7.240.5900: Flags [P.], cksum 0x2e05 (correct), seq 897777982:897777992, ack 3706019967, win 258, length 10
        18:48:54.129169 00:22:6b:3a:7e:a4 > 00:1b:21:9e:d7:a1, ethertype IPv4 (0x0800), length 64: (tos 0x0, ttl 126, id 149, offset 0, flags [DF], proto TCP (6), length 50)
            172.31.100.196.49331 > 16.1.7.240.5900: Flags [P.], cksum 0x2ddb (correct), seq 10:20, ack 33, win 258, length 10

        The rules are the picture …

        pfsense1.jpg
        pfsense1.jpg_thumb

        1 Reply Last reply Reply Quote 0
        • P Offline
          phil.davis
          last edited by

          The source and destination look around the wrong way. In the capture at site 1 the packet has source 16.1.7.240.5900 so the rule on LAN at site 1 has to have source IP 16.1.7.240 port 5900, destination other LAN port any.

          As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
          If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

          1 Reply Last reply Reply Quote 0
          • Z Offline
            zoummuoz
            last edited by

            HOOOO !!! Sorry for the mistake …
            I inverted the site in my resume ...

            Please read :
            Packet capture SITE 2 LINKSYS2 :
            18:48:46.091734 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 86: (tos 0x0, ttl 122, id 4478, offset 0, flags [DF], proto TCP (6), length 72)
                16.1.7.240.5900 > 172.31.100.196.49331: Flags [P.], cksum 0xc85e (correct), seq 3706019731:3706019763, ack 897777932, win 16251, length 32
            18:48:46.263957 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 122, id 4479, offset 0, flags [DF], proto TCP (6), length 40)
                16.1.7.240.5900 > 172.31.100.196.49331: Flags [.], cksum 0xfc9c (correct), seq 32, ack 11, win 16241, length 0
            18:48:48.092669 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 97: (tos 0x0, ttl 122, id 4480, offset 0, flags [DF], proto TCP (6), length 83)
                16.1.7.240.5900 > 172.31.100.196.49331: Flags [P.], cksum 0x10d5 (correct), seq 32:75, ack 11, win 16241, length 43
            18:48:48.275501 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 122, id 4481, offset 0, flags [DF], proto TCP (6), length 40)
                16.1.7.240.5900 > 172.31.100.196.49331: Flags [.], cksum 0xfc71 (correct), seq 75, ack 21, win 16231, length 0

            Packet capture SITE 1 LINKSYS2 :
            18:48:53.131360 00:22:6b:3a:7e:a4 > 00:1b:21:9e:d7:a1, ethertype IPv4 (0x0800), length 64: (tos 0x0, ttl 126, id 148, offset 0, flags [DF], proto TCP (6), length 50)
                172.31.100.196.49331 > 16.1.7.240.5900: Flags [P.], cksum 0x2e05 (correct), seq 897777982:897777992, ack 3706019967, win 258, length 10
            18:48:54.129169 00:22:6b:3a:7e:a4 > 00:1b:21:9e:d7:a1, ethertype IPv4 (0x0800), length 64: (tos 0x0, ttl 126, id 149, offset 0, flags [DF], proto TCP (6), length 50)
                172.31.100.196.49331 > 16.1.7.240.5900: Flags [P.], cksum 0x2ddb (correct), seq 10:20, ack 33, win 258, length 10

            BUT !!!!! :

            PACKET CAPTURE WAN DEFAUT GATEWAY SITE2 :
            18:48:46.091734 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 86: (tos 0x0, ttl 122, id 4478, offset 0, flags [DF], proto TCP (6), length 72)
                16.1.7.240.5900 > 172.31.100.196.49331: Flags [P.], cksum 0xc85e (correct), seq 3706019731:3706019763, ack 897777932, win 16251, length 32
            18:48:46.263957 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 122, id 4479, offset 0, flags [DF], proto TCP (6), length 40)
                16.1.7.240.5900 > 172.31.100.196.49331: Flags [.], cksum 0xfc9c (correct), seq 32, ack 11, win 16241, length 0
            18:48:48.092669 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 97: (tos 0x0, ttl 122, id 4480, offset 0, flags [DF], proto TCP (6), length 83)
                16.1.7.240.5900 > 172.31.100.196.49331: Flags [P.], cksum 0x10d5 (correct), seq 32:75, ack 11, win 16241, length 43
            18:48:48.275501 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 122, id 4481, offset 0, flags [DF], proto TCP (6), length 40)
                16.1.7.240.5900 > 172.31.100.196.49331: Flags [.], cksum 0xfc71 (correct), seq 75, ack 21, win 16231, length 0

            PACKET CAPTURE WAN DEFAUT GATEWAY SITE1 :
            18:48:53.131360 00:22:6b:3a:7e:a4 > 00:1b:21:9e:d7:a1, ethertype IPv4 (0x0800), length 64: (tos 0x0, ttl 126, id 148, offset 0, flags [DF], proto TCP (6), length 50)
                172.31.100.196.49331 > 16.1.7.240.5900: Flags [P.], cksum 0x2e05 (correct), seq 897777982:897777992, ack 3706019967, win 258, length 10
            18:48:54.129169 00:22:6b:3a:7e:a4 > 00:1b:21:9e:d7:a1, ethertype IPv4 (0x0800), length 64: (tos 0x0, ttl 126, id 149, offset 0, flags [DF], proto TCP (6), length 50)
                172.31.100.196.49331 > 16.1.7.240.5900: Flags [P.], cksum 0x2ddb (correct), seq 10:20, ack 33, win 258, length 10

            1 Reply Last reply Reply Quote 0
            • P Offline
              phil.davis
              last edited by

              The rules should be on Site1 LAN and Site2 LAN - are they?
              Are there other rules above those?
              If so, what are they? They might be matching first.

              As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
              If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

              1 Reply Last reply Reply Quote 0
              • Z Offline
                zoummuoz
                last edited by

                Thank's for your question.

                You'll find attach all the rules on SITE1 and SITE2

                On SITE1 PFSENSE is in PRODUCTION. On SITE2 PFSENSE is in TEST mode …

                Many thanks for you help and interest.

                Regards.

                Pfsense_Site_1.jpg_thumb
                Pfsense_Site_1.jpg
                Pfsense_Site_2.jpg_thumb
                Pfsense_Site_2.jpg

                1 Reply Last reply Reply Quote 0
                • Z Offline
                  zoummuoz
                  last edited by

                  Hi all,

                  Alway the same pb.

                  I made a change : no nat when packet use the ADSL VPN.

                  The systeme worked during an half day. This moning after the reboot of the PFSENSE, with no change … it don't work ...

                  I joint here a schem of the install ...

                  vpns.png
                  vpns.png_thumb

                  1 Reply Last reply Reply Quote 0
                  • H Offline
                    heper
                    last edited by

                    do you "own" the 16.1.7.0/24 subnet? (it's not in private address space)

                    CDIR
                    16.0.0.0/8
                    Country Code
                    US
                    Registry
                    arin
                    Registration Date
                    18/05/1989
                    ASN Name
                    HP-INTERNET-AS Hewlett-Packard Company

                    1 Reply Last reply Reply Quote 0
                    • Z Offline
                      zoummuoz
                      last edited by

                      Hi,

                      Thank's for your question.
                      No, it's only an OOOOLLLLDDDD (1985) lan. And there is many old services on it (as400 for ex).
                      I know i'm not in the rfc … but at this time it's not possible to change that.

                      Regards.

                      1 Reply Last reply Reply Quote 0
                      • Z Offline
                        zoummuoz
                        last edited by

                        Thank for interest,
                        No , not possible to change that … at this time.
                        The problem remains unsolved for me, and becomes critical (meaning death) : 3 months late ...
                        but I do not have much time right now (and it takes time to test all this ... ).
                        But i've read many and many docs about pfsense and route ... it appear that i'm not isolated. The defaut route for return packet seem to be discus in vpn and group gateway ...

                        Actualy, I'm looking around NAT (apparently, for the return packet, pfsense take the first NAT rule that match -> defaut interface ...).
                        Too : I'm looking around Floating Rules, with the Quick option ...
                        And too : grouping my ADSL gateways and mount a vpn on it between my to sites ...

                        If anyone has a similar architecture to that I want to put in place, thank you to tell me, it works with me ...

                        1 Reply Last reply Reply Quote 0
                        • Z Offline
                          zoummuoz
                          last edited by

                          Hello everyone ,

                          Yep , I am clinging to my topic !!!  :P

                          My latest tests and my conclusion :

                          I have been testing various protocol and it turns out they do not react the same way .

                          As I have said above , ICMP is left of those below :

                          With a ping, ICMP packets do well by the designated interface on the firewall 1 ICMP rule to the firewall 2 via the well VPN link and come on site 2, then arrive at the destination machine , leave and resume … well take the route specified in the rule of pfSense 2 Site 2 and finally arrive destination on the site 1 has taking good VPN link !

                          (remember, for all that is not ICMP, the return is ALWAYS ON DEFAULT GATEWAY on the distant PFSENSE !).
                          ICMP -> Layer 3 of the OSI model max -> no no FLAG TCP ACK !!!!

                          In fact : pfSense returns the default interface ALL PACKTES marked ACK ! ( when a packet traverses a rule it is tagged by pfSense ACK).

                          So new question for advanced users : how to solve my problem knowing that, without mounting a gas plant with floatings rules and proxy ... ???

                          Thank's for your help and interest.

                          Hope some people will be interest by the challenge !

                          1 Reply Last reply Reply Quote 0
                          • First post
                            Last post
                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.