Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Multi WAN - 2 sites - packets routing

    Scheduled Pinned Locked Moved Routing and Multi WAN
    12 Posts 3 Posters 1.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Z Offline
      zoummuoz
      last edited by

      Thank you for your answer!
      Of course, I did it. But here are the results.
      The packets still pass over the WAN by default …
      I do not understand why!

      Packet capture SITE 1 LINKSYS2 :
      18:48:46.091734 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 86: (tos 0x0, ttl 122, id 4478, offset 0, flags [DF], proto TCP (6), length 72)
          16.1.7.240.5900 > 172.31.100.196.49331: Flags [P.], cksum 0xc85e (correct), seq 3706019731:3706019763, ack 897777932, win 16251, length 32
      18:48:46.263957 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 122, id 4479, offset 0, flags [DF], proto TCP (6), length 40)
          16.1.7.240.5900 > 172.31.100.196.49331: Flags [.], cksum 0xfc9c (correct), seq 32, ack 11, win 16241, length 0
      18:48:48.092669 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 97: (tos 0x0, ttl 122, id 4480, offset 0, flags [DF], proto TCP (6), length 83)
          16.1.7.240.5900 > 172.31.100.196.49331: Flags [P.], cksum 0x10d5 (correct), seq 32:75, ack 11, win 16241, length 43
      18:48:48.275501 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 122, id 4481, offset 0, flags [DF], proto TCP (6), length 40)
          16.1.7.240.5900 > 172.31.100.196.49331: Flags [.], cksum 0xfc71 (correct), seq 75, ack 21, win 16231, length 0

      Packet capture SITE 2 LINKSYS2 :
      18:48:53.131360 00:22:6b:3a:7e:a4 > 00:1b:21:9e:d7:a1, ethertype IPv4 (0x0800), length 64: (tos 0x0, ttl 126, id 148, offset 0, flags [DF], proto TCP (6), length 50)
          172.31.100.196.49331 > 16.1.7.240.5900: Flags [P.], cksum 0x2e05 (correct), seq 897777982:897777992, ack 3706019967, win 258, length 10
      18:48:54.129169 00:22:6b:3a:7e:a4 > 00:1b:21:9e:d7:a1, ethertype IPv4 (0x0800), length 64: (tos 0x0, ttl 126, id 149, offset 0, flags [DF], proto TCP (6), length 50)
          172.31.100.196.49331 > 16.1.7.240.5900: Flags [P.], cksum 0x2ddb (correct), seq 10:20, ack 33, win 258, length 10

      BUT !!!!! :

      PACKET CAPTURE WAN DEFAUT GATEWAY SITE1 :
      18:48:46.091734 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 86: (tos 0x0, ttl 122, id 4478, offset 0, flags [DF], proto TCP (6), length 72)
          16.1.7.240.5900 > 172.31.100.196.49331: Flags [P.], cksum 0xc85e (correct), seq 3706019731:3706019763, ack 897777932, win 16251, length 32
      18:48:46.263957 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 122, id 4479, offset 0, flags [DF], proto TCP (6), length 40)
          16.1.7.240.5900 > 172.31.100.196.49331: Flags [.], cksum 0xfc9c (correct), seq 32, ack 11, win 16241, length 0
      18:48:48.092669 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 97: (tos 0x0, ttl 122, id 4480, offset 0, flags [DF], proto TCP (6), length 83)
          16.1.7.240.5900 > 172.31.100.196.49331: Flags [P.], cksum 0x10d5 (correct), seq 32:75, ack 11, win 16241, length 43
      18:48:48.275501 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 122, id 4481, offset 0, flags [DF], proto TCP (6), length 40)
          16.1.7.240.5900 > 172.31.100.196.49331: Flags [.], cksum 0xfc71 (correct), seq 75, ack 21, win 16231, length 0

      PACKET CAPTURE WAN DEFAUT GATEWAY SITE2 :
      18:48:53.131360 00:22:6b:3a:7e:a4 > 00:1b:21:9e:d7:a1, ethertype IPv4 (0x0800), length 64: (tos 0x0, ttl 126, id 148, offset 0, flags [DF], proto TCP (6), length 50)
          172.31.100.196.49331 > 16.1.7.240.5900: Flags [P.], cksum 0x2e05 (correct), seq 897777982:897777992, ack 3706019967, win 258, length 10
      18:48:54.129169 00:22:6b:3a:7e:a4 > 00:1b:21:9e:d7:a1, ethertype IPv4 (0x0800), length 64: (tos 0x0, ttl 126, id 149, offset 0, flags [DF], proto TCP (6), length 50)
          172.31.100.196.49331 > 16.1.7.240.5900: Flags [P.], cksum 0x2ddb (correct), seq 10:20, ack 33, win 258, length 10

      The rules are the picture …

      pfsense1.jpg
      pfsense1.jpg_thumb

      1 Reply Last reply Reply Quote 0
      • P Offline
        phil.davis
        last edited by

        The source and destination look around the wrong way. In the capture at site 1 the packet has source 16.1.7.240.5900 so the rule on LAN at site 1 has to have source IP 16.1.7.240 port 5900, destination other LAN port any.

        As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
        If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

        1 Reply Last reply Reply Quote 0
        • Z Offline
          zoummuoz
          last edited by

          HOOOO !!! Sorry for the mistake …
          I inverted the site in my resume ...

          Please read :
          Packet capture SITE 2 LINKSYS2 :
          18:48:46.091734 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 86: (tos 0x0, ttl 122, id 4478, offset 0, flags [DF], proto TCP (6), length 72)
              16.1.7.240.5900 > 172.31.100.196.49331: Flags [P.], cksum 0xc85e (correct), seq 3706019731:3706019763, ack 897777932, win 16251, length 32
          18:48:46.263957 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 122, id 4479, offset 0, flags [DF], proto TCP (6), length 40)
              16.1.7.240.5900 > 172.31.100.196.49331: Flags [.], cksum 0xfc9c (correct), seq 32, ack 11, win 16241, length 0
          18:48:48.092669 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 97: (tos 0x0, ttl 122, id 4480, offset 0, flags [DF], proto TCP (6), length 83)
              16.1.7.240.5900 > 172.31.100.196.49331: Flags [P.], cksum 0x10d5 (correct), seq 32:75, ack 11, win 16241, length 43
          18:48:48.275501 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 122, id 4481, offset 0, flags [DF], proto TCP (6), length 40)
              16.1.7.240.5900 > 172.31.100.196.49331: Flags [.], cksum 0xfc71 (correct), seq 75, ack 21, win 16231, length 0

          Packet capture SITE 1 LINKSYS2 :
          18:48:53.131360 00:22:6b:3a:7e:a4 > 00:1b:21:9e:d7:a1, ethertype IPv4 (0x0800), length 64: (tos 0x0, ttl 126, id 148, offset 0, flags [DF], proto TCP (6), length 50)
              172.31.100.196.49331 > 16.1.7.240.5900: Flags [P.], cksum 0x2e05 (correct), seq 897777982:897777992, ack 3706019967, win 258, length 10
          18:48:54.129169 00:22:6b:3a:7e:a4 > 00:1b:21:9e:d7:a1, ethertype IPv4 (0x0800), length 64: (tos 0x0, ttl 126, id 149, offset 0, flags [DF], proto TCP (6), length 50)
              172.31.100.196.49331 > 16.1.7.240.5900: Flags [P.], cksum 0x2ddb (correct), seq 10:20, ack 33, win 258, length 10

          BUT !!!!! :

          PACKET CAPTURE WAN DEFAUT GATEWAY SITE2 :
          18:48:46.091734 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 86: (tos 0x0, ttl 122, id 4478, offset 0, flags [DF], proto TCP (6), length 72)
              16.1.7.240.5900 > 172.31.100.196.49331: Flags [P.], cksum 0xc85e (correct), seq 3706019731:3706019763, ack 897777932, win 16251, length 32
          18:48:46.263957 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 122, id 4479, offset 0, flags [DF], proto TCP (6), length 40)
              16.1.7.240.5900 > 172.31.100.196.49331: Flags [.], cksum 0xfc9c (correct), seq 32, ack 11, win 16241, length 0
          18:48:48.092669 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 97: (tos 0x0, ttl 122, id 4480, offset 0, flags [DF], proto TCP (6), length 83)
              16.1.7.240.5900 > 172.31.100.196.49331: Flags [P.], cksum 0x10d5 (correct), seq 32:75, ack 11, win 16241, length 43
          18:48:48.275501 1c:17:d3:72:df:53 > 00:16:e6:f6:5a:a2, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 122, id 4481, offset 0, flags [DF], proto TCP (6), length 40)
              16.1.7.240.5900 > 172.31.100.196.49331: Flags [.], cksum 0xfc71 (correct), seq 75, ack 21, win 16231, length 0

          PACKET CAPTURE WAN DEFAUT GATEWAY SITE1 :
          18:48:53.131360 00:22:6b:3a:7e:a4 > 00:1b:21:9e:d7:a1, ethertype IPv4 (0x0800), length 64: (tos 0x0, ttl 126, id 148, offset 0, flags [DF], proto TCP (6), length 50)
              172.31.100.196.49331 > 16.1.7.240.5900: Flags [P.], cksum 0x2e05 (correct), seq 897777982:897777992, ack 3706019967, win 258, length 10
          18:48:54.129169 00:22:6b:3a:7e:a4 > 00:1b:21:9e:d7:a1, ethertype IPv4 (0x0800), length 64: (tos 0x0, ttl 126, id 149, offset 0, flags [DF], proto TCP (6), length 50)
              172.31.100.196.49331 > 16.1.7.240.5900: Flags [P.], cksum 0x2ddb (correct), seq 10:20, ack 33, win 258, length 10

          1 Reply Last reply Reply Quote 0
          • P Offline
            phil.davis
            last edited by

            The rules should be on Site1 LAN and Site2 LAN - are they?
            Are there other rules above those?
            If so, what are they? They might be matching first.

            As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
            If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

            1 Reply Last reply Reply Quote 0
            • Z Offline
              zoummuoz
              last edited by

              Thank's for your question.

              You'll find attach all the rules on SITE1 and SITE2

              On SITE1 PFSENSE is in PRODUCTION. On SITE2 PFSENSE is in TEST mode …

              Many thanks for you help and interest.

              Regards.

              Pfsense_Site_1.jpg_thumb
              Pfsense_Site_1.jpg
              Pfsense_Site_2.jpg_thumb
              Pfsense_Site_2.jpg

              1 Reply Last reply Reply Quote 0
              • Z Offline
                zoummuoz
                last edited by

                Hi all,

                Alway the same pb.

                I made a change : no nat when packet use the ADSL VPN.

                The systeme worked during an half day. This moning after the reboot of the PFSENSE, with no change … it don't work ...

                I joint here a schem of the install ...

                vpns.png
                vpns.png_thumb

                1 Reply Last reply Reply Quote 0
                • H Offline
                  heper
                  last edited by

                  do you "own" the 16.1.7.0/24 subnet? (it's not in private address space)

                  CDIR
                  16.0.0.0/8
                  Country Code
                  US
                  Registry
                  arin
                  Registration Date
                  18/05/1989
                  ASN Name
                  HP-INTERNET-AS Hewlett-Packard Company

                  1 Reply Last reply Reply Quote 0
                  • Z Offline
                    zoummuoz
                    last edited by

                    Hi,

                    Thank's for your question.
                    No, it's only an OOOOLLLLDDDD (1985) lan. And there is many old services on it (as400 for ex).
                    I know i'm not in the rfc … but at this time it's not possible to change that.

                    Regards.

                    1 Reply Last reply Reply Quote 0
                    • Z Offline
                      zoummuoz
                      last edited by

                      Thank for interest,
                      No , not possible to change that … at this time.
                      The problem remains unsolved for me, and becomes critical (meaning death) : 3 months late ...
                      but I do not have much time right now (and it takes time to test all this ... ).
                      But i've read many and many docs about pfsense and route ... it appear that i'm not isolated. The defaut route for return packet seem to be discus in vpn and group gateway ...

                      Actualy, I'm looking around NAT (apparently, for the return packet, pfsense take the first NAT rule that match -> defaut interface ...).
                      Too : I'm looking around Floating Rules, with the Quick option ...
                      And too : grouping my ADSL gateways and mount a vpn on it between my to sites ...

                      If anyone has a similar architecture to that I want to put in place, thank you to tell me, it works with me ...

                      1 Reply Last reply Reply Quote 0
                      • Z Offline
                        zoummuoz
                        last edited by

                        Hello everyone ,

                        Yep , I am clinging to my topic !!!  :P

                        My latest tests and my conclusion :

                        I have been testing various protocol and it turns out they do not react the same way .

                        As I have said above , ICMP is left of those below :

                        With a ping, ICMP packets do well by the designated interface on the firewall 1 ICMP rule to the firewall 2 via the well VPN link and come on site 2, then arrive at the destination machine , leave and resume … well take the route specified in the rule of pfSense 2 Site 2 and finally arrive destination on the site 1 has taking good VPN link !

                        (remember, for all that is not ICMP, the return is ALWAYS ON DEFAULT GATEWAY on the distant PFSENSE !).
                        ICMP -> Layer 3 of the OSI model max -> no no FLAG TCP ACK !!!!

                        In fact : pfSense returns the default interface ALL PACKTES marked ACK ! ( when a packet traverses a rule it is tagged by pfSense ACK).

                        So new question for advanced users : how to solve my problem knowing that, without mounting a gas plant with floatings rules and proxy ... ???

                        Thank's for your help and interest.

                        Hope some people will be interest by the challenge !

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post
                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.