Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Multi WAN Strange Routing

    Scheduled Pinned Locked Moved
    Routing and Multi WAN
    4
    13
    6.3k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • W
      Wasca
      last edited by

      Hi Guys

      I have 2 WAN's, WAN is a ADSL2+ and WAN2 is a symetrical 10Mbit line. I have static IP's on both interfaces.

      I also have a /27 public subnet on my DMZ interface (routable via WAN2)

      When I do a traceroute to any of the /27 public IP's the trace result displays my static WAN IP as the hop just before it hits the public IP. See below

      Trace to 203.82.xxx.aaa

      1. customer-reverse-entry.64.xxx.xxx.xxx
      2. border-core2.sfo2.servepath.com
      3. te-3-4.car2.SanFrancisco1.Level3.net
      4. ae-4-4.ebr2.SanJose1.Level3.net
      5. ae-82-82.csw3.SanJose1.Level3.net
      6. ae-34-89.car4.SanJose1.Level3.net
      7. te-7-4-71.sntcca1wch010.wcg.Level3.net
      8. GigE-15-0.sntcca1wcx1.wcg.Level3.net
      9. pos-9-0-oc48.anhmca1wcx3.wcg.level3.net
      10. 64.200.249.178
      11. anhmca1wct1-powertel-atm.wcg.net
      12. ge-0-2-36.cust-gw03.powertel.net.au
      13. 203.185.xxx.ccc <====== This upstream router on the WAN2 link
      14. xxx-xxx-xxx-68.static.tpgi.com.au  <===== This is my ADSL2+ WAN IP
      15. 203.82.xxx.aaa <====== This is the public IP in the DMZ

      When I traceroute to the DMZ Interface on the PFSence Router (same /27 public subnet) I get this result

      Trace to 203.82.xxx.bbb

      1. customer-reverse-entry.64.151.96.33
      2. border-core2.sfo2.servepath.com
      3. te-3-4.car2.SanFrancisco1.Level3.net
      4. ae-4-4.ebr2.SanJose1.Level3.net
      5. ae-82-82.csw3.SanJose1.Level3.net
      6. ae-34-89.car4.SanJose1.Level3.net
      7. te-7-4-71.sntcca1wch010.wcg.Level3.net
      8. GigE-15-0.sntcca1wcx1.wcg.Level3.net
      9. pos-9-0-oc48.anhmca1wcx3.wcg.level3.net
      10. 64.200.249.178
      11. anhmca1wct1-powertel-atm.wcg.net
      12. ge-0-2-36.cust-gw03.powertel.net.au
      13. 203.185.xxx.ccc <====== This is the upstream router on the WAN2 link
      14. 203.82.xxx.bbb <====== This is the public IP on the DMZ interface on the PFSense router

      Why when i trace to the public DMZ IP range does the hop appear to hit my WAN IP interface?

      1 Reply Last reply Reply Quote 0
      • W
        Wasca
        last edited by

        This one's really got me stumped! Could it just be that the WAN IP is used to present to the world when tracing to the routable /27 Public network even thought your coming across the WAN2 interface to get to it?

        1 Reply Last reply Reply Quote 0
        • W
          Wasca
          last edited by

          Hey Guys

          This is causing me some serious issues.

          When the WAN (Bridged ADSL2+) is disconnected on my PFSense install It stops access from the outside world getting to my public IP range on my DMZ ( the /27 routed via WAN2)

          This does not make any sense.

          Could this be a firewall rule issue?

          1 Reply Last reply Reply Quote 0
          • dotdashD
            dotdash
            last edited by

            It would help if you described how the interfaces were configured- is DMZ bridged to WAN2? Also, knowing the firewall rules on the DMZ, as well as the gateway settings of the machine you are tracerouting would be nice.

            1 Reply Last reply Reply Quote 0
            • W
              Wasca
              last edited by

              OK here goes

              The DMZ interface is on OPT2 it's a /27  (203.xxx.xxx.160) public network. The DMZ Interface is static and is the first usable IP in the /27 network (203.xxx.xxx.161). My WAN2 ISP routes to the DMZ via my WAN2 interface IP.
              I'm using Advanced Outbound NAT for my LAN on both WAN1 and WAN2.

              WAN2  192.168.1.0/24  *  *  *  *  *  NO NAT to LAN Subnet
              WAN1  192.168.1.0/24  *  *  *  *  *  NO NAT to LAN Subnet

              Rules on the DMZ

              Lets just base this on Ping and Traces since that where the issue lies at the moment.

              ICMP  DMZ net  *  ! INTNETWORKS  *  203.xxx.xxx.153  <=== If Not LAN send out WAN2 Gateway
              ICMP  DMZ net  *            *                  *            *            <=== If any send out default routing table

              Rules on WAN2

              ICMP      *              *  DMZ net          *            *            <=== Allow ICMP to DMZ

              Interface settings of the server I'm trying to trace to:

              iface eth0 inet static
                      address 203.xxx.xxx.170
                      netmask 255.255.255.224
                      network 203.xxx.xxx.160
                      broadcast 203.xxx.xxx.191
                      gateway 203.xxx.xxx.161
                      # dns-* options are implemented by the resolvconf package, if installed
                      dns-nameservers 203.xxx.xxx.161 203.xxx.xxx.9 203.xxx.xxx.10

              I have no rules on the WAN inteface as I have no need for antything coming in on that interface (only used for LAN clients browsinig the web)

              Here are the static routes I have so as to make use of the DNS Servers on the WAN link by my LAN clients

              WAN  203.xxx.xxx.35/32  10.xxx.xxx.214          WAN DNS#1  <=== I realise the gateway is a private network this is what the ADSL2+ ISP hands out, not sure why
              WAN 203.xxx.xxx.36/32 10.xxx.xxx.214         WAN DNS#2 
              WAN2  203.xxx.xxx.10/32 203.xxx.xxx.153 WAN2 DNS#2 
              WAN2  203.xxx.xxx.9/32 203.xxx.xxx.153 WAN2 DNS#1
              WAN2  203.xxx.xxx.98/32        203.xxx.xxx.153 Gateway to IPSEC END POINT out WAN2

              I noticed these routes in my routing table

              default           10.xxx.xxx.214       UGS 0 1492837 1492   ng0
              10.xxx.xxx.214       123.xxx.xxx.68             UH     3     4041       1492 ng0
              123.xxx.xxx.68               lo0                 UHS 0   0           16384      lo0  <=== I have my suspicions of this entry (WAN IP)
              127.0.0.1             127.0.0.1                 UH 0   0              16384      lo0
              203.xxx.xxx.152/29        link#4               UC      0 0         1500     em3 <=== This is the WAN2 subnet
              203.xxx.xxx.158      00:xxx:23:xxx:ea:c1 UHLW 1 15         1500     lo0  <=== This is WAN2 IP. This looks sus also!! why the same interface (lo0) as WAN
              203.xxx.xxx.160/27   link#2             UC        0 0         1500     em1 <=== This is the DMZ Subnet
              203.xxx.xxx.161      00:04:xxx:a5:xxx:f3 UHLW 1   8           1500      lo0  <=== This is the DMZ Interface IP. Why is it lo0?

              Ok so now that you have all that, just recap, when I disconnect my WAN (PPPoE) at the interfaces page I can not trace to the DMZ or the WAN2 interface from externally.

              It appears to destroy the route to my WAN2 at this route
              203.xxx.xxx.158      00:xxx:23:xxx:ea:c1 UHLW 1 15         1500     lo0

              I suppose if you bring down this route maybe it destroys the WAN2 interface also at interface lo0?
              123.xxx.xxx.68               lo0                 UHS 0   0           16384      lo0

              I hope some one can help me, let me know if you need anything else

              1 Reply Last reply Reply Quote 0
              • dotdashD
                dotdash
                last edited by

                Ok, so I didn't read all of that very carefully, but it appears you have the DMZ setup in a rather unusual fashion.
                I suspect you problems are due to this. If the DMZ is coming in through WAN2, than it seems the DMZ should be bridged to the WAN…
                I would just put a private subnet on the DMZ, add the VIPs from the /27 to WAN2, and NAT the DMZ; but if you want the servers in the DMZ to have public IPs, the usual configuration is to bridge the interface.

                1 Reply Last reply Reply Quote 0
                • H
                  hoba
                  last edited by

                  You can't use public IPs in your dmz if you want to work with multiwan. You have to use NAT. The public IPs are known to the internet to be behind one of the wans only. It might work if you use nat on the interface that these public IPs are not behind only.

                  1 Reply Last reply Reply Quote 0
                  • W
                    Wasca
                    last edited by

                    Maybe I didn't explain my self properly.

                    My secondary ISP is on WAN2 (OPT2) It's a /29 subnet. My publicly routable network /27 (OPT3 / called DMZ) is routed to my WAN2 IP from my WAN2 ISP.

                    I'm only Natting the LAN.

                    Surely I can have a routable public network off my OPT3 interface? Why would I need to bridge my /29 WAN2 network with the /27 public routable DMZ network?

                    Ok so the question again is why is my WAN IP appearing in the trace route when I'm tracing to a public IP that is routed via my WAN2? Once that is answered, I may then be able to understand why when my WAN (ADSL2+) goes down (and the WAN interface looses it's IP) I can't trace or gain any access to the public IP's on my OPT3 interface from outside of my office

                    Trace to 203.82.xxx.aaa This is a public IP on my OPT3 network routed via my WAN2 interface IP

                    1. customer-reverse-entry.64.xxx.xxx.xxx
                    2. border-core2.sfo2.servepath.com
                    3. te-3-4.car2.SanFrancisco1.Level3.net
                    4. ae-4-4.ebr2.SanJose1.Level3.net
                    5. ae-82-82.csw3.SanJose1.Level3.net
                    6. ae-34-89.car4.SanJose1.Level3.net
                    7. te-7-4-71.sntcca1wch010.wcg.Level3.net
                    8. GigE-15-0.sntcca1wcx1.wcg.Level3.net
                    9. pos-9-0-oc48.anhmca1wcx3.wcg.level3.net
                    10. 64.200.249.178
                    11. anhmca1wct1-powertel-atm.wcg.net
                    12. ge-0-2-36.cust-gw03.powertel.net.au
                    13. 203.185.xxx.ccc <====== This is the upstream (next hop/gateway) router on the WAN2 link
                    14. xxx-xxx-xxx-68.static.tpgi.com.au  <===== This is my ADSL2+ WAN IP (WHY IS THIS NOT MY WAN2 IP??)
                    15. 203.82.xxx.aaa <====== This is the public IP on the OPT3 network

                    I pose this question, Could it be that I need to add a Virtual IP like this below on my WAN2?

                    Type: Proxy Arp
                    Interface: WAN2
                    IPAddress(s): Network
                                      203.82.xxx.xxx/27
                    Description: Proxy Arp for /27 Network on OPT3

                    or do I use CARP?

                    Type: Carp
                    Interface: WAN2
                    IPAddress(s): Network
                                      203.82.xxx.xxx/27
                    VHID Group: 1
                    Advertising Frequency: 0
                    Description: CARP for /27 Network on OPT3

                    What do you think?

                    1 Reply Last reply Reply Quote 0
                    • dotdashD
                      dotdash
                      last edited by

                      @Wasca:

                      Maybe I didn't explain my self properly.

                      My secondary ISP is on WAN2 (OPT2) It's a /29 subnet. My publicly routable network /27 (OPT3 / called DMZ) is routed to my WAN2 IP from my WAN2 ISP.

                      I got that part of it. Maybe I did not explain myself properly. Let me put this way- you're doing it wrong. You are slapping an additional subnet coming in via WAN2 on a separate interface, not bridged to the interface the block is routed from, and expecting everything to work smoothly in mult-wan.
                      Perhaps this sounds short, but people are trying to give you some helpful advice, and it seems like you don't want to hear it. What you are trying may be possible, but it is not a normal or supported configuration. We were trying to point you toward known working methods- if you want to go your own way, it will limit the amount of help others are able to provide.

                      1 Reply Last reply Reply Quote 0
                      • W
                        Wasca
                        last edited by

                        You are slapping an additional subnet coming in via WAN2 on a separate interface, not bridged to the interface the block is routed from, and expecting everything to work smoothly in mult-wan.

                        So are you saying you can't route public IP networks to other interfaces in a multi-wan setup, only if you have a single WAN? that sounds very strange.

                        Ok, So I did a little more searching and found this reply below from this thread http://forum.pfsense.org/index.php/topic,8990.0.html

                        Maybe you need to clarify first:
                        Does this /29 subnet get routed by your ISP to the public IP you have on your WAN?

                        Since you have a public /29 subnet there are multiple approaches:

                        1: You bridge the OPT1, on which your public IP's are used, to it's WAN.

                        2: You create virtual IP's on your WAN, use private addresses in your OPT1 and just forward the traffic you need from the VIP's to your private IP's.

                        3: You route your public IP's. though with only a /29 you would waste one of your 6 IP's and you'd be left with only 5.
                        (This only works of you have another public IP on WAN that's not within this /29

                        The first approach is maybe the best if you want the public IP's directly on you servers, and your subnet does NOT get routed to a public address on your WAN. Downside is you use one of your public IP's up for the WAN.
                        (You still can create NAT-forwardings from the WAN-address to computers in your personal LAN but that's probably not what you want.)

                        The second approach lets you use all the IP's out of your /29 subnet –> you can forward ports from all 6 IP's.
                        The downside is that your servers use private IP's which migh crete problems for certain setups.

                        The third approach would be imo the most clean approach. But it only works if your ISP routes your /29 to another public IP you have on the WAN. You disable NAT for your server-subnet and just route the IP's Smiley

                        If I was being asked this question, I would answer….

                        Yes my /27 is routed by my WAN2 ISP. My WAN2 interface IP is 203.82.xxx.aaa and on the WAN2 ISP's router they have a route to my /27 subnet via 203.82.xxx.aaa. The WAN2 interface IP of 203.82.xxx.aaa exists on it's own /29 subnet (The network between my PFSense box and the WAN2 ISP)

                        I want to do option 3 and this is currently what is setup and working. The problem occurs when my WAN (ADSL2+) connections drops offline I can't access the /27 subnet from externally, but servers on the OPT3 network can still ping out via WAN2 just fine. This can be explained by the first traceroute at the top of this thread (see the trace getting a response from the WAN IP not the WAN2 IP)

                        I apologise if I'm sounding stubborn but I can't see how Bridging my OPT3 with WAN2 (or the other way round, what ever it is) is the correct method when trying to route a public network, hence why option 3 above is what I currently use and is working, just not when my WAN goes down.

                        Tell me, does option 3 simply not work when you have more than 1 WAN? is this a limitation of PFSense, I would be highly surprised if it was.

                        1 Reply Last reply Reply Quote 0
                        • GruensFroeschliG
                          GruensFroeschli
                          last edited by

                          I'm not sure if i understood your setup correctly.
                          (Screenshots and not text of the rules would help)

                          Since you want to route your DMZ through WAN2.
                          Did you create a rule on the DMZ tab that has as gateway "WAN2" and not *
                          Did you remove all AoN rules related to your DMZ (you dont want to NAT it)?

                          I would try to get this working step by step.

                          1: Have both WAN's up. i assume this is working

                          2: Get LAN working. what do you want here? loadbalancing the LAN subnet to WAN1 and WAN2?
                          I see that you already created 2 AoN rules that NAT the LAN subnet to WAN1 and WAN2.

                          3: Set up the Addressing in the DMZ. Make sure that the client in the DMZ use the pfSense DMZ-interface IP as gateway.

                          4: assure that there are NO AoN rules for the DMZ.

                          5: create firewall-rules on DMZ: Assure that the rule allowing traffic on the DMZ uses as gateway the WAN2 (the reply should go back to where the request came from).

                          6: create firewall-rules on WAN2: allow as "destination your DMZ" access depending on your needs.

                          Like this should now:

                          • Traffic from the LAN will be NATed to the WAN or WAN2 ip (depends on what you set up)
                          • Traffic from the DMZ will not be NATed, but routed through the WAN2.
                          • Traffic comming in on the WAN2 destined to the DMZ will reach the DMZ.

                          Maybe you could show screenshots of your rules after the changes you did?
                          (text is kind of unhandy).

                          We do what we must, because we can.

                          Asking questions the smart way: http://www.catb.org/esr/faqs/smart-questions.html

                          1 Reply Last reply Reply Quote 0
                          • W
                            Wasca
                            last edited by

                            Since you want to route your DMZ through WAN2.
                            Did you create a rule on the DMZ tab that has as gateway "WAN" and not *
                            Did you remove all AoN rules related to your DMZ (you dont want to NAT it)?

                            All rules point to the WAN2 as the gateway.
                            I only have AoN rules for my LAN (and other natted private subnets) see attached image

                            1: Have both WAN's up. i assume this is working

                            Yes both WAN's are up and working with traffic going across them, interface settings attached

                            2: Get LAN working. what do you want here? loadbalancing the LAN subnet to WAN1 and WAN2?
                            I see that you already created 2 AoN rules that NAT the LAN subnet to WAN1 and WAN2.

                            LAN is working, No Load balancing or failover needed (trying to keep it simple to begin with).
                            I do have some port forward rules on the WAN2 and WAN pointing to PC's on the LAN (used for VNC Listen sessions)

                            3: Set up the Addressing in the DMZ. Make sure that the client in the DMZ use the pfSense DMZ-interface IP as gateway.

                            DMZ interface attached.

                            This is done on all servers, see below

                            iface eth0 inet static
                                    address 203.xxx.xxx.170
                                    netmask 255.255.255.224
                                    network 203.xxx.xxx.160
                                    broadcast 203.xxx.xxx.191
                                    gateway 203.xxx.xxx.161  <=====THIS IS THE OPT3 (DMZ) interface IP
                                    # dns-* options are implemented by the resolvconf package, if installed
                                    dns-nameservers 203.xxx.xxx.161 203.xxx.xxx.9 203.xxx.xxx.10 <==== IM USING THE OPT3 INTERFACE AS MY FIRST DNS SERVER

                            4: assure that there are NO AoN rules for the DMZ.

                            Nope, none there, I only apply these rules to networks I want natted like the LAN. see attached image

                            5: create rules on DMZ: Assure that the rule allowing traffic on the DMZ uses as gateway the WAN2 (the reply should go back to where the request came from).

                            I have all the rules point to the gateway on WAN2, except where the traffic from the DMZ needs to go the LAN or other private networks on the PFSense Router then I use * as the gateway. See attached image.

                            6: create rules on WAN2: allow as "destination your DMZ" access depending on your needs.

                            This has been done see attached picture.

                            Like this should now:

                            • Traffic from the LAN will be NATed to the WAN or WAN2 ip (depends on what you set up)
                            • Traffic from the DMZ will not be NATed, but routed through the WAN2.
                            • Traffic comming in on the WAN2 destined to the DMZ will reach the DMZ.

                            The setup is like this and has been from the begining.

                            As I have said in my previous posts, tracing out looks correct and works

                            This is a trace from one of my DMZ servers 203.xxx.xxx.171

                            Host                                                                  Loss%  Snt  Last  Avg  Best  Wrst StDev
                            1. 203.xxx.xxx.153                                                      0.0%    17    2.6  2.5  2.3  3.0  0.2  <==== This is my WAN2 Gateway IP
                            2. 203.xxx.xxx.185                                                    0.0%    17    2.5  2.8  2.4  4.8  0.7
                            3. qbrisbrdr01-ge01.powertel.net.au                                    0.0%    17    3.4  3.5  3.1  5.9  0.7
                            4. Ve1.rq-127creek-core-02.pipenetworks.com                          11.8%    17    3.8 844.4  3.5 5772. 1846.
                            5. Fast0-0-1203.rn-400harris-core-01.pipenetworks.com                  0.0%    17  16.0  16.4  15.3  18.7  0.8
                            6. pyro-xxx.xxxx.com                                        0.0%    16  16.1  16.7  15.7  21.6  1.4
                            7. cms1-xxx.xxxx.com                                        0.0%    16  16.4  16.6  15.7  17.6  0.5

                            And this is the trace back from the other end

                            Trace to 203.xxx.xxx.171
                                                                                              Packets              Pings
                            Host                                                            Loss%  Snt  Last  Avg  Best  Wrst StDev
                            1. 203.161.136.81                                                0.0%    7    0.2  0.2  0.1  0.2  0.0     
                            2. RN-400HARRIS-CORE-01.pipenetworks.com                        0.0%    7    1.0  49.0  0.8 171.1  81.9
                            3. Ve4.rq-148brunswick-core-01.pipenetworks.com                16.7%    6  13.3  19.8  13.1  45.6  14.4
                            4. AS9837.brisbane.pipenetworks.com                              0.0%    6  13.5  65.8  13.2 173.0  81.0
                            5. qbriscust01-ge02.powertel.net.au                              0.0%    6  14.1  26.0  14.1  52.2  18.2
                            6. 203.xxx.xxx.186                                              0.0%    6  169.3  83.2  14.6 180.2  79.2
                            7. 123-xxx-xxx-68.static.tpgi.com.au                            0.0%    6  50.3  69.3  16.1 257.2  94.0  <==== THIS IS THE WAN IP OF MY PFSense (IT SHOULD BE THE WAN2 IP)
                            8. 203.xxx.xxx.171                                                0.0%    6  16.1  16.2  15.8  17.3  0.5

                            Now the dilemma occurs when the WAN goes down on my PFSense. Imangine taking out hop #7 from the trace above, I now no longer can get access to the DMZ Servers.

                            I hope this clears up what I have set-up and what problems I'm having.

                            Does anyone else out there have 2 WANs and a routable network on an optional interface that can test this for me?

                            What I need tested is tracing in and out of a server on a public routable network hanging off an Optional Interface, make sure the trace goes in and out the same WAN (preferable your secondary WAN)

                            Thanks

                            Let me know if you need any more info, I'm very interested to find an answer to this.

                            AoN.gif
                            AoN.gif_thumb
                            ![DMZ Rules.gif](/public/imported_attachments/1/DMZ Rules.gif)
                            ![DMZ Rules.gif_thumb](/public/imported_attachments/1/DMZ Rules.gif_thumb)
                            WAN2-Rule.gif
                            WAN2-Rule.gif_thumb
                            ![WAN Interface.gif](/public/imported_attachments/1/WAN Interface.gif)
                            ![WAN Interface.gif_thumb](/public/imported_attachments/1/WAN Interface.gif_thumb)
                            ![WAN2 Interface.gif](/public/imported_attachments/1/WAN2 Interface.gif)
                            ![WAN2 Interface.gif_thumb](/public/imported_attachments/1/WAN2 Interface.gif_thumb)
                            ![DMZ Interface.gif](/public/imported_attachments/1/DMZ Interface.gif)
                            ![DMZ Interface.gif_thumb](/public/imported_attachments/1/DMZ Interface.gif_thumb)

                            1 Reply Last reply Reply Quote 0
                            • W
                              Wasca
                              last edited by

                              I was wondering if any of the devs could take another look at this, as I believe there could be a bug causing this issue.

                              I have complied with the appropriate setup for this kind of network and this is still causing me issues.

                              If this can't be resolved over the forum, I'm planning on swapping the WAN's so that the backup ADSL2+ connection becomes WAN2 and I use a half bridged setup.

                              Again, thanks for all your help so far, I really hope we can work this one out, as I don't believe I'm doing anything out of the ordinary.

                              Thanks

                              1 Reply Last reply Reply Quote 0
                              • First post
                                Last post