• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Multi WAN Strange Routing

Scheduled Pinned Locked Moved Routing and Multi WAN
13 Posts 4 Posters 6.5k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • D
    dotdash
    last edited by Apr 29, 2008, 5:46 PM

    It would help if you described how the interfaces were configured- is DMZ bridged to WAN2? Also, knowing the firewall rules on the DMZ, as well as the gateway settings of the machine you are tracerouting would be nice.

    1 Reply Last reply Reply Quote 0
    • W
      Wasca
      last edited by Apr 30, 2008, 4:12 AM Apr 30, 2008, 3:45 AM

      OK here goes

      The DMZ interface is on OPT2 it's a /27  (203.xxx.xxx.160) public network. The DMZ Interface is static and is the first usable IP in the /27 network (203.xxx.xxx.161). My WAN2 ISP routes to the DMZ via my WAN2 interface IP.
      I'm using Advanced Outbound NAT for my LAN on both WAN1 and WAN2.

      WAN2  192.168.1.0/24  *  *  *  *  *  NO NAT to LAN Subnet
      WAN1  192.168.1.0/24  *  *  *  *  *  NO NAT to LAN Subnet

      Rules on the DMZ

      Lets just base this on Ping and Traces since that where the issue lies at the moment.

      ICMP  DMZ net  *  ! INTNETWORKS  *  203.xxx.xxx.153  <=== If Not LAN send out WAN2 Gateway
      ICMP  DMZ net  *            *                  *            *            <=== If any send out default routing table

      Rules on WAN2

      ICMP      *              *  DMZ net          *            *            <=== Allow ICMP to DMZ

      Interface settings of the server I'm trying to trace to:

      iface eth0 inet static
              address 203.xxx.xxx.170
              netmask 255.255.255.224
              network 203.xxx.xxx.160
              broadcast 203.xxx.xxx.191
              gateway 203.xxx.xxx.161
              # dns-* options are implemented by the resolvconf package, if installed
              dns-nameservers 203.xxx.xxx.161 203.xxx.xxx.9 203.xxx.xxx.10

      I have no rules on the WAN inteface as I have no need for antything coming in on that interface (only used for LAN clients browsinig the web)

      Here are the static routes I have so as to make use of the DNS Servers on the WAN link by my LAN clients

      WAN  203.xxx.xxx.35/32  10.xxx.xxx.214          WAN DNS#1  <=== I realise the gateway is a private network this is what the ADSL2+ ISP hands out, not sure why
      WAN 203.xxx.xxx.36/32 10.xxx.xxx.214         WAN DNS#2 
      WAN2  203.xxx.xxx.10/32 203.xxx.xxx.153 WAN2 DNS#2 
      WAN2  203.xxx.xxx.9/32 203.xxx.xxx.153 WAN2 DNS#1
      WAN2  203.xxx.xxx.98/32        203.xxx.xxx.153 Gateway to IPSEC END POINT out WAN2

      I noticed these routes in my routing table

      default           10.xxx.xxx.214       UGS 0 1492837 1492   ng0
      10.xxx.xxx.214       123.xxx.xxx.68             UH     3     4041       1492 ng0
      123.xxx.xxx.68               lo0                 UHS 0   0           16384      lo0  <=== I have my suspicions of this entry (WAN IP)
      127.0.0.1             127.0.0.1                 UH 0   0              16384      lo0
      203.xxx.xxx.152/29        link#4               UC      0 0         1500     em3 <=== This is the WAN2 subnet
      203.xxx.xxx.158      00:xxx:23:xxx:ea:c1 UHLW 1 15         1500     lo0  <=== This is WAN2 IP. This looks sus also!! why the same interface (lo0) as WAN
      203.xxx.xxx.160/27   link#2             UC        0 0         1500     em1 <=== This is the DMZ Subnet
      203.xxx.xxx.161      00:04:xxx:a5:xxx:f3 UHLW 1   8           1500      lo0  <=== This is the DMZ Interface IP. Why is it lo0?

      Ok so now that you have all that, just recap, when I disconnect my WAN (PPPoE) at the interfaces page I can not trace to the DMZ or the WAN2 interface from externally.

      It appears to destroy the route to my WAN2 at this route
      203.xxx.xxx.158      00:xxx:23:xxx:ea:c1 UHLW 1 15         1500     lo0

      I suppose if you bring down this route maybe it destroys the WAN2 interface also at interface lo0?
      123.xxx.xxx.68               lo0                 UHS 0   0           16384      lo0

      I hope some one can help me, let me know if you need anything else

      1 Reply Last reply Reply Quote 0
      • D
        dotdash
        last edited by Apr 30, 2008, 3:02 PM

        Ok, so I didn't read all of that very carefully, but it appears you have the DMZ setup in a rather unusual fashion.
        I suspect you problems are due to this. If the DMZ is coming in through WAN2, than it seems the DMZ should be bridged to the WAN…
        I would just put a private subnet on the DMZ, add the VIPs from the /27 to WAN2, and NAT the DMZ; but if you want the servers in the DMZ to have public IPs, the usual configuration is to bridge the interface.

        1 Reply Last reply Reply Quote 0
        • H
          hoba
          last edited by Apr 30, 2008, 4:15 PM

          You can't use public IPs in your dmz if you want to work with multiwan. You have to use NAT. The public IPs are known to the internet to be behind one of the wans only. It might work if you use nat on the interface that these public IPs are not behind only.

          1 Reply Last reply Reply Quote 0
          • W
            Wasca
            last edited by May 1, 2008, 2:02 PM Apr 30, 2008, 11:15 PM

            Maybe I didn't explain my self properly.

            My secondary ISP is on WAN2 (OPT2) It's a /29 subnet. My publicly routable network /27 (OPT3 / called DMZ) is routed to my WAN2 IP from my WAN2 ISP.

            I'm only Natting the LAN.

            Surely I can have a routable public network off my OPT3 interface? Why would I need to bridge my /29 WAN2 network with the /27 public routable DMZ network?

            Ok so the question again is why is my WAN IP appearing in the trace route when I'm tracing to a public IP that is routed via my WAN2? Once that is answered, I may then be able to understand why when my WAN (ADSL2+) goes down (and the WAN interface looses it's IP) I can't trace or gain any access to the public IP's on my OPT3 interface from outside of my office

            Trace to 203.82.xxx.aaa This is a public IP on my OPT3 network routed via my WAN2 interface IP

            1. customer-reverse-entry.64.xxx.xxx.xxx
            2. border-core2.sfo2.servepath.com
            3. te-3-4.car2.SanFrancisco1.Level3.net
            4. ae-4-4.ebr2.SanJose1.Level3.net
            5. ae-82-82.csw3.SanJose1.Level3.net
            6. ae-34-89.car4.SanJose1.Level3.net
            7. te-7-4-71.sntcca1wch010.wcg.Level3.net
            8. GigE-15-0.sntcca1wcx1.wcg.Level3.net
            9. pos-9-0-oc48.anhmca1wcx3.wcg.level3.net
            10. 64.200.249.178
            11. anhmca1wct1-powertel-atm.wcg.net
            12. ge-0-2-36.cust-gw03.powertel.net.au
            13. 203.185.xxx.ccc <====== This is the upstream (next hop/gateway) router on the WAN2 link
            14. xxx-xxx-xxx-68.static.tpgi.com.au  <===== This is my ADSL2+ WAN IP (WHY IS THIS NOT MY WAN2 IP??)
            15. 203.82.xxx.aaa <====== This is the public IP on the OPT3 network

            I pose this question, Could it be that I need to add a Virtual IP like this below on my WAN2?

            Type: Proxy Arp
            Interface: WAN2
            IPAddress(s): Network
                              203.82.xxx.xxx/27
            Description: Proxy Arp for /27 Network on OPT3

            or do I use CARP?

            Type: Carp
            Interface: WAN2
            IPAddress(s): Network
                              203.82.xxx.xxx/27
            VHID Group: 1
            Advertising Frequency: 0
            Description: CARP for /27 Network on OPT3

            What do you think?

            1 Reply Last reply Reply Quote 0
            • D
              dotdash
              last edited by May 1, 2008, 2:57 PM

              @Wasca:

              Maybe I didn't explain my self properly.

              My secondary ISP is on WAN2 (OPT2) It's a /29 subnet. My publicly routable network /27 (OPT3 / called DMZ) is routed to my WAN2 IP from my WAN2 ISP.

              I got that part of it. Maybe I did not explain myself properly. Let me put this way- you're doing it wrong. You are slapping an additional subnet coming in via WAN2 on a separate interface, not bridged to the interface the block is routed from, and expecting everything to work smoothly in mult-wan.
              Perhaps this sounds short, but people are trying to give you some helpful advice, and it seems like you don't want to hear it. What you are trying may be possible, but it is not a normal or supported configuration. We were trying to point you toward known working methods- if you want to go your own way, it will limit the amount of help others are able to provide.

              1 Reply Last reply Reply Quote 0
              • W
                Wasca
                last edited by May 2, 2008, 12:52 AM May 2, 2008, 12:13 AM

                You are slapping an additional subnet coming in via WAN2 on a separate interface, not bridged to the interface the block is routed from, and expecting everything to work smoothly in mult-wan.

                So are you saying you can't route public IP networks to other interfaces in a multi-wan setup, only if you have a single WAN? that sounds very strange.

                Ok, So I did a little more searching and found this reply below from this thread http://forum.pfsense.org/index.php/topic,8990.0.html

                Maybe you need to clarify first:
                Does this /29 subnet get routed by your ISP to the public IP you have on your WAN?

                Since you have a public /29 subnet there are multiple approaches:

                1: You bridge the OPT1, on which your public IP's are used, to it's WAN.

                2: You create virtual IP's on your WAN, use private addresses in your OPT1 and just forward the traffic you need from the VIP's to your private IP's.

                3: You route your public IP's. though with only a /29 you would waste one of your 6 IP's and you'd be left with only 5.
                (This only works of you have another public IP on WAN that's not within this /29

                The first approach is maybe the best if you want the public IP's directly on you servers, and your subnet does NOT get routed to a public address on your WAN. Downside is you use one of your public IP's up for the WAN.
                (You still can create NAT-forwardings from the WAN-address to computers in your personal LAN but that's probably not what you want.)

                The second approach lets you use all the IP's out of your /29 subnet –> you can forward ports from all 6 IP's.
                The downside is that your servers use private IP's which migh crete problems for certain setups.

                The third approach would be imo the most clean approach. But it only works if your ISP routes your /29 to another public IP you have on the WAN. You disable NAT for your server-subnet and just route the IP's Smiley

                If I was being asked this question, I would answer….

                Yes my /27 is routed by my WAN2 ISP. My WAN2 interface IP is 203.82.xxx.aaa and on the WAN2 ISP's router they have a route to my /27 subnet via 203.82.xxx.aaa. The WAN2 interface IP of 203.82.xxx.aaa exists on it's own /29 subnet (The network between my PFSense box and the WAN2 ISP)

                I want to do option 3 and this is currently what is setup and working. The problem occurs when my WAN (ADSL2+) connections drops offline I can't access the /27 subnet from externally, but servers on the OPT3 network can still ping out via WAN2 just fine. This can be explained by the first traceroute at the top of this thread (see the trace getting a response from the WAN IP not the WAN2 IP)

                I apologise if I'm sounding stubborn but I can't see how Bridging my OPT3 with WAN2 (or the other way round, what ever it is) is the correct method when trying to route a public network, hence why option 3 above is what I currently use and is working, just not when my WAN goes down.

                Tell me, does option 3 simply not work when you have more than 1 WAN? is this a limitation of PFSense, I would be highly surprised if it was.

                1 Reply Last reply Reply Quote 0
                • G
                  GruensFroeschli
                  last edited by May 2, 2008, 2:27 PM May 2, 2008, 11:14 AM

                  I'm not sure if i understood your setup correctly.
                  (Screenshots and not text of the rules would help)

                  Since you want to route your DMZ through WAN2.
                  Did you create a rule on the DMZ tab that has as gateway "WAN2" and not *
                  Did you remove all AoN rules related to your DMZ (you dont want to NAT it)?

                  I would try to get this working step by step.

                  1: Have both WAN's up. i assume this is working

                  2: Get LAN working. what do you want here? loadbalancing the LAN subnet to WAN1 and WAN2?
                  I see that you already created 2 AoN rules that NAT the LAN subnet to WAN1 and WAN2.

                  3: Set up the Addressing in the DMZ. Make sure that the client in the DMZ use the pfSense DMZ-interface IP as gateway.

                  4: assure that there are NO AoN rules for the DMZ.

                  5: create firewall-rules on DMZ: Assure that the rule allowing traffic on the DMZ uses as gateway the WAN2 (the reply should go back to where the request came from).

                  6: create firewall-rules on WAN2: allow as "destination your DMZ" access depending on your needs.

                  Like this should now:

                  • Traffic from the LAN will be NATed to the WAN or WAN2 ip (depends on what you set up)
                  • Traffic from the DMZ will not be NATed, but routed through the WAN2.
                  • Traffic comming in on the WAN2 destined to the DMZ will reach the DMZ.

                  Maybe you could show screenshots of your rules after the changes you did?
                  (text is kind of unhandy).

                  We do what we must, because we can.

                  Asking questions the smart way: http://www.catb.org/esr/faqs/smart-questions.html

                  1 Reply Last reply Reply Quote 0
                  • W
                    Wasca
                    last edited by May 2, 2008, 3:20 PM May 2, 2008, 3:08 PM

                    Since you want to route your DMZ through WAN2.
                    Did you create a rule on the DMZ tab that has as gateway "WAN" and not *
                    Did you remove all AoN rules related to your DMZ (you dont want to NAT it)?

                    All rules point to the WAN2 as the gateway.
                    I only have AoN rules for my LAN (and other natted private subnets) see attached image

                    1: Have both WAN's up. i assume this is working

                    Yes both WAN's are up and working with traffic going across them, interface settings attached

                    2: Get LAN working. what do you want here? loadbalancing the LAN subnet to WAN1 and WAN2?
                    I see that you already created 2 AoN rules that NAT the LAN subnet to WAN1 and WAN2.

                    LAN is working, No Load balancing or failover needed (trying to keep it simple to begin with).
                    I do have some port forward rules on the WAN2 and WAN pointing to PC's on the LAN (used for VNC Listen sessions)

                    3: Set up the Addressing in the DMZ. Make sure that the client in the DMZ use the pfSense DMZ-interface IP as gateway.

                    DMZ interface attached.

                    This is done on all servers, see below

                    iface eth0 inet static
                            address 203.xxx.xxx.170
                            netmask 255.255.255.224
                            network 203.xxx.xxx.160
                            broadcast 203.xxx.xxx.191
                            gateway 203.xxx.xxx.161  <=====THIS IS THE OPT3 (DMZ) interface IP
                            # dns-* options are implemented by the resolvconf package, if installed
                            dns-nameservers 203.xxx.xxx.161 203.xxx.xxx.9 203.xxx.xxx.10 <==== IM USING THE OPT3 INTERFACE AS MY FIRST DNS SERVER

                    4: assure that there are NO AoN rules for the DMZ.

                    Nope, none there, I only apply these rules to networks I want natted like the LAN. see attached image

                    5: create rules on DMZ: Assure that the rule allowing traffic on the DMZ uses as gateway the WAN2 (the reply should go back to where the request came from).

                    I have all the rules point to the gateway on WAN2, except where the traffic from the DMZ needs to go the LAN or other private networks on the PFSense Router then I use * as the gateway. See attached image.

                    6: create rules on WAN2: allow as "destination your DMZ" access depending on your needs.

                    This has been done see attached picture.

                    Like this should now:

                    • Traffic from the LAN will be NATed to the WAN or WAN2 ip (depends on what you set up)
                    • Traffic from the DMZ will not be NATed, but routed through the WAN2.
                    • Traffic comming in on the WAN2 destined to the DMZ will reach the DMZ.

                    The setup is like this and has been from the begining.

                    As I have said in my previous posts, tracing out looks correct and works

                    This is a trace from one of my DMZ servers 203.xxx.xxx.171

                    Host                                                                  Loss%  Snt  Last  Avg  Best  Wrst StDev
                    1. 203.xxx.xxx.153                                                      0.0%    17    2.6  2.5  2.3  3.0  0.2  <==== This is my WAN2 Gateway IP
                    2. 203.xxx.xxx.185                                                    0.0%    17    2.5  2.8  2.4  4.8  0.7
                    3. qbrisbrdr01-ge01.powertel.net.au                                    0.0%    17    3.4  3.5  3.1  5.9  0.7
                    4. Ve1.rq-127creek-core-02.pipenetworks.com                          11.8%    17    3.8 844.4  3.5 5772. 1846.
                    5. Fast0-0-1203.rn-400harris-core-01.pipenetworks.com                  0.0%    17  16.0  16.4  15.3  18.7  0.8
                    6. pyro-xxx.xxxx.com                                        0.0%    16  16.1  16.7  15.7  21.6  1.4
                    7. cms1-xxx.xxxx.com                                        0.0%    16  16.4  16.6  15.7  17.6  0.5

                    And this is the trace back from the other end

                    Trace to 203.xxx.xxx.171
                                                                                      Packets              Pings
                    Host                                                            Loss%  Snt  Last  Avg  Best  Wrst StDev
                    1. 203.161.136.81                                                0.0%    7    0.2  0.2  0.1  0.2  0.0     
                    2. RN-400HARRIS-CORE-01.pipenetworks.com                        0.0%    7    1.0  49.0  0.8 171.1  81.9
                    3. Ve4.rq-148brunswick-core-01.pipenetworks.com                16.7%    6  13.3  19.8  13.1  45.6  14.4
                    4. AS9837.brisbane.pipenetworks.com                              0.0%    6  13.5  65.8  13.2 173.0  81.0
                    5. qbriscust01-ge02.powertel.net.au                              0.0%    6  14.1  26.0  14.1  52.2  18.2
                    6. 203.xxx.xxx.186                                              0.0%    6  169.3  83.2  14.6 180.2  79.2
                    7. 123-xxx-xxx-68.static.tpgi.com.au                            0.0%    6  50.3  69.3  16.1 257.2  94.0  <==== THIS IS THE WAN IP OF MY PFSense (IT SHOULD BE THE WAN2 IP)
                    8. 203.xxx.xxx.171                                                0.0%    6  16.1  16.2  15.8  17.3  0.5

                    Now the dilemma occurs when the WAN goes down on my PFSense. Imangine taking out hop #7 from the trace above, I now no longer can get access to the DMZ Servers.

                    I hope this clears up what I have set-up and what problems I'm having.

                    Does anyone else out there have 2 WANs and a routable network on an optional interface that can test this for me?

                    What I need tested is tracing in and out of a server on a public routable network hanging off an Optional Interface, make sure the trace goes in and out the same WAN (preferable your secondary WAN)

                    Thanks

                    Let me know if you need any more info, I'm very interested to find an answer to this.

                    AoN.gif
                    AoN.gif_thumb
                    ![DMZ Rules.gif](/public/imported_attachments/1/DMZ Rules.gif)
                    ![DMZ Rules.gif_thumb](/public/imported_attachments/1/DMZ Rules.gif_thumb)
                    WAN2-Rule.gif
                    WAN2-Rule.gif_thumb
                    ![WAN Interface.gif](/public/imported_attachments/1/WAN Interface.gif)
                    ![WAN Interface.gif_thumb](/public/imported_attachments/1/WAN Interface.gif_thumb)
                    ![WAN2 Interface.gif](/public/imported_attachments/1/WAN2 Interface.gif)
                    ![WAN2 Interface.gif_thumb](/public/imported_attachments/1/WAN2 Interface.gif_thumb)
                    ![DMZ Interface.gif](/public/imported_attachments/1/DMZ Interface.gif)
                    ![DMZ Interface.gif_thumb](/public/imported_attachments/1/DMZ Interface.gif_thumb)

                    1 Reply Last reply Reply Quote 0
                    • W
                      Wasca
                      last edited by May 4, 2008, 11:57 PM

                      I was wondering if any of the devs could take another look at this, as I believe there could be a bug causing this issue.

                      I have complied with the appropriate setup for this kind of network and this is still causing me issues.

                      If this can't be resolved over the forum, I'm planning on swapping the WAN's so that the backup ADSL2+ connection becomes WAN2 and I use a half bridged setup.

                      Again, thanks for all your help so far, I really hope we can work this one out, as I don't believe I'm doing anything out of the ordinary.

                      Thanks

                      1 Reply Last reply Reply Quote 0
                      13 out of 13
                      • First post
                        13/13
                        Last post
                      Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                        This community forum collects and processes your personal information.
                        consent.not_received