Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Multi-WAN Setup with 4G CradlePoint Not Working

    Scheduled Pinned Locked Moved Routing and Multi WAN
    19 Posts 4 Posters 5.2k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • B
      bhsense
      last edited by

      I think it's not just a DNS issue, since we can't ping 8.8.8.8 from the computers behind the firewall, but we'll try also with 208.67.222.222 next time.
      I'll check the logs and will post a screenshot as well at that time. Last time I checked, I did see WAN going down but not about the FAILOVER group.
      Thank you!

      1 Reply Last reply Reply Quote 0
      • DerelictD
        Derelict LAYER 8 Netgate
        last edited by

        Did you create outbound NAT rules for WAN2?  It's been a while but I'm pretty sure Multi-WAN isn't handled by Automatic Outbound NAT.

        (ETA: Automatic makes rules for multi-WAN.  I was all wet.  But if you have configured manual outbound you need to duplicate the WAN rules for WAN2).

        Chattanooga, Tennessee, USA
        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
        Do Not Chat For Help! NO_WAN_EGRESS(TM)

        1 Reply Last reply Reply Quote 0
        • B
          bhsense
          last edited by

          OK, we did another test and got the logs, but I didn't change the DNS settings and wasn't able to get all the logs due to time constraint.
          We've also added IPSec rules, hope this doesn't matter. (At least for WAN/FAILOVER connectivity, and DNS for local PC is configured at 8.8.8.8)
          I've attached the logs, but please let me know if I should get another one with the logs right from when we unplugged the port.

          And yes, we have the outbound rule set at Automatic Outbound NAT, I think I'll go ahead and test out manual outbound NAT next time and make sure to copy the rules to the FAILOVER port.
          Thank you very much for the info!

          SystemLog.png
          SystemLog.png_thumb
          GatewayLog.png
          GatewayLog.png_thumb

          1 Reply Last reply Reply Quote 0
          • P
            phil.davis
            last edited by

            I can't spot what is wrong there. I do something similar at home with "FAILOVER" being a device off OPT1 that has a 3G dongle in it for mobile data.

            A traceroute from a LAN client to some internet location when working normally and when it has supposedly failed over would be interesting - see what path the packet is trying to route. (Use the actual IP address if you have difficulty getting DNS when it is failed over)

            /tmp/rules.debug contains the pf rule set - take a copy of that before failover, and then during failover, and compare. pfSense should change things in that to push the traffic out a different interface.

            There has to be some simple setting somewhere that needs changing - this sort of thing works for me in a few locations.

            Anyone else got a good idea?

            As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
            If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

            1 Reply Last reply Reply Quote 0
            • C
              cmb
              last edited by

              Leave it to automatic outbound NAT, if there is any problem there it's in the interface's config. You'll need at least one DNS server going out the second WAN. Though if you can't ping out by IP, that wouldn't fix the issue.

              How is your "Failover" interface configured under Interfaces>Failover? If it's static IP, you must select the gateway on that page. If it's not, you're telling the system that's not an Internet connection, which will break the automatic outbound NAT config as well as a variety of other things.

              1 Reply Last reply Reply Quote 0
              • B
                bhsense
                last edited by

                First of all, thank you everyone for all your help.
                And sorry, I haven't had a chance to test out the DNS side just yet, I'll post the results along with the traceroute and the rule set before/after the failover at that time.

                As far as the interface setup, yes it is set with Static IP with a separate gateway for the failover interface (FAILOVERGW).
                Thanks again.

                1 Reply Last reply Reply Quote 0
                • B
                  bhsense
                  last edited by

                  OK, I finally got the chance to test this out again; it's still not working.
                  The steps I did are as follows, along with the system logs.

                  1. Pluged in both Cables to the pfSense Box
                  2. Started pinging to 8.8.8.8
                  3. Unpluged the cable from the Comcast Modem
                  4. Checked to see if the dynamic dns was updated, got a ping back of the new IP in less than a minute
                  5. Waited 20 minutes for the 8.8.8.8 pings to come back, but no response
                  6. Pluged back the Comcast Modem's cable, ping to 8.8.8.8 returned instantaneously
                  7. Checked to see if the dynamic dns was updated, got a ping back of the new IP in less than a minute
                  8. IPSec VPN did not come back until I restarted the racoon service on the other side (pfSense also)

                  Jan 23 16:07:54 check_reload_status: Linkup starting vr1
                  Jan 23 16:07:54 kernel: vr1: link state changed to DOWN
                  Jan 23 16:07:59 php: rc.linkup: Hotplug event detected for WAN(wan) but ignoring since interface is configured with static IP (... )
                  Jan 23 16:08:12 check_reload_status: updating dyndns WANGW
                  Jan 23 16:08:12 check_reload_status: Restarting ipsec tunnels
                  Jan 23 16:08:12 check_reload_status: Restarting OpenVPN tunnels/interfaces
                  Jan 23 16:08:12 check_reload_status: Reloading filter
                  Jan 23 16:08:21 php: rc.dyndns.update: Default gateway down setting FAILOVERGW as default!
                  Jan 23 16:08:22 php: rc.dyndns.update: MONITOR: WANGW is down, removing from routing group FAILGROUP
                  Jan 23 16:08:22 php: rc.filter_configure_sync: Default gateway down setting FAILOVERGW as default!
                  Jan 23 16:08:22 php: rc.dyndns.update: Default gateway down setting FAILOVERGW as default!
                  Jan 23 16:08:22 php: rc.filter_configure_sync: MONITOR: WANGW is down, removing from routing group FAILGROUP
                  Jan 23 16:08:22 php: rc.dyndns.update: MONITOR: WANGW is down, removing from routing group FAILGROUP
                  Jan 23 16:08:22 php: rc.dyndns.update: Default gateway down setting FAILOVERGW as default!
                  Jan 23 16:08:22 php: rc.dyndns.update: MONITOR: WANGW is down, removing from routing group FAILGROUP
                  Jan 23 16:08:22 php: rc.dyndns.update: Default gateway down setting FAILOVERGW as default!
                  Jan 23 16:08:23 php: rc.dyndns.update: MONITOR: WANGW is down, removing from routing group FAILGROUP
                  Jan 23 16:08:24 php: rc.dyndns.update: phpDynDNS: updating cache file /conf/dyndns_FAILGROUP***'..'0.cache: ...***
                  Jan 23 16:08:24 php: rc.dyndns.update: phpDynDNS (...): (Success) DNS hostname update successful.
                  Jan 23 16:08:36 php: rc.newipsecdns: IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.
                  Jan 23 16:08:37 php: rc.newipsecdns: Default gateway down setting FAILOVERGW as default!
                  Jan 23 16:08:37 php: rc.newipsecdns: MONITOR: WANGW is down, removing from routing group FAILGROUP
                  Jan 23 16:08:37 php: rc.newipsecdns: Default gateway down setting FAILOVERGW as default!
                  Jan 23 16:08:37 php: rc.newipsecdns: MONITOR: WANGW is down, removing from routing group FAILGROUP
                  Jan 23 16:08:37 php: rc.newipsecdns: Default gateway down setting FAILOVERGW as default!
                  Jan 23 16:08:37 php: rc.newipsecdns: MONITOR: WANGW is down, removing from routing group FAILGROUP
                  Jan 23 16:08:37 php: rc.newipsecdns: Default gateway down setting FAILOVERGW as default!
                  Jan 23 16:08:37 php: rc.newipsecdns: MONITOR: WANGW is down, removing from routing group FAILGROUP
                  Jan 23 16:08:37 php: rc.newipsecdns: Default gateway down setting FAILOVERGW as default!
                  Jan 23 16:08:37 php: rc.newipsecdns: MONITOR: WANGW is down, removing from routing group FAILGROUP
                  Jan 23 16:08:37 php: rc.newipsecdns: Default gateway down setting FAILOVERGW as default!
                  Jan 23 16:08:37 php: rc.newipsecdns: MONITOR: WANGW is down, removing from routing group FAILGROUP
                  Jan 23 16:08:37 php: rc.newipsecdns: Default gateway down setting FAILOVERGW as default!
                  Jan 23 16:08:37 php: rc.newipsecdns: MONITOR: WANGW is down, removing from routing group FAILGROUP
                  Jan 23 16:08:37 php: rc.newipsecdns: Default gateway down setting FAILOVERGW as default!
                  Jan 23 16:08:37 php: rc.newipsecdns: MONITOR: WANGW is down, removing from routing group FAILGROUP
                  Jan 23 16:08:37 php: rc.newipsecdns: Default gateway down setting FAILOVERGW as default!
                  Jan 23 16:08:37 php: rc.newipsecdns: MONITOR: WANGW is down, removing from routing group FAILGROUP
                  Jan 23 16:24:38 check_reload_status: Linkup starting vr1
                  Jan 23 16:24:38 kernel: vr1: link state changed to UP
                  Jan 23 16:24:42 php: rc.linkup: Hotplug event detected for WAN(wan) but ignoring since interface is configured with static IP (... )
                  Jan 23 16:24:42 check_reload_status: rc.newwanip starting vr1
                  Jan 23 16:24:47 php: rc.newwanip: rc.newwanip: Informational is starting vr1.
                  Jan 23 16:24:47 php: rc.newwanip: rc.newwanip: on (IP address: ...) (interface: WAN[wan]) (real interface: vr1).
                  Jan 23 16:24:47 check_reload_status: Reloading filter
                  Jan 23 16:24:49 check_reload_status: updating dyndns WANGW
                  Jan 23 16:24:49 check_reload_status: Restarting ipsec tunnels
                  Jan 23 16:24:49 check_reload_status: Restarting OpenVPN tunnels/interfaces
                  Jan 23 16:24:49 check_reload_status: Reloading filter
                  Jan 23 16:25:00 php: rc.dyndns.update: phpDynDNS: updating cache file /conf/dyndns_FAILGROUP***'..'0.cache: ...***
                  Jan 23 16:25:00 php: rc.dyndns.update: phpDynDNS (..***): (Success) DNS hostname update successful.
                  Jan 23 16:25:12 php: rc.newipsecdns: IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.

                  And, sorry for not getting this info out initially, but we did try to install squid and run it first, although we didn't succeed so we stopped it and uninstalled this.
                  After that we even did a factory reset, but could it be that this is still running in the background?

                  Any help is greatly appreciated, thank you.

                  1 Reply Last reply Reply Quote 0
                  • DerelictD
                    Derelict LAYER 8 Netgate
                    last edited by

                    Did you leave the same ping running and failover the ports?  I don't think failover moves states from one interface to the other does it?  That would be a trick, considering the inside global address changes.

                    Did you try stopping the ping and starting a new one while it was failed over?

                    It minimizes downtime but it's not magic.

                    Chattanooga, Tennessee, USA
                    A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                    DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                    Do Not Chat For Help! NO_WAN_EGRESS(TM)

                    1 Reply Last reply Reply Quote 0
                    • B
                      bhsense
                      last edited by

                      Thank you for your support!
                      We've tried stopping and starting a new command prompt to ping, but it was still not working.
                      Then we've noticed that the new version 2.2 came out, so we tested it out and we are now to browse the web just fine. Thank you again for all your support!

                      But now when setting up the VPN with the multi-wan setup, it's not working.
                      If someone can shed some light on this problem, that would be great.
                      Here's our setup:

                      1. Setup Dynamic DNS for the Failover Group
                      2. Setup VPN with Interface as the Failover Group
                      3. Setup VPN on the other side with Remote Gateway as the Dynamic DNS hostname

                      The VPN is working fine if everything is connected.
                      When uplugging the main WAN line, Dynamic DNS does get updated; pings start going to the alternate IP, and the pfSense on the other side does try to connect to the alternate IP.
                      However, although the IPSec Status does show the Local IP as the alternate IP address, the VPN connection status gets stuck at "Connecting…".
                      Looking at the logs, it does show that it is attempting to still connect with the main IP. Is there a setting we missed somewhere?

                      1 Reply Last reply Reply Quote 0
                      • B
                        bhsense
                        last edited by

                        After writing the above, it seems that the VPN connection has also become unstable… it will lose connection about 3-4 times a day.
                        When the connection gets lost, I was still able to access the local firewall from the other side of the VPN Connection, but the local users weren't able to ping the other side. Restarting the racoon service on the local firewall seems to fix the issue temporarily.
                        I'll update this post if I find anything new; any advice is welcome.
                        Thank you!

                        1 Reply Last reply Reply Quote 0
                        • B
                          bhsense
                          last edited by

                          Now it seems that I can't even connect from the other side as well…
                          As I look at my VPN status it says connected, but there was 4 child SA entries with 0 Bytes-in. Now when I look at my Key Exchange version it says V1 so I'll try to switch to V2 and see how it goes.
                          I'll update the results here, thank you,

                          1 Reply Last reply Reply Quote 0
                          • B
                            bhsense
                            last edited by

                            OK, here's the final update!
                            After going through the VPN forum, the general consensus was the IPSec in version 2.2 is somewhat broken, so we changed the VPN settings to OpenVPN.
                            We tested the failover, and it switched over to the failover port and re-established the VPN connection in about 15 seconds. All is good! (Now I'm regretting getting the dynamic dns service set up, but anyway)
                            So I'll mark this as officially solved, thank you very much for all your support.
                            Best regards.

                            1 Reply Last reply Reply Quote 0
                            • First post
                              Last post
                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.