Navigation

    Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    2 HA pfsense boxes with 1 public IP working but…

    HA/CARP/VIPs
    3
    27
    4382
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • F
      francesco1 last edited by

      For some reason after a few hours it seems to stop working. This is what happens: everything seems fine until I unplug the master WAN port and fail-over works but when I plug the master WAN port back in, the internet stops working. It appears the master role goes back to the master pfsense but nothing goes through. If I unplug the master WAN it fails over to the slave and internet works.

      If I go to Interfaces -> WAN and click on SAVE (on the master), internet comes back online but I lose connectivity to the slave pfsense web interface. The only way I could find to solve that problem, is to reboot the slave and all is good after that (for a few hours at least). Happened twice so far.

      This is the setup:

      WAN                      LAN                      PFSYNC
      PF1  10.10.10.1/24  192.168.111.1/24  172.16.222.1/30
      PF2  10.11.11.1/24  192.168.111.2/24  172.16.222.2/30
      CARP 1 Public IP        192.168.111.3/24
      GW    Public IP

      It works… for a few hours. Then I unplug the master WAN and, when I plug it back in, internet just stops working as explained above.

      I'm using v2.3.2

      What would cause this?

      1 Reply Last reply Reply Quote 0
      • F
        francesco1 last edited by

        BTW, I am using v2.3.2

        1 Reply Last reply Reply Quote 0
        • F
          francesco1 last edited by

          I wiped out 2.3.2 and installed 2.2.6 from scratch and same thing.

          What I noticed is that  the problem only happens when the WAN goes down. If I unplug LAN, WAN and SYNC, I have to plug in WAN first then SYNC and LAN. Also, when the VM cannot get on the internet, neither the laptop connected to the LAN can.

          ????

          1 Reply Last reply Reply Quote 0
          • Derelict
            Derelict LAYER 8 Netgate last edited by

            Define "stops working." Do some Connectivity Troubleshooting and determine what is actually failing and report back.

            Chattanooga, Tennessee, USA
            The pfSense Book is free of charge!
            DO NOT set a source port in a port forward or firewall rule unless you KNOW you need it!
            Do Not Chat For Help! NO_WAN_EGRESS(TM)

            1 Reply Last reply Reply Quote 0
            • F
              francesco1 last edited by

              Stops working as in no longer being able to get on the internet. Says ping timeout or host unreachable.

              1 Reply Last reply Reply Quote 0
              • Derelict
                Derelict LAYER 8 Netgate last edited by

                Need more details such as what can't ping what from where. You are the one at the site who can properly troubleshoot your network. Nobody else is.

                Chattanooga, Tennessee, USA
                The pfSense Book is free of charge!
                DO NOT set a source port in a port forward or firewall rule unless you KNOW you need it!
                Do Not Chat For Help! NO_WAN_EGRESS(TM)

                1 Reply Last reply Reply Quote 0
                • dotdash
                  dotdash last edited by

                  You should have your WAN interfaces on the same bogus network, your setup shows them in separate /24s. When I did it, I  used a private /30 as the public IP was a /30. That's probably not needed.

                  1 Reply Last reply Reply Quote 0
                  • F
                    francesco1 last edited by

                    @Derelict: I was pinging various valid IP addresses on the internet.

                    @dotdash: I used a /24 subnet because that is what the ISP is giving here in the test lab. I final setup supposed to be a /30.

                    Today, I changed the setup. I took the whole thing off the internet and setup "my own internet" with public IP addresses and set up pfSense to route public IP addresses from the WAN to a DMZ. I got it to work but displays the same behavior as the previous setup.

                    Basically, if I unplug the master firewall's WAN cable, routing switches to the slave firewall properly within a second. When I see the ping timeout and then resumes receiving replies I plug the WAN cable back in to the master firewall. Sometimes it switches back to the master without a problem but sometimes it does not.

                    When it does not come back successfully, I unplug the master firewall's WAN cable again and the slave takes over successfully. It seems like I have to unplug all of the cables and plug them in an order where the WAN is plugged in first and then the rest.

                    1 Reply Last reply Reply Quote 0
                    • Derelict
                      Derelict LAYER 8 Netgate last edited by

                      For starters, stop unplugging cables. Yes, you should test eventually but start with basic functionality.

                      All of your CARP VIPs should be MASTER on primary and BACKUP on secondary. If not, fix that.

                      Use the Temporarily Disable CARP button on Status > CARP on the primary. All of your CARP VIPs should go MASTER on the secondary (Again, Status > CARP on the secondary). If they do not, fix that.

                      What documentation did you follow in setting up your HA cluster?

                      Chattanooga, Tennessee, USA
                      The pfSense Book is free of charge!
                      DO NOT set a source port in a port forward or firewall rule unless you KNOW you need it!
                      Do Not Chat For Help! NO_WAN_EGRESS(TM)

                      1 Reply Last reply Reply Quote 0
                      • F
                        francesco1 last edited by

                        I used this document:
                        https://doc.pfsense.org/index.php/Configuring_pfSense_Hardware_Redundancy_%28CARP%29

                        The IP, CARP and HA part works; just not 100% of the time. All CARP related settings, rules etc., copy from master to slave.

                        I’m unplugging cables to test fault tolerance. What if the WAN connection goes down for 3 seconds and comes back up? If the master doesn’t come back online properly, then everything going through the firewalls will go down. Is unplugging the WAN not a valid test? This behavior that I am seeing with pfSense HA something acceptable?

                        You can see what I’m talking about in this video https://youtu.be/pepq4VLOUHE

                        This is the current setup:

                        “ISP” SIDE CONFIGURATION

                        66.66.77.10/24 Windows Server

                        “ISP“ Router
                        66.66.77.1/24 Public DMZ Interface
                        66.66.66.1/30 WAN Interface

                        MY SIDE CONFIGURATION

                        PFS1
                        10.10.10.1/30 WAN Interface Dummy IP
                        66.66.66.2/30 WAN CARP IP
                        77.77.77.2/24 Public DMZ Interface
                        77.77.77.1/24 Public DMZ Interface CARP IP

                        PFS2
                        10.11.11.1/30 WAN Interface Dummy IP
                        66.66.66.2/30 WAN CARP IP
                        77.77.77.3/24 Public DMZ Interface
                        77.77.77.1/24 Public DMZ Interface CARP IP

                        77.77.77.40/24 Windows Server

                        1 Reply Last reply Reply Quote 0
                        • Derelict
                          Derelict LAYER 8 Netgate last edited by

                          What is a "Dummy IP?"

                          So you are trying to do HA without the required 3 public IP addresses (minimum /29) on WAN?

                          You are still not providing the necessary details to properly help you diagnose your problem. Run through the connectivity troubleshooting steps and say what actually fails and where.

                          Chattanooga, Tennessee, USA
                          The pfSense Book is free of charge!
                          DO NOT set a source port in a port forward or firewall rule unless you KNOW you need it!
                          Do Not Chat For Help! NO_WAN_EGRESS(TM)

                          1 Reply Last reply Reply Quote 0
                          • dotdash
                            dotdash last edited by

                            @Francesco:

                            MY SIDE CONFIGURATION

                            PFS1
                            10.10.10.1/30 WAN Interface Dummy IP

                            PFS2
                            10.11.11.1/30 WAN Interface Dummy IP

                            I pointed this out before, but you need to have them in the same subnet. Why not try 10.10.10.1/30 and 10.10.10.2/30 ??

                            1 Reply Last reply Reply Quote 0
                            • F
                              francesco1 last edited by

                              @Derelict:

                              What is a "Dummy IP?"

                              So you are trying to do HA without the required 3 public IP addresses (minimum /29) on WAN?

                              You are still not providing the necessary details to properly help you diagnose your problem. Run through the connectivity troubleshooting steps and say what actually fails and where.

                              Yes, I am trying to do HA without 3 public IP addresses on WAN. What other info do you need? Dummy IP = any IP placed in the IPv4 Address field of the WAN Interface that is not going to conflict with any other interface, host etc. The "Dummy IP" is required because PFS requires you to enter an IP address in that field. And, as I said before, I do NOT have the option of using more than 1 public IP address and overall seems to work.

                              @dotdash:

                              @Francesco:

                              MY SIDE CONFIGURATION

                              PFS1
                              10.10.10.1/30 WAN Interface Dummy IP

                              PFS2
                              10.11.11.1/30 WAN Interface Dummy IP

                              I pointed this out before, but you need to have them in the same subnet. Why not try 10.10.10.1/30 and 10.10.10.2/30 ??

                              Good question. I tried it before and it didn't make a difference. I will try it again.

                              1 Reply Last reply Reply Quote 0
                              • F
                                francesco1 last edited by

                                @dotdash:

                                @Francesco:

                                MY SIDE CONFIGURATION

                                PFS1
                                10.10.10.1/30 WAN Interface Dummy IP

                                PFS2
                                10.11.11.1/30 WAN Interface Dummy IP

                                I pointed this out before, but you need to have them in the same subnet. Why not try 10.10.10.1/30 and 10.10.10.2/30 ??

                                Just tried it again and same problem.

                                1 Reply Last reply Reply Quote 0
                                • Derelict
                                  Derelict LAYER 8 Netgate last edited by

                                  Again - pretty sparse with the details.

                                  Chattanooga, Tennessee, USA
                                  The pfSense Book is free of charge!
                                  DO NOT set a source port in a port forward or firewall rule unless you KNOW you need it!
                                  Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                  1 Reply Last reply Reply Quote 0
                                  • F
                                    francesco1 last edited by

                                    @Derelict:

                                    Again - pretty sparse with the details.

                                    Again, what more details do you need??? I asked what details you need before. I'll provide the details needed.

                                    1 Reply Last reply Reply Quote 0
                                    • Derelict
                                      Derelict LAYER 8 Netgate last edited by

                                      You are still not providing the necessary details to properly help you diagnose your problem. Run through the connectivity troubleshooting steps and say what actually fails and where.

                                      Help us help you.

                                      Chattanooga, Tennessee, USA
                                      The pfSense Book is free of charge!
                                      DO NOT set a source port in a port forward or firewall rule unless you KNOW you need it!
                                      Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                      1 Reply Last reply Reply Quote 0
                                      • F
                                        francesco1 last edited by

                                        @Derelict:

                                        You are still not providing the necessary details to properly help you diagnose your problem. Run through the connectivity troubleshooting steps and say what actually fails and where.

                                        Help us help you.

                                        I have no idea what details you are looking for. All I know right now is this:

                                        1. I start ping command on server on both ends
                                        2. Unplug Master Firewall WAN connection
                                        3. I see Slave Firewall takes over when a packet is dropped and ping resumes
                                        4. Right after that I plug the Master Firewall WAN connection
                                        5. Master Firewall takes over but all pings fail
                                        1 Reply Last reply Reply Quote 0
                                        • Derelict
                                          Derelict LAYER 8 Netgate last edited by

                                          Details like what are you pinging from where when you're testing. Specifics, like interfaces and IP addresses.

                                          Chattanooga, Tennessee, USA
                                          The pfSense Book is free of charge!
                                          DO NOT set a source port in a port forward or firewall rule unless you KNOW you need it!
                                          Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                          1 Reply Last reply Reply Quote 0
                                          • F
                                            francesco1 last edited by

                                            I am pinging 77.77.77.40 from 66.66.77.10. I am also pinging 66.66.77.10 from 77.77.77.40.

                                            Diagram of setup is attached. FYI, this is a closed lab setup.


                                            1 Reply Last reply Reply Quote 0
                                            • Derelict
                                              Derelict LAYER 8 Netgate last edited by

                                              What address is 77.77.77.1/X routed to on 66.66.66 ?

                                              Chattanooga, Tennessee, USA
                                              The pfSense Book is free of charge!
                                              DO NOT set a source port in a port forward or firewall rule unless you KNOW you need it!
                                              Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                              1 Reply Last reply Reply Quote 0
                                              • F
                                                francesco1 last edited by

                                                @Derelict:

                                                What address is 77.77.77.1/X routed to on 66.66.66 ?

                                                I do not understand your question.

                                                1 Reply Last reply Reply Quote 0
                                                • Derelict
                                                  Derelict LAYER 8 Netgate last edited by

                                                  How do the 77.77.77.X addresses get from the ISP to you? They have to be routed to you somehow.

                                                  Chattanooga, Tennessee, USA
                                                  The pfSense Book is free of charge!
                                                  DO NOT set a source port in a port forward or firewall rule unless you KNOW you need it!
                                                  Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                                  1 Reply Last reply Reply Quote 0
                                                  • F
                                                    francesco1 last edited by

                                                    @Derelict:

                                                    How do the 77.77.77.X addresses get from the ISP to you? They have to be routed to you somehow.

                                                    If you are asking me where the 77.77.77.0/24 addresses come from, that does not matter because this  is a lab designed for testing only with no real connection to the Internet. It is only for testing PFS HA functionality.

                                                    If you are asking what am I using to route IP traffic between 66.66.66.0/30 and 77.77.77.0/24, I am using a separate installation of PFS.

                                                    1 Reply Last reply Reply Quote 0
                                                    • Derelict
                                                      Derelict LAYER 8 Netgate last edited by

                                                      OK it doesn't matter if it is routed to the CARP VIP or not. I'm done.

                                                      Chattanooga, Tennessee, USA
                                                      The pfSense Book is free of charge!
                                                      DO NOT set a source port in a port forward or firewall rule unless you KNOW you need it!
                                                      Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                                      1 Reply Last reply Reply Quote 0
                                                      • F
                                                        francesco1 last edited by

                                                        @Derelict:

                                                        OK it doesn't matter if it is routed to the CARP VIP or not. I'm done.

                                                        That's fine. You weren't reading the details, exaggerating information needed and making things more difficult than they really are.

                                                        1 Reply Last reply Reply Quote 0
                                                        • dotdash
                                                          dotdash last edited by

                                                          @Francesco:

                                                          That's fine. You weren't reading the details, exaggerating information needed and making things more difficult than they really are.

                                                          No, he was trying to point out that you could have a dozen different things wrong with your lab setup which no one can easily sort out. e.g. something on the WAN side by default can't ping your LAN, so the fact that 77 whatever can't ping 66 whatever is probably irrelevant; the 'isp router' config is unknown, etc…  I'm not sure what you are trying to test with your methodology either, someone unplugging the WAN on the master seems an unlikely event. The HA failures I've dealt with usually involve failed hardware. If I was going to test, I'd pull power on the master and see what happens. Anyway, you are seeking free assistance from strangers on the Internet. If you don't want to work with someone who steps up, fine, but don't be offended if no one else wants to spend time trying to figure out what's wrong with your setup.

                                                          1 Reply Last reply Reply Quote 0
                                                          • First post
                                                            Last post