Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Backhaul

    Scheduled Pinned Locked Moved Routing and Multi WAN
    37 Posts 6 Posters 4.5k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A Offline
      afreaken
      last edited by

      Well I guess my rules are fine, I for some reason cannot ping the gateway on the remote side as well as the wifi router, however now that I have fixed the pfsense box rules, I can connect to the box from this side. cannot ping any of it's interfaces though, or the remote network

      1 Reply Last reply Reply Quote 0
      • DerelictD Offline
        Derelict LAYER 8 Netgate
        last edited by

        You need static routes to the remote network.  Did you allow any traffic or just TCP/UDP?  If the latter, it will block ICMP (ping).

        Chattanooga, Tennessee, USA
        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
        Do Not Chat For Help! NO_WAN_EGRESS(TM)

        1 Reply Last reply Reply Quote 0
        • A Offline
          afreaken
          last edited by

          @Derelict:

          You need static routes to the remote network.  Did you allow any traffic or just TCP/UDP?  If the latter, it will block ICMP (ping).

          Allow Any, Figured out the ping issue, there was an old rule set that I had to delete. Will set statics now. This is where I have been confused before, when I set the static for a remote network over the multiWAN WiFi backhaul, it wants to choose an interface, and I cannot replicate the rule over another interface. Does the multiWAN gateway handle the second route?

          I noticed if I use an alias it seems to allow me to set statics for the same alias across multiple GW's, is this OK?

          1 Reply Last reply Reply Quote 0
          • DerelictD Offline
            Derelict LAYER 8 Netgate
            last edited by

            You need static routes for the cisco to pfsense and pfsense to cisco paths.  From pfSense over the wireless links, use policy routing on your LAN rules to the gateway group.

            The Cisco1 needs to know how to get to 192.168.2.0/24.  It needs a route for 192.168.2.0/24 to pfSense 1.
            pfSense 1 needs to know how to get to 192.168.2.0/24. It needs a policy route on LAN source any dest 192.168.2.0/24 with the wireless gateway group set.
            pfSense 2 needs to know how to get to 192.168.2.0/24.  It needs a gateway created for Cisco2. and a static route for 192.168.2.0/24 to Cisco2.
            Cisco2 does not need a route because 192.168.2.0/24 is a connected network.

            And the reverse for the other direction.  This is assuming the wireless links/pfSense is not the default gateway at either location.  in that case just point the default to the next upstream device.

            Chattanooga, Tennessee, USA
            A comprehensive network diagram is worth 10,000 words and 15 conference calls.
            DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
            Do Not Chat For Help! NO_WAN_EGRESS(TM)

            1 Reply Last reply Reply Quote 0
            • ? This user is from outside of this forum
              Guest
              last edited by

              @Derelict:

              My point is why not just failover on the Ciscos?

              Ok this is true, but on the other hand why not laod balance using policy based routing
              between the pfSense firewalls?

              1 Reply Last reply Reply Quote 0
              • P Offline
                phil.davis
                last edited by

                I am not sure why you are using floating rules - that is where the confusion about In/Out is coming from, floating rules let you apply the rule on traffic coming In or Out of an Interface, there is also the terminology of In/Out for limiters - a different place in the GUI and different thing.

                I think you really want:

                1. Static route on each pfSense pointing to its local Cisco for a route to the subnet behind the Cisco.
                2. Do not do any NAT on the "WAN" backhaul interfaces - perhaps just make those ordinary LAN-style interfaces (no Upstream Gateway defined)
                3. Gateway defined for the IP address on the other side of each "WAN" backhaul link.
                4. Gateway group/s that include the 2 gateways with whatever tiers you want to make it load-balance or fail-over.
                5. Pass rule/s on the LAN interface to pass traffic source: the subnet behind the Cisco (and the local pfSense LAN subnet for completeness), destination: the subnet behind remote end Cisco (and the remote end pfSense LAN subnet for completeness), gateway: the gateway group you made.
                6. Pass rule/s on the WAN backhaul interfaces to allow incoming traffic from the other end (or just allow all to get it going).
                  (These "pass" rules on individual interfaces are "In" rules - you do not get a choice about that - they will apply to traffic flows initiated from the interface they are on)

                It should all work in a conceptually similar way to having a couple of site-to-site VPN tunnels between pfSense systems and routing intranet traffic across them.

                As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
                If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

                1 Reply Last reply Reply Quote 0
                • A Offline
                  afreaken
                  last edited by

                  @phil.davis:

                  I am not sure why you are using floating rules - that is where the confusion about In/Out is coming from, floating rules let you apply the rule on traffic coming In or Out of an Interface, there is also the terminology of In/Out for limiters - a different place in the GUI and different thing.

                  I think you really want:

                  1. Static route on each pfSense pointing to its local Cisco for a route to the subnet behind the Cisco.
                  2. Do not do any NAT on the "WAN" backhaul interfaces - perhaps just make those ordinary LAN-style interfaces (no Upstream Gateway defined)
                  3. Gateway defined for the IP address on the other side of each "WAN" backhaul link.
                  4. Gateway group/s that include the 2 gateways with whatever tiers you want to make it load-balance or fail-over.
                  5. Pass rule/s on the LAN interface to pass traffic source: the subnet behind the Cisco (and the local pfSense LAN subnet for completeness), destination: the subnet behind remote end Cisco (and the remote end pfSense LAN subnet for completeness), gateway: the gateway group you made.
                  6. Pass rule/s on the WAN backhaul interfaces to allow incoming traffic from the other end (or just allow all to get it going).
                    (These "pass" rules on individual interfaces are "In" rules - you do not get a choice about that - they will apply to traffic flows initiated from the interface they are on)

                  It should all work in a conceptually similar way to having a couple of site-to-site VPN tunnels between pfSense systems and routing intranet traffic across them.

                  Thanks, I will try this out. What Derelict recommended may work, I need to go to the remote site and fix/replace the remote pfsense box. It gets stuck in the shutdown process when restarting. I got sick last week and took time off so I haven't gotten around to it yet. I will try both and respond back with the results.

                  1 Reply Last reply Reply Quote 0
                  • A Offline
                    afreaken
                    last edited by

                    @afreaken:

                    @phil.davis:

                    I am not sure why you are using floating rules - that is where the confusion about In/Out is coming from, floating rules let you apply the rule on traffic coming In or Out of an Interface, there is also the terminology of In/Out for limiters - a different place in the GUI and different thing.

                    I think you really want:

                    1. Static route on each pfSense pointing to its local Cisco for a route to the subnet behind the Cisco.
                    2. Do not do any NAT on the "WAN" backhaul interfaces - perhaps just make those ordinary LAN-style interfaces (no Upstream Gateway defined)
                    3. Gateway defined for the IP address on the other side of each "WAN" backhaul link.
                    4. Gateway group/s that include the 2 gateways with whatever tiers you want to make it load-balance or fail-over.
                    5. Pass rule/s on the LAN interface to pass traffic source: the subnet behind the Cisco (and the local pfSense LAN subnet for completeness), destination: the subnet behind remote end Cisco (and the remote end pfSense LAN subnet for completeness), gateway: the gateway group you made.
                    6. Pass rule/s on the WAN backhaul interfaces to allow incoming traffic from the other end (or just allow all to get it going).
                      (These "pass" rules on individual interfaces are "In" rules - you do not get a choice about that - they will apply to traffic flows initiated from the interface they are on)

                    It should all work in a conceptually similar way to having a couple of site-to-site VPN tunnels between pfSense systems and routing intranet traffic across them.

                    Thanks, I will try this out. What Derelict recommended may work, I need to go to the remote site and fix/replace the remote pfsense box. It gets stuck in the shutdown process when restarting. I got sick last week and took time off so I haven't gotten around to it yet. I will try both and respond back with the results.

                    After doing my testing, I got the setup working. Used this as the guide and it does work. The pfsense box is deployed on 1 end, the end which was in need of being replaced. Had a few issues which needed to be resolved as we have several networks to join up and allow traffic to, however they are not critical to business processes, only critical for IT and some small use cases for some users. They are resolved afaik, guess I'll find out when something doesn't work and someone complains about it to me.

                    A big thank you to phil.davis and Derelict for the helpful input. I learned a lot through the input from you two. Also thanks to the others who gave input, promoting useful discussion of this topic.

                    1 Reply Last reply Reply Quote 0
                    • A Offline
                      afreaken
                      last edited by

                      One issue I've been seeing is that sometimes pings will not resolve going across the wifi link between the pfsense and opposite linux box. The ping is going from client to server or server to client, both do not resolve, and not all clients experience the problems. They are on the same network.

                      1 Reply Last reply Reply Quote 0
                      • DerelictD Offline
                        Derelict LAYER 8 Netgate
                        last edited by

                        What does "resolve" mean in the context of using ping/icmp?

                        Chattanooga, Tennessee, USA
                        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                        Do Not Chat For Help! NO_WAN_EGRESS(TM)

                        1 Reply Last reply Reply Quote 0
                        • A Offline
                          afreaken
                          last edited by

                          @Derelict:

                          What does "resolve" mean in the context of using ping/icmp?

                          requests time out. Also, if a gateway goes down in a group, clients that were once able to ping, can no longer ping.

                          1 Reply Last reply Reply Quote 0
                          • A Offline
                            afreaken
                            last edited by

                            well this may be unnecessary. I replaced the box on this side with a pfsense box and the timeouts stopped. It may have been an issue with the linux box on this side. Will post with updates after it's been running overnight.

                            1 Reply Last reply Reply Quote 0
                            • A Offline
                              afreaken
                              last edited by

                              One question I have is how do I give this box access to the internet? Normally you would have a wan connected to the internet, but this is routing for a private network. Do I need to create policy routing?

                              pfsense box [LAN port 10.2] –-- [10.1 WiFi backhaul port] Cisco [to external IP]

                              1 Reply Last reply Reply Quote 0
                              • DerelictD Offline
                                Derelict LAYER 8 Netgate
                                last edited by

                                NAT for 10.2 on the Cisco.  Set pfSense's default gateway to 10.1 and configure DNS, etc.

                                Chattanooga, Tennessee, USA
                                A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                1 Reply Last reply Reply Quote 0
                                • DerelictD Offline
                                  Derelict LAYER 8 Netgate
                                  last edited by

                                  Also, if you're pinging across a load-balanced link and the link your state is on goes down, your ping will stop because the state doesn't move to the other circuit.  Stopping and restarting the ping should result in pings again as a new state is created using the other link.

                                  Chattanooga, Tennessee, USA
                                  A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                  DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                  Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                  1 Reply Last reply Reply Quote 0
                                  • A Offline
                                    afreaken
                                    last edited by

                                    @Derelict:

                                    Also, if you're pinging across a load-balanced link and the link your state is on goes down, your ping will stop because the state doesn't move to the other circuit.  Stopping and restarting the ping should result in pings again as a new state is created using the other link.

                                    Which is what I tried to do, however the machiens would still not return a ping. I tried even resetting the states within the pfsense box, but nothing happened. There are some issues with the linux boxes we are replacing, and this seems to be one of them. After switching out both ends, compared to only 1 end, pings have been very stable. A lot of the routing it should be doing, doesn't seem to be done. The interface will say one thing, but it will do something else completely. We also have little control over the backend of the linux box.

                                    1 Reply Last reply Reply Quote 0
                                    • A Offline
                                      afreaken
                                      last edited by

                                      @afreaken:

                                      well this may be unnecessary. I replaced the box on this side with a pfsense box and the timeouts stopped. It may have been an issue with the linux box on this side. Will post with updates after it's been running overnight.

                                      Well things have been running smoothly for the past few days, things are looking good.

                                      1 Reply Last reply Reply Quote 0
                                      • First post
                                        Last post
                                      Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.