• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Routing problem with a bit complex network setup

Scheduled Pinned Locked Moved General pfSense Questions
9 Posts 4 Posters 414 Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • L
    Leksandr
    last edited by Leksandr Apr 22, 2020, 4:42 PM Apr 22, 2020, 10:38 AM

    Hello

    Introduction: I set up my lab enviroment on separate LAN and ISP. I set up and successfully tested WAN failover on both of my pfsenses (if ISP 2 go down, "pfsense 1" will use "Home router" and its LAN IP as a gateway, "pfsense 2" will use "pfsense 1" LAN3 IP as a gateway). Everything in this setup works fine, but I want to measure latency of each gateways and log these numbers to my Zabbix server. Network diagram below.

    Updated diagram

    simplified network diagram v2.png

    Old and incorrect diagram

    simplified network diagram.png

    My problem is that I can't ping (Diagnostics -> Ping -> manually choose source address) 8.8.8.8 from "pfsense 1" (LAN1 port) using "Home router" as a gateway. By "can't" I mean that the result pinging 8.8.8.8 using "ISP 2" (fron WAN port) and "Home router" (from LAN1 port) are the same.

    The thing gets different, when I change "IPv4 Upstrem gateway" setting on "pfsense 1" LAN1 port from "None" to "Home router IP". With this change, ping results are finally different, but PC in LAN1 with manually entered gateway IP address of "pfsense 1" LAN1 port couldn't get Internet connection.

    At the same time, WAN failover with IPS 2 has been disconnected AND "IPv4 Upstrem gateway" setting changed works perfectly. But when I restore ISP 2 connection, PC from LAN 1 cannot get access to Internet using pfsense LAN1 as a gatway.

    My ultimate goals are: measure latency of each of my gateways and log those nubmers to Zabbix AND at the same time, be able to access internet from LAN 1 using "pfsense 1" "LAN1" IP as a gateway. How I can achieve these goals? :)

    1 Reply Last reply Reply Quote 0
    • S
      stephenw10 Netgate Administrator
      last edited by stephenw10 Apr 22, 2020, 2:37 PM Apr 22, 2020, 2:37 PM

      Hmm, I have many questions about the way this is setup! It seems unnecessarily complex and full of things that might break routing, as you are seeing. 😬

      Are the two pfSense instances running as HA? It seems they cannot be from your description.

      In which case why do you have two pfSense instances?

      You are almost certainly hitting some asymmetric routing here. I imagine that why hosts in the LAN1 subnet cannot connect out when using the LAN1 IP as a gateway. They are probably being redirected to the home router unless you've disabled that.

      You should be able to ping 8.8.8.8 from each gateway independently. However if you have used 8.8.8.8 as a local dns server or as a gateway monitoring IP anywhere that will have added a static route via that gateway.

      Steve

      L 1 Reply Last reply Apr 22, 2020, 5:37 PM Reply Quote 0
      • L
        Leksandr @stephenw10
        last edited by Apr 22, 2020, 5:37 PM

        @stephenw10 sorry, I attached wrong diagram, now I updated it and edit my post )

        Also, I added some description.

        Are the two pfSense instances running as HA?
        No, they are two completely independent instance of pfsense with different IP addresses and WAN IPs. Each of them is primary for their own LAN. "Pfsense 1" is primary for LAN 2 (PROD LAN) and "Pfsense 2" is primary for LAN 3 (LAB LAN).

        In which case why do you have two pfSense instances?
        Because I want to test all new cool and fancy features of some packages at home in complete separate environment, that is why I need LAB LAN with pfsense 2.
        But at the same time, I want to host some of my application at home, that is why I need PROD LAN with pfsense 1.

        Additionally, I do not want that any of my tests reflect on my home network in any way AND I want to replace my "Home router" with pfsense (virtualized one) in the near future, when this setup will be stable enough.

        You are almost certainly hitting some asymmetric routing here. I imagine that why hosts in the LAN1 subnet cannot connect out when using the LAN1 IP as a gateway. They are probably being redirected to the home router unless you've disabled that.
        I did not disable "Home router", its running alongside this setup. Hmm... I will try to packet capture to figure out whats could be wrong with redirection.

        You should be able to ping 8.8.8.8 from each gateway independently. However if you have used 8.8.8.8 as a local dns server or as a gateway monitoring IP anywhere that will have added a static route via that gateway.
        And I am able to ping 8.8.8.8 from each gateway independently, but it looks like that Pfsense could have only one active gateway at a time.
        But it looks like "Gateway monitoring" works differently than simple ping (fping in my case) inside Pfsense. And for some reason I was unable to use ping (not fping) with -I option, but fping with --src option seems to work fine, but only if I change "IPv4 Upstrem gateway" setting for LAN 1 on pfsense 1.

        pfsense-ping-with-i-option.png

        I set up redundant WAN connection according this manual - https://techexpert.tips/pfsense/pfsense-multiple-wan-failover-configuration/

        1 Reply Last reply Reply Quote 0
        • N
          NollipfSense
          last edited by Apr 22, 2020, 6:07 PM

          All I can say is that keeping it simple always wins!

          pfSense+ 23.09 Lenovo Thinkcentre M93P SFF Quadcore i7 dual Raid-ZFS 128GB-SSD 32GB-RAM PCI-Intel i350-t4 NIC, -Intel QAT 8950.
          pfSense+ 23.09 VM-Proxmox, Dell Precision Xeon-W2155 Nvme 500GB-ZFS 128GB-RAM PCIe-Intel i350-t4, Intel QAT-8950, P-cloud.

          L 1 Reply Last reply Apr 22, 2020, 6:31 PM Reply Quote 0
          • L
            Leksandr @NollipfSense
            last edited by Apr 22, 2020, 6:31 PM

            @NollipfSense and I try to keep it as simply as possible, but I need to completely separate LAN with separate routers. And I need to measure each of my gateways latency in zabbix somehow (

            1 Reply Last reply Reply Quote 0
            • J
              johnpoz LAYER 8 Global Moderator
              last edited by Apr 22, 2020, 6:45 PM

              I'm with @stephenw10 here, you are most likely going to see some asymmetrical issues.. Without looking specifics - because non where given.. My gut reaction to what looks like your using your lan off of your Home router as a transit into pfsense 1.. If you have any devices on this home lan asymmetrical is going to be problematic..

              If you want to use pfsense downstream from this isp 1 connections router, then that should be a actual transit network off your home router with no devices in it.. And to pfsense 1 this would be a wan connection..

              An intelligent man is sometimes forced to be drunk to spend time with his fools
              If you get confused: Listen to the Music Play
              Please don't Chat/PM me for help, unless mod related
              SG-4860 24.11 | Lab VMs 2.7.2, 24.11

              1 Reply Last reply Reply Quote 0
              • S
                stephenw10 Netgate Administrator
                last edited by Apr 22, 2020, 8:59 PM

                I am assuming you are not using 8.8.8.8 as a DNS server or gateway monitor target anywhere, you didn't confirm that?

                You can specify a source IP to ping from and it will use that. But in yout case you are using a LAN interface as a WAN. If you do not have a gateway defined on that interface then pfSense will treat it as a LAN in the default config. That means that with outbound NAT still in automatic mode there will be NAT rules for the LAN1 subnet leaving the WAN. When you specify the LAN IP as source that traffic leaves the default route and is NAT'd.
                You will probably need to use hybrid or manual outbound NAT mode because it looks like you're trying to use that interface as both a WAN and a LAN.

                If you set 8.8.8.8 as the LAN1 gateway monitor IP it will set a static route for that via the LAN1 gateway and you should get the monitoring you want.

                Steve

                1 Reply Last reply Reply Quote 0
                • L
                  Leksandr
                  last edited by Apr 23, 2020, 12:27 AM

                  @stephenw10 @johnpoz @NollipfSense
                  Thank you all very much! Problem solved.

                  I've changed "IPv4 Upstrem gateway" setting on "LAN1" interface of "pfsense 1" to IP of my home router, then I've checked packet capture results on "pfsense 1" WAN port while trying to access Internet from PC in LAN_HOME subnet. It turned out that traffic comes with no NAT, thats why my PC lose Internet connectivity.

                  So I changed "Outbound NAT Mode" to "Hybrid Outbound NAT" and added rule for LAN_HOME, after that everything works fine.

                  To sum up

                  1. I thought that pfsense's Outbound NAT Mode would NAT every incoming packet to its gateway. I was wrong :(
                  2. I did not know, that without gateway defined (on interface) pfsense will treat it as a LAN.
                  3. I did not know, that pfsense adds route to "monitor ip" on each gateway.
                  4. For some reason, I did not checked packet capture results in the first place

                  Solving this problem... was interesting. Now I know how pfsense works a little bit better :)

                  1 Reply Last reply Reply Quote 1
                  • S
                    stephenw10 Netgate Administrator
                    last edited by Apr 23, 2020, 10:44 PM

                    Nice. Yeah the automatic outbound NAT setting will NAT all traffic from internal subnets to the interface IP of any WAN type interfaces. But here it was seeing the LAN interface as a WAN so did not NAT traffic from that subnet as it left the actual WAN.

                    Steve

                    1 Reply Last reply Reply Quote 0
                    9 out of 9
                    • First post
                      9/9
                      Last post
                    Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                      This community forum collects and processes your personal information.
                      consent.not_received