Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Upstream gateway vs default gateway

    Scheduled Pinned Locked Moved Routing and Multi WAN
    8 Posts 2 Posters 12.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      marcus_1302
      last edited by

      Hi,

      can someone explains what's exactly the difference between the upstream gateway configured on the wan interface and the default gateway also configured on wan interface at pfSense?

      I know you use the upstream gateway for the ISP Router IP for the outbound traffic to the internet, but why do I need this
      gateway and why does it not work when you only configure the ISP Router IP as default GW on the wan interface.

      I had a strange behaviour with the following configuration:

      One pfSense as border-gw with wan and dmz interface and one pfSense in the back with lan and dmz interface.
      At the back pfSense on the dmz Interface I wrongly configured the upstream gateway with the ip from the front
      pfSense dmz interface ip instead only as default gateway.

      But never had problems and the back pfSense routes outbound traffic correctly to the internal dmz interface of the border-gw pfSense and from there over the correct upstream gateway on the wan interface.

      Now I had a second front pfSense which is only responsible for an IPSec VPN tunnel to an Azure VNet. Everthing works fine when I access VMs in Azure, but the Azure VMs cannot start a connection to onPremise VMs, only ICMP works sometimes.

      Routing is ok and the problem was only the wrongly set upstream gateway on the back firewall instead only a default gateway. With the upstream gateway configured on the back, pfSense routes traffic which is for the second pfSense with the IPSec tunnel to Azure over the normal border-Gw and use here the upstream gateway instead the correct static routes to the pfSense for Azure.

      After I set the upstream gateway on the back pfSense to none and therefore had only a default gw to the normal border-gw everthing works fine and traffic defined in the static routes goes correctly to the Azure pfSense.

      I want to understand what's here the difference and why the back pfSense ignore the static routes if an upstream gateway is configured on the dmz interface?

      thanks

      1 Reply Last reply Reply Quote 0
      • DerelictD
        Derelict LAYER 8 Netgate
        last edited by

        It has to do with applying route-to and reply-to to the states created in and out of the interface.

        If you have multiple upstream gateways it is generally better to not set a gateway on the interface setting itself (like the upstream gateway on the Interfaces > WAN settings, but create a gateway and static routes in System > Routing instead.

        Chattanooga, Tennessee, USA
        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
        Do Not Chat For Help! NO_WAN_EGRESS(TM)

        1 Reply Last reply Reply Quote 0
        • M
          marcus_1302
          last edited by

          Thanks for the reply but I still not understand what's exactly the difference for pfSense between this upstream gateway and the default gateway on the wan interface.

          The only documentation I found is in the pfSense book under https://docs.netgate.com/pfsense/en/latest/book/interfaces/ipv4-wan-types.html?highlight=upstream%20gateway

          °Selecting an IPv4 Upstream Gateway from the drop-down list or adding and selecting a new gateway will make pfSense® treat this interface as a WAN type interface for NAT and related functions. This is not desirable for internal interfaces such as LAN or a DMZ. Gateways may still be used on internal interfaces for the purpose of static routes without selecting an IPv4 Upstream Gateway here on the interfaces screen.°(information text)

          But this explanation doesn't really help me to understand the what's going under the hood different to the default gateway.

          Most or all other Routers or Firewalls I know (Fortigate, Juniper, IPCop, TMG, ... ) in a static wan ip configruration, you configure on the wan interface a default gw and period.

          The most weird about this I don't get is what pfSense does on the back firewall (responsible for traffic between LAN and DMZ ) different with or without configured upstream gateway on the DMZ interface? Both the default GW and the upstream GW are the IP of the front Firwalls internal dmz interface.

          When the upstream GW is assigned, traffic initiated from the LAN to the second front Firewall (other then upstream GW IP) flows correct over the static route entry to the second front firewall. But when traffic initiated from remote site which comes over the second front firewall, this traffic routes to the correct inside LAN VM, but the reply then routes not correct back over the second front firewall an the defined static route on the back firewall, instead it uses the first front firewall.

          It looks like only the reply from the remote site takes the upstream GW IP instead the defined static route. But why only the reply and not initial traffic from the LAN to the remote site??? Therefore I want to understand how pfSense treat this upstream gateway in difference to a default gw each on the same interface configured.

          M 1 Reply Last reply Reply Quote 0
          • M
            marcus_1302 @marcus_1302
            last edited by

            @marcus_1302

            btw. the ip from the remote site comes over the VPN tunnel, so there is no NAT and the reply IP is the same as the initiated IP

            1 Reply Last reply Reply Quote 0
            • DerelictD
              Derelict LAYER 8 Netgate
              last edited by

              Look at the differences in the firewall rules generated (/tmp/rules.debug) when there is or is not a gateway defined on the interface configuration itself.

              It is almost never correct to put a gateway on an inside interface configuration, only put them on outside/WAN interfaces unless you understand route-to and reply-to and know you need it.

              Chattanooga, Tennessee, USA
              A comprehensive network diagram is worth 10,000 words and 15 conference calls.
              DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
              Do Not Chat For Help! NO_WAN_EGRESS(TM)

              1 Reply Last reply Reply Quote 1
              • M
                marcus_1302
                last edited by

                Thank's for the tip with /tmp/rules.debug! The reply-to as you mentioned was the problem.

                Now I know why only the reply traffic to the initiated remote site traffic was routed over the upstream gateway instead the defined static route.

                In case the upstream gateway was set, pfSense added to all allowed incoming traffic a userrule with:

                    pass in quick on $<DMZ Interface back firewall> reply-to ( DMZ Interface front firewall IP) inet proto ......
                

                which routes the reply for all incoming traffic back fix to the upstream gateway without regard of any static routes. This makes of course sense for the real WAN Interface and in case you had multiple WAN Gateways, so that traffic flows back on the same Interface/IP as the incoming traffic especially for TCP Connections.

                Additional he adds a pass out route-to rule which allows all outbound traffic fom pfSense itself to the upstream gateway. And in case NAT is on pfSense enabled, he adds NAT rules to the WAN interface which in my case of the back firewall was not the case and NAT is there disabled.

                Regarding not to set a gateway on inside interfaces of course you are right, but in case of a classic DMZ with front and back firewall, I had to set at the back firewall on the interface linked to the dmz a default gateway or static route with 0.0.0.0/0 for the internal Interface of the front firewall, otherwise traffic cannot flow to the internet.

                M 1 Reply Last reply Reply Quote 1
                • DerelictD
                  Derelict LAYER 8 Netgate
                  last edited by

                  Yes, of course proper routing has to be in place on any downstream routers.

                  Chattanooga, Tennessee, USA
                  A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                  DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                  Do Not Chat For Help! NO_WAN_EGRESS(TM)

                  1 Reply Last reply Reply Quote 0
                  • M
                    marcus_1302 @marcus_1302
                    last edited by

                    @marcus_1302

                    btw. just saw that even when you set the upstream gateway on the WAN interface, you can prevent pfSense from adding a reply-to userrule.

                    Under System - Advanced - Firewall & Nat there is a checkbox

                    Disable reply-to on WAN rules

                    With Multi-WAN it is generally desired to ensure traffic leaves the same interface it arrives on, hence reply-to is added automatically by default. When using bridging, this behavior must be disabled if the WAN gateway IP is different from the gateway IP of the hosts behind the bridged interface.

                    1 Reply Last reply Reply Quote 0
                    • First post
                      Last post
                    Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.