Upstream gateway vs default gateway

marcus_1302 · Dec 28, 2019, 7:43 PM

Hi,

can someone explains what's exactly the difference between the upstream gateway configured on the wan interface and the default gateway also configured on wan interface at pfSense?

I know you use the upstream gateway for the ISP Router IP for the outbound traffic to the internet, but why do I need this
gateway and why does it not work when you only configure the ISP Router IP as default GW on the wan interface.

I had a strange behaviour with the following configuration:

One pfSense as border-gw with wan and dmz interface and one pfSense in the back with lan and dmz interface.
At the back pfSense on the dmz Interface I wrongly configured the upstream gateway with the ip from the front
pfSense dmz interface ip instead only as default gateway.

But never had problems and the back pfSense routes outbound traffic correctly to the internal dmz interface of the border-gw pfSense and from there over the correct upstream gateway on the wan interface.

Now I had a second front pfSense which is only responsible for an IPSec VPN tunnel to an Azure VNet. Everthing works fine when I access VMs in Azure, but the Azure VMs cannot start a connection to onPremise VMs, only ICMP works sometimes.

Routing is ok and the problem was only the wrongly set upstream gateway on the back firewall instead only a default gateway. With the upstream gateway configured on the back, pfSense routes traffic which is for the second pfSense with the IPSec tunnel to Azure over the normal border-Gw and use here the upstream gateway instead the correct static routes to the pfSense for Azure.

After I set the upstream gateway on the back pfSense to none and therefore had only a default gw to the normal border-gw everthing works fine and traffic defined in the static routes goes correctly to the Azure pfSense.

I want to understand what's here the difference and why the back pfSense ignore the static routes if an upstream gateway is configured on the dmz interface?

thanks

Derelict · Dec 29, 2019, 7:35 AM

It has to do with applying route-to and reply-to to the states created in and out of the interface.

If you have multiple upstream gateways it is generally better to not set a gateway on the interface setting itself (like the upstream gateway on the Interfaces > WAN settings, but create a gateway and static routes in System > Routing instead.

marcus_1302 · Dec 29, 2019, 2:01 PM

Thanks for the reply but I still not understand what's exactly the difference for pfSense between this upstream gateway and the default gateway on the wan interface.

The only documentation I found is in the pfSense book under https://docs.netgate.com/pfsense/en/latest/book/interfaces/ipv4-wan-types.html?highlight=upstream%20gateway

°Selecting an IPv4 Upstream Gateway from the drop-down list or adding and selecting a new gateway will make pfSense treat this interface as a WAN type interface for NAT and related functions. This is not desirable for internal interfaces such as LAN or a DMZ. Gateways may still be used on internal interfaces for the purpose of static routes without selecting an IPv4 Upstream Gateway here on the interfaces screen.°(information text)

But this explanation doesn't really help me to understand the what's going under the hood different to the default gateway.

Most or all other Routers or Firewalls I know (Fortigate, Juniper, IPCop, TMG, ... ) in a static wan ip configruration, you configure on the wan interface a default gw and period.

The most weird about this I don't get is what pfSense does on the back firewall (responsible for traffic between LAN and DMZ ) different with or without configured upstream gateway on the DMZ interface? Both the default GW and the upstream GW are the IP of the front Firwalls internal dmz interface.

When the upstream GW is assigned, traffic initiated from the LAN to the second front Firewall (other then upstream GW IP) flows correct over the static route entry to the second front firewall. But when traffic initiated from remote site which comes over the second front firewall, this traffic routes to the correct inside LAN VM, but the reply then routes not correct back over the second front firewall an the defined static route on the back firewall, instead it uses the first front firewall.

It looks like only the reply from the remote site takes the upstream GW IP instead the defined static route. But why only the reply and not initial traffic from the LAN to the remote site??? Therefore I want to understand how pfSense treat this upstream gateway in difference to a default gw each on the same interface configured.

marcus_1302 · Dec 29, 2019, 2:05 PM

@marcus_1302

btw. the ip from the remote site comes over the VPN tunnel, so there is no NAT and the reply IP is the same as the initiated IP

Derelict · Dec 29, 2019, 6:43 PM

Look at the differences in the firewall rules generated (/tmp/rules.debug) when there is or is not a gateway defined on the interface configuration itself.

It is almost never correct to put a gateway on an inside interface configuration, only put them on outside/WAN interfaces unless you understand route-to and reply-to and know you need it.

marcus_1302 · Dec 29, 2019, 8:45 PM

Thank's for the tip with /tmp/rules.debug! The reply-to as you mentioned was the problem.

Now I know why only the reply traffic to the initiated remote site traffic was routed over the upstream gateway instead the defined static route.

In case the upstream gateway was set, pfSense added to all allowed incoming traffic a userrule with:

    pass in quick on $<DMZ Interface back firewall> reply-to ( DMZ Interface front firewall IP) inet proto ......

which routes the reply for all incoming traffic back fix to the upstream gateway without regard of any static routes. This makes of course sense for the real WAN Interface and in case you had multiple WAN Gateways, so that traffic flows back on the same Interface/IP as the incoming traffic especially for TCP Connections.

Additional he adds a pass out route-to rule which allows all outbound traffic fom pfSense itself to the upstream gateway. And in case NAT is on pfSense enabled, he adds NAT rules to the WAN interface which in my case of the back firewall was not the case and NAT is there disabled.

Regarding not to set a gateway on inside interfaces of course you are right, but in case of a classic DMZ with front and back firewall, I had to set at the back firewall on the interface linked to the dmz a default gateway or static route with 0.0.0.0/0 for the internal Interface of the front firewall, otherwise traffic cannot flow to the internet.

Derelict · Dec 29, 2019, 8:46 PM

Yes, of course proper routing has to be in place on any downstream routers.

marcus_1302 · Dec 29, 2019, 11:39 PM

@marcus_1302

btw. just saw that even when you set the upstream gateway on the WAN interface, you can prevent pfSense from adding a reply-to userrule.

Under System - Advanced - Firewall & Nat there is a checkbox

Disable reply-to on WAN rules

With Multi-WAN it is generally desired to ensure traffic leaves the same interface it arrives on, hence reply-to is added automatically by default. When using bridging, this behavior must be disabled if the WAN gateway IP is different from the gateway IP of the hosts behind the bridged interface.