Outbound NAT on Multi-WAN system

jc2it

Recently we upgraded our multi-wan system from an older unsupported version to 22.05 release.

We have a server we would like to use to send files to a vendor and only allow that vendor's public IP to be the recipient. The vendor supports ftps. When connecting to the vendor the connect times out before a welcome message is received. The vendor is blocking the communication because it is coming from the wrong WAN.

This is a multi-WAN system with the default gateway on the W-mpls WAN port.
I have set up a LAN rule to allow the traffic out to the vendor specifying only the gateway (C-ens on opt1).
I have outbound NAT setup to redirect all outbound traffic as the Public IP on the C-ens WAN connection.
The vendor only sees packets coming from the default gateway IP address on W-mpls.
When I turn on logging in the LAN rule and put it near the top of the LAN list nothing is logged.
What troubleshooting steps should I undertake to:

Find out/track the outbound packets.
Ensure the Outbound NAT rules are functioning correctly.
Provide information which shows the setup to be correct or not.

Thanks!

viragomann

@jc2it
So obviously the policy routing rule is not applied. You can verify by doing a packet on the default WAN.

Consider to flush states after changing the rule set.

Ensure that there is no other matching rule allowing this outbound traffic above of the policy routing rule. Also check rules on interface groups where the LAN is member of and floating rules.

jc2it

@viragomann said in Outbound NAT on Multi-WAN system:

@jc2it
So obviously the policy routing rule is not applied. You can verify by doing a packet on the default WAN.

I do not understand how to "verify by doing a packet on the default WAN."

Consider to flush states after changing the rule set.

This is a good idea and I will try this. I have rebooted since making changes and it didn't fix it.

Ensure that there is no other matching rule allowing this outbound traffic above of the policy routing rule. Also check rules on interface groups where the LAN is member of and floating rules.

There are no matching rules above it. All rules are set to log. The data doesn't appear in the "Status/System Log/Firewall" log. The data goes out the default gateway instead of following the outbound NAT and the policy rule.

When I look at the log file in the gui it is difficult to search historically. How would I investigate this from the command line?

I will check the rules on interface groups but I don't believe there are any. Unless there are some automatically defined by a package.

There are no floating rules.

Thanks!

viragomann

@jc2it said in Outbound NAT on Multi-WAN system:

I do not understand how to "verify by doing a packet on the default WAN."

Should mean "by doing a packet capture". Diagnostic > Packet Capture or even run a tcpdump on WAN.

Can you post your rules?

There are no matching rules above it. All rules are set to log. The data doesn't appear in the "Status/System Log/Firewall" log. The data goes out the default gateway instead of following the outbound NAT and the policy rule.

You can go through all your rules and enable logging. Then you should see the packets in the log.

When I look at the log file in the gui it is difficult to search historically. How would I investigate this from the command line?

Don't you have access to the web GUI?

If you ssh into pfSense you can write the log file to your local machine:
cat /var/log/filter.log > destination.file

jc2it

@viragomann
Posting rules, even outbound rules, isn't allowed by security policy.

I ran a session and went to the command shell and ran

cat filter.log | grep xxx.xxx.xxx.xxx

where xxx.xxx.xxx.xxx is the Public IP address of the FTP server I am connecting to.

there were no results.

So now try again with a packet capture.

jc2it

@jc2it
Packet capture shows the packets but doesn't show a "Path" through the firewall. There is nothing in the packet capture that points to the originating interface (LAN) and the outbound interface in a multi-WAN install.

So now I see a disconnect between my logging and the packet capture but I am unsure how to rectify it.

I have been through the rules with a fine tooth comb.
Reorganized the rules to put the rule near the top.
Turned logging on all the rules and no logs were generated even though pfSense passed the traffic.
NOTE: My last LAN rule denies all remaining traffic and at least that should show a block but it is like the logging or pf just passes it. And that is not desirable.

jc2it

@jc2it @jimp
By playing with the packet capture some more I found I can find settings that capture the packets to the public IP destination and then retry the packet capture for each WAN.
I expect my LAN rule to send all traffic from LAN IP Address x.x.x.x port ANY to Destination ANY port 21 through gateway C-ens.

According to the packet captures I am seeing that traffic pass through gateway W-mpls.

I use aliases in the rule and I double-checked each alias to ensure I wasn't missing some detail. It is like the packets skip right over the rules.

jc2it

@viragomann @jimp

I modified the LAN rule to use aliases that were not subject to any security settings but passed traffic to the correct gateway. Then I copied the LAN rule, made it a block rule and changed the gateway to the gateway we don't want that traffic to exit on.
RESULT: Traffic still passes to the wrong gateway.

Then I switched the order of the rules. Traffic was unchanged. The packet captures still show the traffic flowing from LAN to W-mpls instead of being blocked or flowing to C-ens.

Nothing is logged for these connections. I think I found a bug.