Pfsense missing return packets during NAT

[PetersonG17]

Hey All,

I am having an issue with my Pfsense router. I have a NAT rule to allow SFTP (Post 22) communication from an external IP to an internal private IP for our File Server. The NAT does have the associated firewall rule in place. The issue is this: Packets will come from the external IP successfully through the firewall and NAT rule to the internal File Server. The File Servers Syn_ACK response never makes it back out of the Pfsense box though. It gets through the internal interface facing the File Server but seems to disappear before hittin g any firewall rules. Attached are some screenshots.

I do have multiple WAN connections and firewall rules set to adjust traffic leaving the Pfsense LAN Interface. At first I thought that this might be the issue and so added a rule to make sure that anything leaving headed to port 22 would go out the same gateway that it came in from.

Additional info:
External testing IP: 174.208.9.223 (Source IP) (This is a member of the SFTP_IP Alias)
WAN IP for PFsense box: 199.244.15.85 (REVNET_SERVER)
LAN IP for PFsense Box: 10.1.13.1
Private File Server IP: 10.1.13.10

Also see these screenshots of my firewall and NAT rules:

WAN Interface Rules:

NAT Rules:

LAN Interface Rules:

WAN Interface Packet Capture:

File Server Packet Capture:

LAN Interface Packet Capture:

As I stated before, I have checked the firewall logs for all of the LAN Interface rules and cannot find where the packets are being dropped. They just don't show up.

My mind is really boggled on this one. Any thoughts on what is going on?

Thanks,

[Grimson]

LAN rules have no effect on connections from the WAN interface, these are initially managed by the WAN rules and the active state after that.

[PetersonG17]

Thanks for that input.

So, just to clarify, my LAN rule does not effect traffic which is originating from the WAN side because of the state. My NAT rule/firewall rule on the WAN side should be forcing response packets coming from the LAN side (Which originated from the WAN connection) back out of the correct gateway. I just wanted to be sure to avoid responding through a different gateway than where the connection originated, but it sounds like the state tracking takes care of that already.

Still confused at why the response packets from the LAN are not making it back to the original External IP. If not a firewall rule, then perhaps another hidden setting in Pfsense that would need to be adjusted?

[PetersonG17]

Wanted to post some additional info. I decided to do some checking of firewall logs (Now that I know Connections initiated on the WAN are checked by the WAN firewall rules) and this is what I found:

Turned on logging for default block rule:

When I tried to connect to SFTP server from my external IP, this happened. I can see the pass/NAT rule in effect on port 22 but right after there is another packet blocked by the default rule stemming from my external IP heading to the same external WAN IP on the Pfsense but not translated and on port 21.

No idea if this is related, but hoping that all of my digging will lead to a solution.

[heper]

ftp(21) is a different can of worms

[PetersonG17]

Another bit of info that I just thought of. Since making these NAT and firewall changes, I have not reset the state table or rebooted the Pfsense box. Would either of those be required to make sure that the new NAT and associated Firewall Rules are functioning correctly?

Also, when checking the firewall rules, I can't see any current states with the ones in question.

Is there a way to reset the states only for specific rules?

[Derelict]

Your NAT port forward needs to be on WAN, not REVNET_SERVER.

If the screen shots above are not what you currently have, please post new ones for WAN, LAN, and the the port forward.

Whatever is responding as 10.1.13.1 is saying it has no route for 174.208.9.223.

Are you sure you don't have something else on the network on 10.1.13.1?

You probably want to post the output of this executed in Diagnostics > Command Prompt:

netstat -rnfinet

[PetersonG17]

Thanks for the reply. The screenshots in the original post are current.

REVNET_SERVER is a WAN Interface, as are:

COMCAST_VOICE
COMCAST_DATA
CENTURYLINK_FAILOVER

My LAN Interfaces are:

DATA
MGMT
SERVERS
PHONES
FINANCE
GUESTWIFI

If I look in the Interfaces >> Assignments, there is no WAN option.

Interfaces:

To my knowledge I have nothing else on the 10.1.13.1 network but that one server at 10.1.13.10.

Here is the output of the command you listed:

Under the NAT rule there is no option for "WAN". But as I said previously the REVNET_SERVER is a WAN interface. It uses DHCP to receive its Reserved IP from my ISP.

I apologize if I am still misunderstanding. Thank you so much for helping me figure out this issue.

I have never used the command you listed "netstat -rnfinet" and am not familiar with the output. Is it revealing?

[Derelict]

You have no default gateway so there is no route in the firewall for traffic to 174.208.9.223.

[Derelict]

Do you have a gateway defined on the REVNET_SERVER interface configuration?

[PetersonG17]

Now that you mention it, I did just remove another line of internet around the same time this started happening. It was probably the default gateway.

Does it matter which of my Gateways I mark as default?

[Derelict]

You mark the one you want to use as the default. It will be the route for traffic that is not policy routed.

Derelict Netgate 2 minutes ago

Do you have a gateway defined on the REVNET_SERVER interface configuration?

This question matters.

[PetersonG17]

Yes, I do have a gateway defined on that interface (It is not the default though)

If I define a different gateway as the default gateway, will it mess with the routing of these packets on the return trip?

The way I understand it, the packets should take this path:
External IP >> REVNET_SERVER Interface >> SERVERS Interface >> SFTP Server >> SERVERS Interface >> REVNET_SERVER Interface >> External IP

So the packets should return out of the same gateway that they came in on, correct?

[Derelict]

No. A gateway on the interface configuration itself, under Interfaces > REVNET_SERVER.

OK. It's DHCP. It should have a gateway there.

Yes, the traffic should go out the interface it came into. But it needs a route or FreeBSD will not accept the traffic so pf can do its reply-to thing. Which gateway is selected as the default should not matter with the exception of traffic generated on the firewall itself. That will use whatever is set as the default.

[PetersonG17]

I just made the COMCAST_DATA interface the default gateway, and things have changed a little.

Now the WinSCP client I am using to test from an external IP is getting the "Network Error: Connection to 199.244.15.85 is refused" error. I have never gotten this before, so I assume this is a step in the right direction.

The Pfsense Packet Captures look the same as the screenshots I put in the original post, except for the Wireshark capture on the server which no longer has a "Host unreachable" error.

@Derelict When you say that "it needs a route" are you talking about static routes? Or shouldn't my firewall and NAT policy based routing be enough?

[Derelict]

10.1.13.10 is refusing that connection.

[PetersonG17]

Got it! I had typed the password incorrectly for the user to the SFTP server. Now the connection is working perfectly!

So to sum up, sounds like the issue was that I did not have a default gateway selected. Once I selected a gateway to be the default gateway (and typed in the password correctly for the SFTP Server) then everything functioned.

Can anyone explain why a default gateway selection is required for NAT to work? (Just curious) I had thought that through my firewall rules and policy based routing that I could direct the packets just fine....

Thanks all for your help!

[Derelict]

When the reply packet was received by the firewall it had no route in the routing table for the destination so it returned Destination Unreachable.