Pfsense missing return packets during NAT



  • Hey All,

    I am having an issue with my Pfsense router. I have a NAT rule to allow SFTP (Post 22) communication from an external IP to an internal private IP for our File Server. The NAT does have the associated firewall rule in place. The issue is this: Packets will come from the external IP successfully through the firewall and NAT rule to the internal File Server. The File Servers Syn_ACK response never makes it back out of the Pfsense box though. It gets through the internal interface facing the File Server but seems to disappear before hittin g any firewall rules. Attached are some screenshots.

    I do have multiple WAN connections and firewall rules set to adjust traffic leaving the Pfsense LAN Interface. At first I thought that this might be the issue and so added a rule to make sure that anything leaving headed to port 22 would go out the same gateway that it came in from.

    Additional info:
    External testing IP: 174.208.9.223 (Source IP) (This is a member of the SFTP_IP Alias)
    WAN IP for PFsense box: 199.244.15.85 (REVNET_SERVER)
    LAN IP for PFsense Box: 10.1.13.1
    Private File Server IP: 10.1.13.10

    Also see these screenshots of my firewall and NAT rules:

    WAN Interface Rules:
    0_1533912988901_2018-08-10 08_53_38-Window.png

    NAT Rules:
    0_1533913017275_2018-08-10 08_54_10-Window.png

    LAN Interface Rules:
    0_1533913038293_2018-08-10 08_54_32-Window.png

    WAN Interface Packet Capture:
    0_1533913164326_2018-08-09 07_57_30-Window.png

    File Server Packet Capture:
    0_1533913122742_2018-08-09 07_55_44-Capturing from Ethernet 2.png

    LAN Interface Packet Capture:
    0_1533913155162_2018-08-09 07_55_20-Window.png

    As I stated before, I have checked the firewall logs for all of the LAN Interface rules and cannot find where the packets are being dropped. They just don't show up.

    My mind is really boggled on this one. Any thoughts on what is going on?

    Thanks,



  • LAN rules have no effect on connections from the WAN interface, these are initially managed by the WAN rules and the active state after that.



  • Thanks for that input.

    So, just to clarify, my LAN rule does not effect traffic which is originating from the WAN side because of the state. My NAT rule/firewall rule on the WAN side should be forcing response packets coming from the LAN side (Which originated from the WAN connection) back out of the correct gateway. I just wanted to be sure to avoid responding through a different gateway than where the connection originated, but it sounds like the state tracking takes care of that already.

    Still confused at why the response packets from the LAN are not making it back to the original External IP. If not a firewall rule, then perhaps another hidden setting in Pfsense that would need to be adjusted?



  • Wanted to post some additional info. I decided to do some checking of firewall logs (Now that I know Connections initiated on the WAN are checked by the WAN firewall rules) and this is what I found:

    Turned on logging for default block rule:
    0_1533936335714_2018-08-10 15_23_19-Window.png

    When I tried to connect to SFTP server from my external IP, this happened. I can see the pass/NAT rule in effect on port 22 but right after there is another packet blocked by the default rule stemming from my external IP heading to the same external WAN IP on the Pfsense but not translated and on port 21.
    0_1533936481403_2018-08-10 15_22_35-Window.png

    No idea if this is related, but hoping that all of my digging will lead to a solution.



  • ftp(21) is a different can of worms



  • Another bit of info that I just thought of. Since making these NAT and firewall changes, I have not reset the state table or rebooted the Pfsense box. Would either of those be required to make sure that the new NAT and associated Firewall Rules are functioning correctly?

    Also, when checking the firewall rules, I can't see any current states with the ones in question.

    Is there a way to reset the states only for specific rules?


  • Netgate

    Your NAT port forward needs to be on WAN, not REVNET_SERVER.

    If the screen shots above are not what you currently have, please post new ones for WAN, LAN, and the the port forward.

    Whatever is responding as 10.1.13.1 is saying it has no route for 174.208.9.223.

    Are you sure you don't have something else on the network on 10.1.13.1?

    You probably want to post the output of this executed in Diagnostics > Command Prompt:

    netstat -rnfinet



  • Thanks for the reply. The screenshots in the original post are current.

    REVNET_SERVER is a WAN Interface, as are:

    COMCAST_VOICE
    COMCAST_DATA
    CENTURYLINK_FAILOVER

    My LAN Interfaces are:

    DATA
    MGMT
    SERVERS
    PHONES
    FINANCE
    GUESTWIFI

    If I look in the Interfaces >> Assignments, there is no WAN option.

    Interfaces:
    0_1534183087640_2018-08-13 11_54_58-router.foxpestcontrol.local - Interfaces_ Interface Assignments.png

    To my knowledge I have nothing else on the 10.1.13.1 network but that one server at 10.1.13.10.

    Here is the output of the command you listed:
    0_1534183259808_2018-08-13 11_54_29-router.foxpestcontrol.local - Diagnostics_ Command Prompt.png

    Under the NAT rule there is no option for "WAN". But as I said previously the REVNET_SERVER is a WAN interface. It uses DHCP to receive its Reserved IP from my ISP.
    0_1534183386412_2018-08-13 11_55_29-router.foxpestcontrol.local - Firewall_ NAT_ Port Forward_ Edit.png

    I apologize if I am still misunderstanding. Thank you so much for helping me figure out this issue.

    I have never used the command you listed "netstat -rnfinet" and am not familiar with the output. Is it revealing?


  • Netgate

    You have no default gateway so there is no route in the firewall for traffic to 174.208.9.223.


  • Netgate

    Do you have a gateway defined on the REVNET_SERVER interface configuration?



  • Now that you mention it, I did just remove another line of internet around the same time this started happening. It was probably the default gateway.

    Does it matter which of my Gateways I mark as default?


  • Netgate

    You mark the one you want to use as the default. It will be the route for traffic that is not policy routed.

    Derelict Netgate 2 minutes ago

    Do you have a gateway defined on the REVNET_SERVER interface configuration?

    This question matters.



  • Yes, I do have a gateway defined on that interface (It is not the default though)

    0_1534184180215_2018-08-13 12_15_27-router.foxpestcontrol.local - System_ Routing_ Gateways.png

    0_1534184185261_2018-08-13 12_15_44-router.foxpestcontrol.local - System_ Routing_ Gateways_ Edit.png

    If I define a different gateway as the default gateway, will it mess with the routing of these packets on the return trip?

    The way I understand it, the packets should take this path:
    External IP >> REVNET_SERVER Interface >> SERVERS Interface >> SFTP Server >> SERVERS Interface >> REVNET_SERVER Interface >> External IP

    So the packets should return out of the same gateway that they came in on, correct?


  • Netgate

    No. A gateway on the interface configuration itself, under Interfaces > REVNET_SERVER.

    OK. It's DHCP. It should have a gateway there.

    Yes, the traffic should go out the interface it came into. But it needs a route or FreeBSD will not accept the traffic so pf can do its reply-to thing. Which gateway is selected as the default should not matter with the exception of traffic generated on the firewall itself. That will use whatever is set as the default.



  • I just made the COMCAST_DATA interface the default gateway, and things have changed a little.

    Now the WinSCP client I am using to test from an external IP is getting the "Network Error: Connection to 199.244.15.85 is refused" error. I have never gotten this before, so I assume this is a step in the right direction.

    The Pfsense Packet Captures look the same as the screenshots I put in the original post, except for the Wireshark capture on the server which no longer has a "Host unreachable" error.
    0_1534186741516_2018-08-13 12_56_11-_Ethernet 2.png

    @Derelict When you say that "it needs a route" are you talking about static routes? Or shouldn't my firewall and NAT policy based routing be enough?


  • Netgate

    10.1.13.10 is refusing that connection.



  • Got it! I had typed the password incorrectly for the user to the SFTP server. Now the connection is working perfectly!

    So to sum up, sounds like the issue was that I did not have a default gateway selected. Once I selected a gateway to be the default gateway (and typed in the password correctly for the SFTP Server) then everything functioned.

    Can anyone explain why a default gateway selection is required for NAT to work? (Just curious) I had thought that through my firewall rules and policy based routing that I could direct the packets just fine....

    Thanks all for your help!


  • Netgate

    When the reply packet was received by the firewall it had no route in the routing table for the destination so it returned Destination Unreachable.