Port forward for SSH fails (regardless of which port used), HTTP, HTTPS works



  • I've got an issue I'm completely confused about. I'm gonna try to be as verbose as I can in hopes that someone is able to see what's wrong and help me. Sorry if it's a bit much.

    I recently setup pfsense working as a router/firewall for an esxi box, so the setup is double-NATed, but I'm not having issues there.

    I have been able to get HTTP/S working from both my home network (inside the first home router, the pfsense WAN) and the internet, along with being able to forward a handful of other ports (for plex and a torrent client).

    But for some reason SSH connections do not seem to be forwarded. I've tried the default of 22 and also tried a custom port of 2222. Any machines inside of the pfsense network are able to SSH to the target machine without issues and I am able to SSH into the pfsense router from the network behind the pfsense router, but I can't SSH from the pfsense WAN network in to the target machine.

    I just get "Resource temporarily unavailable" when I attempt to connect via SSH.

    Here is my port forwards config (both the config for port 22 and port 2222 return the same error):

    For System/Advanced/Firewall & NAT I've got these settings:

    • Disable Firewall Scrub: CHECKED

    • System Advanced Firewall & NAT: Pure NAT

    • Enable NAT Reflection for 1:1 NAT: UNCHECKED

    • Enable automatic outbound NAT for Reflection: UNCHECKED

    I tried a package capture and then tried to connect via port 22, got this output:
    (192.168.1.10 is my desktop that I'm trying to connect from, 192.168.1.30 is the pfsense WAN IP)

    16:20:06.661084 b4:b5:2f:cd:a2:c8 > 00:0c:29:35:46:a1, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 31816, offset 0, flags [DF], proto TCP (6), length 52)
        192.168.1.10.58511 > 192.168.1.30.22: Flags [s], cksum 0x94e6 (correct), seq 204555738, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
    16:20:09.662136 b4:b5:2f:cd:a2:c8 > 00:0c:29:35:46:a1, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 31833, offset 0, flags [DF], proto TCP (6), length 52)
        192.168.1.10.58511 > 192.168.1.30.22: Flags [s], cksum 0x94e6 (correct), seq 204555738, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
    16:20:15.663584 b4:b5:2f:cd:a2:c8 > 00:0c:29:35:46:a1, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 128, id 32015, offset 0, flags [DF], proto TCP (6), length 48)
        192.168.1.10.58511 > 192.168.1.30.22: Flags [s], cksum 0xa8f5 (correct), seq 204555738, win 8192, options [mss 1460,nop,nop,sackOK], length 0
    
    If I go to Diagnostics/States/States and I try to filter by the IP that I'm trying to forward port 22 to (192.168.2.21), no states match that filter. If I try to filter by port 22 (:22) I still get no matching states.
    
    I have logging enabled for the firewall and it does appear to pickup the connection (I highlighted the relevant IP):
    
    [img]http://i.imgur.com/IUBzEeG.png[/img]
    
    Please help? I'm at the end of my rope. I'm so desperate I'll paypal the one person that helps me figure this out some beer money.
    
    Let me know if there's any other info or testing I ought to get/do.[/s][/s][/s]
    

  • Netgate

    It sounds like you have been pretty clicky clicky like in disabling scrub. Hard to know if there's something you clicked somewhere that broke this.

    Are you sure these connections aren't being blocked by a local "software" firewall on 192.168.2.21?

    ssh is TCP. Not TCP/UDP, for the record.



  • I disabled scrub as I am planning on it serving NFS to the outside network. I wouldn't assume it would incur an issue here, but figured I'd mention it. It's hard for me to say for sure that I've looked over every setting that might be relevant, but I've gone through all the configuration pages I've entered during my initial setup and looked over all the settings in those pages and noted what I think might have any relevance.

    I mentioned above that I am able to ssh into 192.168.2.21 (and all the other various machines within the pfsense network) from other machines with-in that network. I have not done any custom configuration for software firewall on any of the machines themselves. The target systems are Ubuntu 16.10 (you'll note that the attempt with port 2222 is to a different machine, figured I'd try that just incase it was a machine-specific scenario). 192.168.2.21 is configured to host gitlab using their provided installer, but 192.168.2.5 is a very plain setup, intended to be from where I'd manage all the other machines from and other than installing some packages I've done no customization to it yet.

    And yeah, SSH is TCP. I forgot to set it back after much of my own desperate testing (I've been grasping at straws, figured it couldn't hurt).


  • Netgate

    Local host firewalls often allow connections from the local subnet while denying from outside that subnet which is why I mention it. Very typical of windows. I don't know all the behaviors of the linux distributions.

    Packet capture on the interface with the destination host on it. If you see the SYNs being sent and nothing in reply, it's not pfSense. It's the destination host or something else on that network.

    I disabled scrub as I am planning on it serving NFS to the outside network.

    I wouldn't change a default like that unless you know you need it to solve a problem you are actually experiencing. Knowing that knob is there should be enough for you to try it to solve a specific issue.



  • @Derelict:

    Local host firewalls often allow connections from the local subnet while denying from outside that subnet which is why I mention it. Very typical of windows. I don't know all the behaviors of the linux distributions.

    Understandable.

    It's never been an issue in previous cases where pfsense wasn't involved, but in this case the firewall is disabled. This does give me the idea to expose one of the VMs to the outer network and try to SSH into that machine from work, just another point to test. Something I'll try tomorrow.

    @Derelict:

    Packet capture on the interface with the destination host on it. If you see the SYNs being sent and nothing in reply, it's not pfSense. It's the destination host or something else on that network.

    With the interface set to LAN, address family IPv4 only, any protocol, host 192.168.2.21, and a normal level of detail, this is what I get:

    17:24:18.848776 IP 192.168.1.10.60429 > 192.168.2.21.22: tcp 0
    17:24:21.848919 IP 192.168.1.10.60429 > 192.168.2.21.22: tcp 0
    17:24:27.849269 IP 192.168.1.10.60429 > 192.168.2.21.22: tcp 0
    
    

    Fully admit, a newb here, what is this indicating? What does the tcp 0 indicate?

    @Derelict:

    I wouldn't change a default like that unless you know you need it to solve a problem you are actually experiencing. Knowing that knob is there should be enough for you to try it to solve a specific issue.

    More than fair enough… I can be a bit pre-emptive about things. :/

    I did try doing a new install of pfsense and just connecting one of the machines to it and testing the bare-minimum, but I had issues with getting even HTTP forwarded on that and kinda abandoned that attempt because I couldn't figure out what I had wrong from the new one to the old one... But maybe tomorrow I'll try a more fresh setup and spin up a whole new VM just to test with instead of just connecting one of my existing VMs to the new network (not that I can see how that would make a difference... but hey, what ever I can try I'm gonna go for).

    Thanks for the input so far, and I'll be back in the morning to dig deeper.


  • Rebel Alliance Global Moderator

    17:24:18.848776 IP 192.168.1.10.60429 > 192.168.2.21.22: tcp 0

    This packet being sent to 192.168.2.21 ssh, and not seeing any answer..

    You sure ssh is even listening??  And listening on 22..  Clearly Pfsense sent on your traffic, but got no response..  You need to look to your client to the problem.  Or somewhere else down your network on why that traffic didn't get there.  You sure your sending the traffic to the right IP that is listening on ssh??

    This really is click click.. Here I forwarded ssh to my linux box..

    I then went to canyouseeme and did a test.  While I was sniffing on the pfsense lan where my linux box is..

    You can see canyouseeme says yeah your good.  When I sniff on pfsense lan I see the traffic being sent, and the answer.  If I open that up in wireshark you can see the syn and the syn,ack sent back.

    You are not seeing the syn,ack come back.  So either that box is not listening, that box has a firewall that is dropping it, or somewhere in your network that packet never got to the box.  But clearly pfsense sent it on.



  • Netgate

    Check (really check) everything on the list here:

    https://doc.pfsense.org/index.php/Port_Forward_Troubleshooting



  • Welp, you were right, it was completely client-side. It was really stupid (and I feel just as that). Ended up being a gateway issue on those two specific machines. I really should have tested other systems earlier on. :/

    And so, while your input was probably pretty basic and standard for you, none the less thanks Derelict for getting me on the right path to figure this out. I'm pretty fresh at a more complex network setup and my intention is to use this to learn about more complex networking setups, and this has been a good thing to start my teething on. And I don't back out of promises. PM me an email to paypal you $5 and I'll do that. Treat yourself to a beer or some other thing that you love.


  • Netgate

    Glad you got it working.

    If you really want to part with $5, please send it here: https://www.freebsdfoundation.org/donate/