Cannot ssh from firewall to LAN

[stephenw10]

Ok,
Yes so you have a port forward configured in pfSense to allow SSH from your rempote public IP to the internal IP of the server.

It fails when you nmap (or port test) from pfSense itself because that would be sourced from the pfSense LAN IP and the rule on the server only allows traffic from the remote public IP.

The port forward probably fails because the server has a public IP and hence likely a default route via the other NIC. Hence it replies out on the wrong interface creating an asymmetric route.

Correct the routing on the server or add an outbound NAT rule on the pfSense LAN and change the rule on the server to allow that as source instead.

Steve

[lewis]

Thank you, at least you confirm there is nothing wrong on the firewall side. I do believe there is a route missing on the openvpn server that as you say, is causing outgoing lan traffic to come back via the public IP but that's what I've been struggling to fix and is how I got to this question.

I'll have to go ask somewhere else for that since it's not pfsense related.

Thank you.

[stephenw10]

If it's only one IP you need access from via the internal interface you could just add that route to the server.

A tight outbound NAT rule on the pfSense LAN interface would allow more flexibility if you need to add more external IPs to that later. It also wouldn't break OpenVPN access from those IPs assuming that would connect to the server public IP directly.

Or you could do as I originally thought you were and ssh into the pfSense box and then ssh fro there to the server when you need to connect to it. The rule on the server would have to allow that of course.

Steve

[lewis]

Yes, I could ssh into the pfsense box and go from there but that's an extra step. Only this server is having this problem. I've got other servers that have dual NICs and those don't have the issue.

I think what I need to find is a way to add a default route for that interface on that server so that traffic going in and out the LAN IP can get routed to the firewall.

However, that's what I've been trying to do and so far, have yet to figure out how to make it work.

The public NIC/IP should be the default for the default but I've not been able to find a way to give the LAN IP a default route without messing up the public side.

Not sure this makes sense now.

[stephenw10]

It would not be a route to the firewall but back to the public IP you're connecting from. That's the problem here, the IP is not local so the server uses it's default route.

You can't set two default routes like that (not in this context anyway) you would need to just add a static route for that one IP.
Adding an outbound NAT rule in pfSense would normally be considered ugly but in this case it makes sense as it doesn't affect other connections from that IP. It's probably what I would do.

Steve

[lewis]

Yes, I didn't mean to say default route. I mean to say give the LAN side IP a route back to the firewall so that its traffic flow to/from there and not through the default route.

I've tried and tried but for some reason, nothing works.
I'll look at adding an outbound and see if that works.

[lewis]

[root@# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 x.x.x.13 0.0.0.0 UG 0 0 0 ens33
10.0.0.0 10.0.0.1 255.255.255.0 UG 0 0 0 ens34
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 ens34
172.16.1.0 0.0.0.0 255.255.255.0 U 0 0 0 tun0
x.x.x.12 0.0.0.0 255.255.255.248 U 0 0 0 ens33

The above was after I added a gateway for the LAN side but still cannot get anything back.

ping -I ens34 4.2.2.2

Nothing, no reply. Now it's back to the following.

[root@# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 x.x.x.13 0.0.0.0 UG 0 0 0 ens33
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 ens34
172.16.1.0 0.0.0.0 255.255.255.0 U 0 0 0 tun0
x.x.x.12 0.0.0.0 255.255.255.248 U 0 0 0 ens33

I must be missing something that is right in my face but am not seeing it.

[stephenw10]

Grab some screenshots for review if that still doesn't work.

[lewis]

I could do that if I can edit out the Ips since they are not mine to show/share. What should I show? Or do you want me to run a few commands and share the outputs?

[stephenw10]

Well we need to see what your WAN firewall rules are, what you port forwrad rules are and what your outbound NAT rule is.

The above route on the server would have done nothing because the incoming traffic comes from a public IP not inside the 10.0.0.0/24 subnet. The outbound NAT rule should change that.
It also does nothing because the server already has a route to 10.0.0.0/24 as it has an interface in that subnet.
The route on the server would need to be:
<the public IP you are connecting from> 10.0.0.1 255.255.255.0 UG 0 0 0 ens34
If you choose to do it that way.
I would use outbound NAT in pfSense for this for the reasons I outlined.

Steve

[lewis]

Here is what the server looks like. Public IP obfuscated. Is this what you wanted or something else, more? I've not had to set any outbound rules on pfsense to get out from other servers where they have a dual NIC also. I checked over and over again to compare and cannot find any reason for this not working.

# iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A POSTROUTING -j MASQUERADE

# iptables -S
-P INPUT DROP
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -p udp -m udp --dport 1194 -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p udp -m udp --dport 7777 -j ACCEPT
-A INPUT -s x.x.x.x/32 -i ens33 -p tcp -m tcp --dport 22 -j ACCEPT
-A FORWARD -i tun+ -o ens160 -j ACCEPT
-A FORWARD -i ens160 -o tun+ -j ACCEPT
-A FORWARD -i tun+ -o ens192 -j ACCEPT
-A FORWARD -i ens192 -o tun+ -j ACCEPT

# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet x.x.x.78  netmask 255.255.255.248  broadcast x.x.x.79
        inet6 fe80::20c:29ff:febd:18ab  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:bd:18:ab  txqueuelen 1000  (Ethernet)
        RX packets 530404  bytes 42805477 (40.8 MiB)
        RX errors 0  dropped 27  overruns 0  frame 0
        TX packets 161657  bytes 11761643 (11.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens34: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.0.14  netmask 255.255.255.0  broadcast 10.0.0.255
        inet6 fe80::20c:29ff:febd:18b5  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:bd:18:b5  txqueuelen 1000  (Ethernet)
        RX packets 23434  bytes 1855892 (1.7 MiB)
        RX errors 0  dropped 75  overruns 0  frame 0
        TX packets 108310  bytes 269122900 (256.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 183  bytes 38352 (37.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 183  bytes 38352 (37.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tun0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1500
        inet 172.16.1.1  netmask 255.255.255.0  destination 172.16.1.1
        inet6 fe80::3f1c:65c6:fa29:d5ae  prefixlen 64  scopeid 0x20<link>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 100  (UNSPEC)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3  bytes 144 (144.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         x.x.x.73        0.0.0.0         UG    0      0        0 ens33
10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 ens34
172.16.1.0      0.0.0.0         255.255.255.0   U     0      0        0 tun0
x.x.x.72        0.0.0.0         255.255.255.248 U     0      0        0 ens33

ping -I ens33 4.2.2.2 works
ping -I ens33 4.2.2.2 (doesn't work, even when the gw was set)

[stephenw10]

It's not working because you have an asymmetric route.

That rule looks wrong though. If you're trying to connect to the server via a port forward to the internal interface shouldn't that be ens34?
Or is that your rule to allow directly SSHing in?

You would need a rule to accept traffic from public IP on ens34 if you are not outbound NATing in pfSense.

Steve

[lewis]

Hi,

Sorry for the delay.
What you see is after all kinds of testing so is not as it used to be.
Since I cannot gain access via the ens34, I have a rule to allow me to ssh into the server from the public side.
Yes, I did also try adding a rule to allow my public IP to connect to ssh via the firewall but that is what is not working.

No where else on this network do I have or need outbound rules so I'd rather avoid doing that if possible.

[stephenw10]

Ok, as I understand it you only see this issue on servers with dual NICs where one interface is a pubic IP, correct?

That is entirely expected if you are port forwarding to the internal interface from pfSense which also has an interface on the internal network.
Port forwading does not change the source IP address so the server will reply to the remote public IP via the closest logical interface which is it's public IP. Hence packets come in on the private interface but replies go out of the public interface, an asymmetric route.

As I said there are two solutions:

Add a route to the server back to the remote IP via the pfSense internal gateway.

Add outbound NAT in pfSense so the server sees the connection as coming from the pfSense internal IP and replies back it it directly.

You would obviously also need to appropriate firewall rule on the server for each case. In the NAT case though it looks like it already allows traffic from it's own subnet.

Steve

[lewis]

Thanks very much for all of your help on this. I've done as you suggested and all is working as it should be now.

Thank you for sticking to this topic and sorry for taking so long to get back to it. I have way too many tabs open :).