Cannot ssh from firewall to LAN

[lewis]

Yes, I didn't mean to say default route. I mean to say give the LAN side IP a route back to the firewall so that its traffic flow to/from there and not through the default route.

I've tried and tried but for some reason, nothing works.
I'll look at adding an outbound and see if that works.

[lewis]

[root@# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 x.x.x.13 0.0.0.0 UG 0 0 0 ens33
10.0.0.0 10.0.0.1 255.255.255.0 UG 0 0 0 ens34
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 ens34
172.16.1.0 0.0.0.0 255.255.255.0 U 0 0 0 tun0
x.x.x.12 0.0.0.0 255.255.255.248 U 0 0 0 ens33

The above was after I added a gateway for the LAN side but still cannot get anything back.

ping -I ens34 4.2.2.2

Nothing, no reply. Now it's back to the following.

[root@# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 x.x.x.13 0.0.0.0 UG 0 0 0 ens33
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 ens34
172.16.1.0 0.0.0.0 255.255.255.0 U 0 0 0 tun0
x.x.x.12 0.0.0.0 255.255.255.248 U 0 0 0 ens33

I must be missing something that is right in my face but am not seeing it.

[stephenw10]

Grab some screenshots for review if that still doesn't work.

[lewis]

I could do that if I can edit out the Ips since they are not mine to show/share. What should I show? Or do you want me to run a few commands and share the outputs?

[stephenw10]

Well we need to see what your WAN firewall rules are, what you port forwrad rules are and what your outbound NAT rule is.

The above route on the server would have done nothing because the incoming traffic comes from a public IP not inside the 10.0.0.0/24 subnet. The outbound NAT rule should change that.
It also does nothing because the server already has a route to 10.0.0.0/24 as it has an interface in that subnet.
The route on the server would need to be:
<the public IP you are connecting from> 10.0.0.1 255.255.255.0 UG 0 0 0 ens34
If you choose to do it that way.
I would use outbound NAT in pfSense for this for the reasons I outlined.

Steve

[lewis]

Here is what the server looks like. Public IP obfuscated. Is this what you wanted or something else, more? I've not had to set any outbound rules on pfsense to get out from other servers where they have a dual NIC also. I checked over and over again to compare and cannot find any reason for this not working.

# iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A POSTROUTING -j MASQUERADE

# iptables -S
-P INPUT DROP
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -p udp -m udp --dport 1194 -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p udp -m udp --dport 7777 -j ACCEPT
-A INPUT -s x.x.x.x/32 -i ens33 -p tcp -m tcp --dport 22 -j ACCEPT
-A FORWARD -i tun+ -o ens160 -j ACCEPT
-A FORWARD -i ens160 -o tun+ -j ACCEPT
-A FORWARD -i tun+ -o ens192 -j ACCEPT
-A FORWARD -i ens192 -o tun+ -j ACCEPT

# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet x.x.x.78  netmask 255.255.255.248  broadcast x.x.x.79
        inet6 fe80::20c:29ff:febd:18ab  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:bd:18:ab  txqueuelen 1000  (Ethernet)
        RX packets 530404  bytes 42805477 (40.8 MiB)
        RX errors 0  dropped 27  overruns 0  frame 0
        TX packets 161657  bytes 11761643 (11.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens34: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.0.14  netmask 255.255.255.0  broadcast 10.0.0.255
        inet6 fe80::20c:29ff:febd:18b5  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:bd:18:b5  txqueuelen 1000  (Ethernet)
        RX packets 23434  bytes 1855892 (1.7 MiB)
        RX errors 0  dropped 75  overruns 0  frame 0
        TX packets 108310  bytes 269122900 (256.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 183  bytes 38352 (37.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 183  bytes 38352 (37.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tun0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1500
        inet 172.16.1.1  netmask 255.255.255.0  destination 172.16.1.1
        inet6 fe80::3f1c:65c6:fa29:d5ae  prefixlen 64  scopeid 0x20<link>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 100  (UNSPEC)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3  bytes 144 (144.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         x.x.x.73        0.0.0.0         UG    0      0        0 ens33
10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 ens34
172.16.1.0      0.0.0.0         255.255.255.0   U     0      0        0 tun0
x.x.x.72        0.0.0.0         255.255.255.248 U     0      0        0 ens33

ping -I ens33 4.2.2.2 works
ping -I ens33 4.2.2.2 (doesn't work, even when the gw was set)

[stephenw10]

It's not working because you have an asymmetric route.

That rule looks wrong though. If you're trying to connect to the server via a port forward to the internal interface shouldn't that be ens34?
Or is that your rule to allow directly SSHing in?

You would need a rule to accept traffic from public IP on ens34 if you are not outbound NATing in pfSense.

Steve

[lewis]

Hi,

Sorry for the delay.
What you see is after all kinds of testing so is not as it used to be.
Since I cannot gain access via the ens34, I have a rule to allow me to ssh into the server from the public side.
Yes, I did also try adding a rule to allow my public IP to connect to ssh via the firewall but that is what is not working.

No where else on this network do I have or need outbound rules so I'd rather avoid doing that if possible.

[stephenw10]

Ok, as I understand it you only see this issue on servers with dual NICs where one interface is a pubic IP, correct?

That is entirely expected if you are port forwarding to the internal interface from pfSense which also has an interface on the internal network.
Port forwading does not change the source IP address so the server will reply to the remote public IP via the closest logical interface which is it's public IP. Hence packets come in on the private interface but replies go out of the public interface, an asymmetric route.

As I said there are two solutions:

Add a route to the server back to the remote IP via the pfSense internal gateway.

Add outbound NAT in pfSense so the server sees the connection as coming from the pfSense internal IP and replies back it it directly.

You would obviously also need to appropriate firewall rule on the server for each case. In the NAT case though it looks like it already allows traffic from it's own subnet.

Steve

[lewis]

Thanks very much for all of your help on this. I've done as you suggested and all is working as it should be now.

Thank you for sticking to this topic and sorry for taking so long to get back to it. I have way too many tabs open :).