Cannot ssh from firewall to LAN
-
A server sits on the network. It has two interfaces.
One has one port open directly onto the Internet, the other is on the LAN side.On single NIC servers, I can ssh/etc into those from the firewall. On dual NIC servers, sometimes I see this problem where I cannot.
In this case, I can ssh into the server from the public IP by allowing a single IP to access ssh. However, I want to allow ssh only from the firewall or LAN side.
I added an iptables rule on the server and the firewall to allow ssh from one public IP but it never gets to the server.
However, if I change the rule on the server to another server on the LAN, it can gain access.Because it is a dual NIC server, I have the public nic as the default gw and on the LAN NIC, I do not.
I suspect something to do with the gateway where the second NIC doesn't know how to reach the firewall or something. I'm able to use the second NIC to ping out, I just can't get any traffic into it.
-
You are actually SSHing from pfSense rather than port forwarding to the internal server?
If that is the case the server has an interface in the same subnet as pfSense, it should always be able to reply to that. That sounds like a server side issue, probably nothing we do in pfSense there.
Steve
-
Hi,
I am ssh'ing from an allowed public IP to the pfsense box which has a rule on it to allow ssh from my IP to that server.
Nothing reaches it from outside of pfsense but I can reach this server from other servers on the LAN side.
On that same server, I can ping to 4.2.2.2 for example using -I on the first (public facing) NIC but using the second (LAN sided NIC) NIC, I get nothing.
# ping -I ens33 4.2.2.2 PING 4.2.2.2 (4.2.2.2) from x.x.x.x ens33: 56(84) bytes of data. 64 bytes from 4.2.2.2: icmp_seq=1 ttl=59 time=9.20 ms ^C --- 4.2.2.2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 9.205/9.205/9.205/0.000 ms # ping -I ens34 4.2.2.2 PING 4.2.2.2 (4.2.2.2) from 10.0.0.14 ens34: 56(84) bytes of data. ^C --- 4.2.2.2 ping statistics --- 2 packets transmitted, 0 received, 100% packet loss, time 999ms -
How is it possible that no one on this site knows why this is happening?
-
Still unclear if you are SSHing to pfSense and then from there to the internal server or to the server directly via a port forward?
If the server that is the target has two NICs and one is a public IP how the traffic going through pfSense? A diagram may help here.
Steve
-
@stephenw10 said in Cannot ssh from firewall to LAN:
Still unclear if you are SSHing to pfSense and then from there to the internal server or to the server directly via a port forward?
If the server that is the target has two NICs and one is a public IP how the traffic going through pfSense? A diagram may help here.
SteveHi,
Wow, the site keeps saying this is a spam comment.
Error
Post content was flagged as spam by Akismet.comI'll try to show a basic diagram.
NIC0---Public IP (openvpn)---Internet
data center server
NIC1---LAN (ssh)---pfsense---InternetOn the server, I have an iptables rule that allows my IP to access the ssh service.
-A INPUT -s x.x.x.x/32 -i ens34 -p tcp -m tcp --dport 4xx -j ACCEPT
On pfsense, there is a rule to allow ssh to the above server via the LAN interface. The ssh service is of course using the custom ssh port.
From pfsense, using the GUI or command line, I can ping this servers LAN IP. However, using nmap from the command line, I see the port is filtered.
PORT STATE SERVICE
4xx/tcp filtered lyskomThere is nothing else running on this port and I use custom ports on everything so this is not new to the setup.
So, no, I'm not trying to ssh from the pfsense directly but through a rule I've created that should forward. Of course, since the port appears to be filtered even from the command line of pfsense, I'm at a loss since I can allow any other server on the LAN side to ssh into it by adding a rule on the server.
-
Ok,
Yes so you have a port forward configured in pfSense to allow SSH from your rempote public IP to the internal IP of the server.It fails when you nmap (or port test) from pfSense itself because that would be sourced from the pfSense LAN IP and the rule on the server only allows traffic from the remote public IP.
The port forward probably fails because the server has a public IP and hence likely a default route via the other NIC. Hence it replies out on the wrong interface creating an asymmetric route.
Correct the routing on the server or add an outbound NAT rule on the pfSense LAN and change the rule on the server to allow that as source instead.
Steve
-
Thank you, at least you confirm there is nothing wrong on the firewall side. I do believe there is a route missing on the openvpn server that as you say, is causing outgoing lan traffic to come back via the public IP but that's what I've been struggling to fix and is how I got to this question.
I'll have to go ask somewhere else for that since it's not pfsense related.
Thank you.
-
stephenw10 Netgate Administratorlast edited by stephenw10 Aug 16, 2020, 7:26 PM Aug 16, 2020, 7:24 PM
If it's only one IP you need access from via the internal interface you could just add that route to the server.
A tight outbound NAT rule on the pfSense LAN interface would allow more flexibility if you need to add more external IPs to that later. It also wouldn't break OpenVPN access from those IPs assuming that would connect to the server public IP directly.
Or you could do as I originally thought you were and ssh into the pfSense box and then ssh fro there to the server when you need to connect to it. The rule on the server would have to allow that of course.
Steve
-
Yes, I could ssh into the pfsense box and go from there but that's an extra step. Only this server is having this problem. I've got other servers that have dual NICs and those don't have the issue.
I think what I need to find is a way to add a default route for that interface on that server so that traffic going in and out the LAN IP can get routed to the firewall.
However, that's what I've been trying to do and so far, have yet to figure out how to make it work.
The public NIC/IP should be the default for the default but I've not been able to find a way to give the LAN IP a default route without messing up the public side.
Not sure this makes sense now.
-
It would not be a route to the firewall but back to the public IP you're connecting from. That's the problem here, the IP is not local so the server uses it's default route.
You can't set two default routes like that (not in this context anyway) you would need to just add a static route for that one IP.
Adding an outbound NAT rule in pfSense would normally be considered ugly but in this case it makes sense as it doesn't affect other connections from that IP. It's probably what I would do.Steve
-
Yes, I didn't mean to say default route. I mean to say give the LAN side IP a route back to the firewall so that its traffic flow to/from there and not through the default route.
I've tried and tried but for some reason, nothing works.
I'll look at adding an outbound and see if that works. -
[root@# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 x.x.x.13 0.0.0.0 UG 0 0 0 ens33
10.0.0.0 10.0.0.1 255.255.255.0 UG 0 0 0 ens34
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 ens34
172.16.1.0 0.0.0.0 255.255.255.0 U 0 0 0 tun0
x.x.x.12 0.0.0.0 255.255.255.248 U 0 0 0 ens33The above was after I added a gateway for the LAN side but still cannot get anything back.
ping -I ens34 4.2.2.2
Nothing, no reply. Now it's back to the following.
[root@# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 x.x.x.13 0.0.0.0 UG 0 0 0 ens33
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 ens34
172.16.1.0 0.0.0.0 255.255.255.0 U 0 0 0 tun0
x.x.x.12 0.0.0.0 255.255.255.248 U 0 0 0 ens33I must be missing something that is right in my face but am not seeing it.
-
Grab some screenshots for review if that still doesn't work.
-
I could do that if I can edit out the Ips since they are not mine to show/share. What should I show? Or do you want me to run a few commands and share the outputs?
-
Well we need to see what your WAN firewall rules are, what you port forwrad rules are and what your outbound NAT rule is.
The above route on the server would have done nothing because the incoming traffic comes from a public IP not inside the 10.0.0.0/24 subnet. The outbound NAT rule should change that.
It also does nothing because the server already has a route to 10.0.0.0/24 as it has an interface in that subnet.
The route on the server would need to be:
<the public IP you are connecting from> 10.0.0.1 255.255.255.0 UG 0 0 0 ens34
If you choose to do it that way.
I would use outbound NAT in pfSense for this for the reasons I outlined.Steve
-
Here is what the server looks like. Public IP obfuscated. Is this what you wanted or something else, more? I've not had to set any outbound rules on pfsense to get out from other servers where they have a dual NIC also. I checked over and over again to compare and cannot find any reason for this not working.
# iptables -t nat -S -P PREROUTING ACCEPT -P INPUT ACCEPT -P OUTPUT ACCEPT -P POSTROUTING ACCEPT -A POSTROUTING -j MASQUERADE # iptables -S -P INPUT DROP -P FORWARD ACCEPT -P OUTPUT ACCEPT -A INPUT -p udp -m udp --dport 1194 -j ACCEPT -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -p udp -m udp --dport 7777 -j ACCEPT -A INPUT -s x.x.x.x/32 -i ens33 -p tcp -m tcp --dport 22 -j ACCEPT -A FORWARD -i tun+ -o ens160 -j ACCEPT -A FORWARD -i ens160 -o tun+ -j ACCEPT -A FORWARD -i tun+ -o ens192 -j ACCEPT -A FORWARD -i ens192 -o tun+ -j ACCEPT # ifconfig ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet x.x.x.78 netmask 255.255.255.248 broadcast x.x.x.79 inet6 fe80::20c:29ff:febd:18ab prefixlen 64 scopeid 0x20<link> ether 00:0c:29:bd:18:ab txqueuelen 1000 (Ethernet) RX packets 530404 bytes 42805477 (40.8 MiB) RX errors 0 dropped 27 overruns 0 frame 0 TX packets 161657 bytes 11761643 (11.2 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 ens34: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.0.0.14 netmask 255.255.255.0 broadcast 10.0.0.255 inet6 fe80::20c:29ff:febd:18b5 prefixlen 64 scopeid 0x20<link> ether 00:0c:29:bd:18:b5 txqueuelen 1000 (Ethernet) RX packets 23434 bytes 1855892 (1.7 MiB) RX errors 0 dropped 75 overruns 0 frame 0 TX packets 108310 bytes 269122900 (256.6 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 1000 (Local Loopback) RX packets 183 bytes 38352 (37.4 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 183 bytes 38352 (37.4 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 tun0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST> mtu 1500 inet 172.16.1.1 netmask 255.255.255.0 destination 172.16.1.1 inet6 fe80::3f1c:65c6:fa29:d5ae prefixlen 64 scopeid 0x20<link> unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 txqueuelen 100 (UNSPEC) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 3 bytes 144 (144.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 # route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 x.x.x.73 0.0.0.0 UG 0 0 0 ens33 10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 ens34 172.16.1.0 0.0.0.0 255.255.255.0 U 0 0 0 tun0 x.x.x.72 0.0.0.0 255.255.255.248 U 0 0 0 ens33 ping -I ens33 4.2.2.2 works ping -I ens33 4.2.2.2 (doesn't work, even when the gw was set) -
It's not working because you have an asymmetric route.
That rule looks wrong though. If you're trying to connect to the server via a port forward to the internal interface shouldn't that be ens34?
Or is that your rule to allow directly SSHing in?You would need a rule to accept traffic from public IP on ens34 if you are not outbound NATing in pfSense.
Steve
-
Hi,
Sorry for the delay.
What you see is after all kinds of testing so is not as it used to be.
Since I cannot gain access via the ens34, I have a rule to allow me to ssh into the server from the public side.
Yes, I did also try adding a rule to allow my public IP to connect to ssh via the firewall but that is what is not working.No where else on this network do I have or need outbound rules so I'd rather avoid doing that if possible.
-
Ok, as I understand it you only see this issue on servers with dual NICs where one interface is a pubic IP, correct?
That is entirely expected if you are port forwarding to the internal interface from pfSense which also has an interface on the internal network.
Port forwading does not change the source IP address so the server will reply to the remote public IP via the closest logical interface which is it's public IP. Hence packets come in on the private interface but replies go out of the public interface, an asymmetric route.As I said there are two solutions:
Add a route to the server back to the remote IP via the pfSense internal gateway.
Add outbound NAT in pfSense so the server sees the connection as coming from the pfSense internal IP and replies back it it directly.
You would obviously also need to appropriate firewall rule on the server for each case. In the NAT case though it looks like it already allows traffic from it's own subnet.
Steve