static route to a network, but only part of it is connectable

tonyalbers · Aug 25, 2018, 6:28 AM

Hi guys,
I have a dual WAN setup:
WAN1 is a fiber connection that goes to a 100.84.x network with a 255.255.240.0 subnet mask. WAN1 Gateway is 100.84.a.b
WAN2 is a wireless that goes to a 192.168.100.0/24 subnet.
WAN2 Gateway is 192.168.100.1

WAN2 Gateway provides access to a network called 10.0.0.0/8.

I've set up static routing in pfsense and created a firewall rule that allows LAN-> 10.0.0.0/8

Now it gets weird:
tba@emu:~$ ping 10.64.1.162
PING 10.64.1.162 (10.64.1.162) 56(84) bytes of data.
64 bytes from 10.64.1.162: icmp_seq=1 ttl=56 time=4.77 ms
64 bytes from 10.64.1.162: icmp_seq=2 ttl=56 time=4.88 ms
^C
--- 10.64.1.162 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 4.771/4.826/4.881/0.055 ms
[control-c]
tba@emu:~$
tba@emu:~$ ping 10.113.64.10
PING 10.113.64.10 (10.113.64.10) 56(84) bytes of data.
64 bytes from 10.113.64.10: icmp_seq=1 ttl=60 time=8.48 ms
[control-c]
--- 10.113.64.10 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 8.482/8.482/8.482/0.000 ms
[control-c]
tba@emu:~$

As you can see, both hosts reply.

Now, if I try to ssh to them, only one works:
tba@emu:~$ ssh root@10.64.1.162

root@10.64.1.162's password:
[control-c]
tba@emu:~$
tba@emu:~$ ssh root@10.113.64.10
ssh: connect to host 10.113.64.10 port 22: Connection refused
tba@emu:~$

What's going on?

If I connect my laptop to another port on the 192.168.100.1 gateway, I can ssh into both hosts.

So something is worng in my box, but I can't figure out what it is.

Any suggestions?

Routing tables

Internet:
Destination Gateway Flags Netif Expire
default 100.84.96.1 UGS nfe0
10.0.0.0/8 192.168.100.1 UGS igb1
100.84.a.b link#1 U nfe0
100.84.x.y link#1 UHS lo0
127.0.0.1 link#6 UH lo0
192.168.1.0/24 link#2 U igb0
192.168.1.1 link#2 UHS lo0
192.168.2.0/24 link#5 U igb3
192.168.2.1 link#5 UHS lo0
192.168.3.0/24 link#4 U igb2
192.168.3.1 link#4 UHS lo0
192.168.100.0/24 link#3 U igb1
192.168.100.10 link#3 UHS lo0
193.162.159.194 192.168.100.1 UGHS igb1
194.239.134.83 100.84.96.1 UGHS nfe0
208.67.222.222 100.84.96.1 UGHS nfe0

Thanks,

/tony

thesurf · Aug 25, 2018, 7:02 AM

You didn't tell us from which network you make your ssh connections. So the IP range from the lan.

I would try to ssh into pfsense and open a shell. The use tcpdump to see if the ssh response reach the pfsense box. If the ssh answer packages don't make it to port wan2 oder don't go out there the problem is in pfsense. Else you need to search on a nether place.

Also you might check firewall log. Maybe you got an asynchronous routing and there fore the firewall block packages because of unknown or wrong state. Icmp does not have session states.

Hope that helps you.

tonyalbers · Aug 26, 2018, 5:43 AM

Thanks surf

Yes, sorry about that. LAN is 192.168.1.0/24

This is the tcpdump of the one that's working:
07:28:07.291298 IP 192.168.100.10.55145 > 10.64.1.162.ssh: Flags [S], seq 4185587354, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1543212549 ecr 0], length 0
07:28:07.295556 IP 10.64.1.162.ssh > 192.168.100.10.55145: Flags [S.], seq 3341411139, ack 4185587355, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
07:28:07.295633 IP 192.168.100.10.55145 > 10.64.1.162.ssh: Flags [.], ack 1, win 513, length 0
07:28:07.297550 IP 192.168.100.10.55145 > 10.64.1.162.ssh: Flags [P.], seq 1:39, ack 1, win 513, length 38
07:28:07.301223 IP 10.64.1.162.ssh > 192.168.100.10.55145: Flags [.], ack 39, win 229, length 0
07:28:07.305306 IP 10.64.1.162.ssh > 192.168.100.10.55145: Flags [P.], seq 1:40, ack 39, win 229, length 39
07:28:07.305361 IP 192.168.100.10.55145 > 10.64.1.162.ssh: Flags [.], ack 40, win 512, length 0
07:28:07.307958 IP 192.168.100.10.55145 > 10.64.1.162.ssh: Flags [P.], seq 39:1375, ack 40, win 513, length 1336
07:28:07.309512 IP 10.64.1.162.ssh > 192.168.100.10.55145: Flags [P.], seq 40:992, ack 39, win 229, length 952
07:28:07.309576 IP 192.168.100.10.55145 > 10.64.1.162.ssh: Flags [.], ack 992, win 505, length 0
and so on.

This is the one that's not:
07:36:34.211123 IP 192.168.100.10.27377 > 10.113.64.10.ssh: Flags [S], seq 3380059398, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1543719469 ecr 0], length 0
07:36:34.222752 IP 10.113.64.10.ssh > 192.168.100.10.27377: Flags [R.], seq 0, ack 3380059399, win 0, length 0
And that's it. So it sends a reset-package, but I don't know why.

tonyalbers · Aug 26, 2018, 5:59 AM

I also tried http, same result:

07:58:12.295711 IP 192.168.100.10.56799 > 10.113.64.10.http: Flags [S], seq 234521428, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1545017553 ecr 0], length 0
07:58:12.303827 IP 10.113.64.10.http > 192.168.100.10.56799: Flags [R.], seq 0, ack 234521429, win 0, length 0

thesurf · Aug 26, 2018, 6:39 AM

Http is TCP on top of IP. So http ist just one step up in osi layer.

johnpoz · Aug 26, 2018, 1:57 PM

If your getting back RST, points to the device your connecting to sending that back.. Pfsense would not send back RST out of the box. Where exactly are you sniffing that at?

thesurf · Aug 26, 2018, 3:16 PM

For me this looks like your box ssh box to which you connect did send the reset.

tonyalbers · Sep 11, 2018, 5:17 AM

Hi guys,

Thanks for your help, I guess it's because of a NAT thing, there is an extra layer between me and the host I cannot connect to.
I'm off to go move some cables around :)

Thx again

/tony