LAN to VLAN ssh access
-
I've setup a separate VLAN as a Kubernetes playground but having an odd issue. From my workstation I can ping the kub-master but can't SSH to it and tcpdump and packet capture on the pfsense says it isn't routing from one interface to the other.
Setup:
[192.168.1.0/24 LAN] <-> pfsense (3100 23.09-RELEASE) <-> [192.168.107.0/24 KUB]
.1 on either network is the pfsense.LAN 1st rule: PASS ipv4+6 allow any to any
KUB 1st rule: PASS ipv4+6 allow any to anyI have disabled the firewall on the kub-master (ufw disable). tcpdump port 22 shows no packets.
On LAN (mvneta1.4091) - capturing port 22:
Running packet capture: /usr/sbin/tcpdump -ni mvneta1.4091 -c '1000' -U -w - '((port 22)) or ((vlan))' 06:30:44.784510 IP 192.168.1.18.35960 > 192.168.107.10.22: tcp 0 06:30:47.562469 IP 192.168.1.18.38238 > 192.168.107.1.22: tcp 52 06:30:47.562537 IP 192.168.107.1.22 > 192.168.1.18.38238: tcp 0 06:30:47.562668 IP 192.168.107.1.22 > 192.168.1.18.38238: tcp 28 06:30:47.562948 IP 192.168.1.18.38238 > 192.168.107.1.22: tcp 0On KUB_NETWORK (mvneta1.107) - capturing port 22:
Running packet capture: /usr/sbin/tcpdump -ni mvneta1.107 -c '1000' -U -w - '((port 22)) or ((vlan))' <no activity> -
@aram535
Do you have any port forwarding on LAN?I'm wondering, why the LAN capture is showing SSH access from your computer to the KUB interface, which is obviously responding then:
06:30:44.784510 IP 192.168.1.18.35960 > 192.168.107.10.22: tcp 0
06:30:47.562469 IP 192.168.1.18.38238 > 192.168.107.1.22: tcp 52
06:30:47.562537 IP 192.168.107.1.22 > 192.168.1.18.38238: tcp 0
06:30:47.562668 IP 192.168.107.1.22 > 192.168.1.18.38238: tcp 28
06:30:47.562948 IP 192.168.1.18.38238 > 192.168.107.1.22: tcp 0Any explanation on this?
I guess, 192.168.107.10 is the kub-master. So I'm expecting to see only this IP as destination, if you don't have any other SSH connection concurrently.
Capture the ping on both interfaces to see if the packets are routed properly.
Remember that ICMP is not a stateful protocol. So the reason could be asymmetric routing. -
@viragomann It's a flat network so there is no other routing other than on the PfSence (unless VLANs are getting ignored).
Network design:
ESXi - vSwitch (standard, 1 host) - portgroup 107 (vlan) ... where all of kub vms exist -> Unifi USW 24 PoE -> trunk port -> TP-Link T1600G-28TS 3.0 -> Kub network on Switch as Untagged ports ] -> Pfsense and Workstation.
workstation (192.168.1.18). Can ping it's own gateway 192.168.1.1 and the 192.168.107.1 (pfsence vlan interface).
However, it cannot ping kub-master (192.168.107.10).workstation:
$ netstat -nr Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 ens192 192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 ens192 $ ping 192.168.107.1 PING 192.168.107.1 (192.168.107.1) 56(84) bytes of data. 64 bytes from 192.168.107.1: icmp_seq=1 ttl=64 time=0.301 ms 64 bytes from 192.168.107.1: icmp_seq=2 ttl=64 time=0.435 ms $ sudo tcpdump -n icmp and net 192.168.107.0/24 dropped privs to tcpdump tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on ens192, link-type EN10MB (Ethernet), snapshot length 262144 bytes 10:09:21.179729 IP 192.168.1.18 > 192.168.107.1: ICMP echo request, id 17, seq 1, length 64 10:09:21.179995 IP 192.168.107.1 > 192.168.1.18: ICMP echo reply, id 17, seq 1, length 64 10:09:22.224486 IP 192.168.1.18 > 192.168.107.1: ICMP echo request, id 17, seq 2, length 64 10:09:22.224855 IP 192.168.107.1 > 192.168.1.18: ICMP echo reply, id 17, seq 2, length 64 10:09:23.248402 IP 192.168.1.18 > 192.168.107.1: ICMP echo request, id 17, seq 3, length 64 10:09:23.248687 IP 192.168.107.1 > 192.168.1.18: ICMP echo reply, id 17, seq 3, length 64 10:09:25.115966 IP 192.168.1.18 > 192.168.107.10: ICMP echo request, id 18, seq 1, length 64 10:09:26.128484 IP 192.168.1.18 > 192.168.107.10: ICMP echo request, id 18, seq 2, length 64 10:09:27.152498 IP 192.168.1.18 > 192.168.107.10: ICMP echo request, id 18, seq 3, length 64 10:09:28.176534 IP 192.168.1.18 > 192.168.107.10: ICMP echo request, id 18, seq 4, length 64 10:09:29.200461 IP 192.168.1.18 > 192.168.107.10: ICMP echo request, id 18, seq 5, length 64 10:09:30.224490 IP 192.168.1.18 > 192.168.107.10: ICMP echo request, id 18, seq 6, length 64 10:09:31.248498 IP 192.168.1.18 > 192.168.107.10: ICMP echo request, id 18, seq 7, length 64The kub-master routing is a bit more complicated:
$ netstat -nr Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 192.168.107.1 0.0.0.0 UG 0 0 0 ens160 10.107.65.0 192.168.107.11 255.255.255.0 UG 0 0 0 tunl0 10.107.66.0 192.168.107.12 255.255.255.0 UG 0 0 0 tunl0 10.107.75.0 192.168.107.13 255.255.255.0 UG 0 0 0 tunl0 10.107.103.0 0.0.0.0 255.255.255.0 U 0 0 0 * 192.168.107.0 0.0.0.0 255.255.255.0 U 0 0 0 ens160 $ cat /etc/netplan/00-static.yaml network: version: 2 renderer: networkd ethernets: myinterface: match: name: ens* dhcp4: false dhcp6: false addresses: [192.168.107.10/24] nameservers: addresses: [192.168.107.1] routes: - to: default via: 192.168.107.1 -
@aram535 said in LAN to VLAN ssh access:
ESXi - vSwitch (standard, 1 host) - portgroup 107 (vlan) ... where all of kub vms exist -> Unifi USW 24 PoE -> trunk port -> TP-Link T1600G-28TS 3.0 -> Kub network on Switch as Untagged ports ] -> Pfsense and Workstation.
So there are some devices in between the KUB devices and the router. Can you even access the internet from this subnet?
Can you ping pfSense from the kub-master or the other way round? -
@viragomann I honestly didn't even check internet but no. Doesn't seem to have access.
kub-master can ping the vmnic on the portgroup which means it is leaving the ESXi server, but then either the ubiquiti or the tp-link is not carrying the packets up to the pfsense.
The ubiquiti says all vlans are being forwarded on the 4 ports connected to the ESXi (all 4 are uplink ports on the port group)
TP-Link has all the ports as tagged:
I even tried them as untagged just in case and no change.
I'm at a loss where to look next.