VLAN troubles, trouble connecting from VLAN 10 to LAN, others work.
-
TL:DR; I am trying to setup a Zabbix server to monitor my machines. My Proxmox server, 192.168.0.2 on LAN is not accessible from VLAN 10, where my Zabbix server is. However, it is accessible from VLAN 20.
EDIT: UPDATE: It was an half-assed IPSEC service grabbing traffic. Make sure you check IPSEC.
I am running an proxmox system with many containers and VMs. I've setup an OVS network for all my VMs. My VLANs are tagged with 10, 20, 30, and 40. VLAN 30 is an extra vlan for future expansion, so there are no devices on it. VLAN 40 is a VPN VLAN group. It all seems to work as intended, but for some reason, but I cannot ping the Proxmox server from VLAN 10. I am stumped as to why. See below for screen shots of various things.
To the proxmox server from the various interfaces. It doesn't work from VLAN10
11:54:30.504445 IP 192.168.0.1 > 192.168.0.2: ICMP echo request, id 51097, seq 0, length 64 11:54:30.504467 IP 192.168.0.2 > 192.168.0.1: ICMP echo reply, id 51097, seq 0, length 64 11:54:46.989918 IP 192.168.20.1 > 192.168.0.2: ICMP echo request, id 59811, seq 0, length 64 11:54:46.990037 IP 192.168.0.2 > 192.168.20.1: ICMP echo reply, id 59811, seq 0, length 64 11:54:49.214475 IP 192.168.30.1 > 192.168.0.2: ICMP echo request, id 58537, seq 0, length 64 11:54:49.214596 IP 192.168.0.2 > 192.168.30.1: ICMP echo reply, id 58537, seq 0, length 64 11:54:51.741363 IP 192.168.40.1 > 192.168.0.2: ICMP echo request, id 7087, seq 0, length 64 11:54:51.741479 IP 192.168.0.2 > 192.168.40.1: ICMP echo reply, id 7087, seq 0, length 64To the Zabbix server on VLAN10.
11:58:58.005643 IP 192.168.0.1 > 192.168.10.4: ICMP echo request, id 13249, seq 0, length 64 11:58:58.005961 IP 192.168.10.4 > 192.168.0.1: ICMP echo reply, id 13249, seq 0, length 64 11:59:01.497792 IP 192.168.10.1 > 192.168.10.4: ICMP echo request, id 41160, seq 0, length 64 11:59:01.498034 IP 192.168.10.4 > 192.168.10.1: ICMP echo reply, id 41160, seq 0, length 64 11:59:03.925592 IP 192.168.20.1 > 192.168.10.4: ICMP echo request, id 49613, seq 0, length 64 11:59:03.925836 IP 192.168.10.4 > 192.168.20.1: ICMP echo reply, id 49613, seq 0, length 64 11:59:06.767907 IP 192.168.30.1 > 192.168.10.4: ICMP echo request, id 36051, seq 0, length 64 11:59:06.768146 IP 192.168.10.4 > 192.168.30.1: ICMP echo reply, id 36051, seq 0, length 64 11:59:09.324393 IP 192.168.40.1 > 192.168.10.4: ICMP echo request, id 26141, seq 0, length 64 11:59:09.324622 IP 192.168.10.4 > 192.168.40.1: ICMP echo reply, id 26141, seq 0, length 64Thanks in advance. What am I missing here?
-
What is not answering your Pings? Does it use pfsense for its gateway, does it run its own firewall?
-
From proxmox (192.168.0.2) perspective, it cannot ping anything on the VLAN10 network (192.168.10.0/24), including the gateway (192.168.10.1). Anything on the VLAN10 Network cannot ping 192.168.0.2 (Proxmox), but it can ping the gateway (192.168.0.1). Proxmox and VLAN 10 network can ping VLAN 20, and VLAN 20 can ping LAN and VLAN 10.
I checked proxmox host. I've disabled the firewall. It still doesn't work. All other symptoms still occur.
-
Most likely the proxmox server has no route or a bad route to 192.168.10.X. Or it has a firewall running on it blocking access from that subnet.
Can you ping out from Proxmox to anything in VLAN10?Steve
-
@stephenw10 I cannot ping anything from Promox to VLAN 10. I can ping from Proxmox to VLAN 20. Proxmox can ping the LAN gateway. How would I diagnose bad routes?
-
Look at its routes... It really should only have the default I would think.
When you do a traceroute does its actually send the traffic to pfsense... Or does it maybe think that network is local to it?
-
Proxmox Routes:
pve:~$ sudo route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default 192.168.0.1 0.0.0.0 UG 0 0 0 vmbr1 192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 vmbr1Zabbix Routes:
zabbix ~# route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default 192.168.10.1 0.0.0.0 UG 0 0 0 eth0 192.168.10.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0webserver routes on VLAN20:
# route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default 192.168.20.1 0.0.0.0 UG 0 0 0 eth0 192.168.20.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0Proxmox traceroute to 192.168.10.4:
david@pve:~$ sudo traceroute 192.168.10.2 traceroute to 192.168.10.2 (192.168.10.2), 30 hops max, 60 byte packets 1 192.168.0.1 (192.168.0.1) 0.312 ms 0.341 ms 0.472 ms 2 * * * 3 * * * 4 * * * 5 * * * 6 * * *Zabbix traceroute to 192.168.0.2
zabbix ~# traceroute 192.168.0.2 traceroute to 192.168.0.2 (192.168.0.2), 30 hops max, 60 byte packets 1 192.168.10.1 (192.168.10.1) 0.437 ms 0.455 ms 0.526 ms 2 * * * 3 * * * 4 * * * 5 * * *web server on VLAN20 to 192.168.0.2
:~# traceroute 192.168.0.2 traceroute to 192.168.0.2 (192.168.0.2), 30 hops max, 60 byte packets 1 192.168.20.1 (192.168.20.1) 0.396 ms 0.424 ms 0.458 ms 2 192.168.0.2 (192.168.0.2) 1.305 ms 1.307 ms 1.327 ms -
Hmm, looks OK.
I'd run packet captures on the interfaces in pfSense whist trying to ping between the hosts. See exactly what is arriving an leaving.
Steve
-
Yeah maybe your mask is wrong on the box your trying to ping... Your multiple vlans getting confusing..
Lets just deal with the vlan talking to the other vlan
You hit the right first hop and then nothing..
Ok then sniff on the other vlan or the client do you see the ping? Does the dest IP your trying to ping have the correct /24 mask?
-
Looks like all the machines have /24 operating.
Here is what the ICMP looks like from the LAN side with packet sniffing.13:54:18.518924 52:5c:ff:1f:67:44 > 32:e8:8d:67:76:e5, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 25026, offset 0, flags [DF], proto ICMP (1), length 84) 192.168.0.2 > 192.168.10.4: ICMP echo request, id 25888, seq 1, length 64Does it come out the other side? Nope. Nothing comes out to VLAN10.
Same way in reverse.
-
So these are all vlan tagged on pfsense and just 1 vitrual nic em1.. Why would you not do the vlans in your vm hosting software..
Makes no sense that pfsense would get a packet and not send it somewhere..
-
Exactly. I don't understand why it wouldn't route it. Yep, here is my config on the proxmox. There has to be something I am missing.
Well..I am. Doing VLANs this way is recommended by Proxmox. Read up more here: linkauto lo iface lo inet loopback allow-ovs vmbr0 allow-vmbr0 eno1 iface eno1 inet manual ovs_type OVSPort ovs_bridge vmbr0 auto vmbr0 iface vmbr0 inet manual ovs_type OVSBridge ovs_ports eno1 auto vmbr1 iface vmbr1 inet static address 192.168.0.2 netmask 255.255.255.0 gateway 192.168.0.1 ovs_type OVSBridge ovs_ports vlan10 vlan20 vlan30 vlan40 allow-vmbr1 vlan10 iface vlan10 inet manual ovs_type OVSIntPort ovs_bridge vmbr1 ovs_options tag=10 allow-vmbr1 vlan20 iface vlan20 inet manual ovs_type OVSIntPort ovs_bridge vmbr1 ovs_options tag=20 allow-vmbr1 vlan30 iface vlan30 inet manual ovs_type OVSIntPort ovs_bridge vmbr1 ovs_options tag=30 allow-vmbr1 vlan40 iface vlan40 inet manual ovs_type OVSIntPort ovs_bridge vmbr1 ovs_options tag=40 -
I should point out that I have a plex machine setup with a NAT for VLAN10. So, it's accessible from WAN.
-
PFSense Routes. Scrubbed of some public information.
IPv4 Routes Destination Gateway Flags Use Mtu Netif Expire default WAN GATEWAY UGS 1058841 1500 em0 10.1.0.0/24 10.1.0.2 UGS 1746 1500 ovpns1 10.1.0.1 link#11 UHS 0 16384 lo0 10.1.0.2 link#11 UH 0 1500 ovpns1 10.41.10.9 link#13 UH 3615 1500 ovpnc3 10.41.10.10 link#13 UHS 0 16384 lo0 10.72.10.5 link#12 UH 3632 1500 ovpnc2 10.72.10.6 link#12 UHS 0 16384 lo0 WAN IP/29 link#1 U 3640 1500 em0 WAN ANOTHER IP link#1 UHS 0 16384 lo0 WAN ANOTHER IP link#1 UHS 0 16384 lo0 WAN ANOTHER IP link#1 UHS 0 16384 lo0 127.0.0.1 link#4 UH 118 16384 lo0 192.168.0.0/24 link#2 U 5068 1500 em1 192.168.0.1 WAN GATEWAY UGHS 0 1500 em0 192.168.10.0/24 link#7 U 396258 1500 em1.10 192.168.10.1 link#7 UHS 0 16384 lo0 192.168.20.0/24 link#8 U 74842 1500 em1.20 192.168.20.1 link#8 UHS 0 16384 lo0 192.168.30.0/24 link#9 U 0 1500 em1.30 192.168.30.1 link#9 UHS 0 16384 lo0 192.168.40.0/24 link#10 U 3 1500 em1.40 192.168.40.1 link#10 UHS 0 16384 lo0 -
Normally I would look for policy based routing here. You don't have a gateway set in any of the rules shown above but you could have floating rules or an interface group passing that traffic and sending it out the WAN.
You might also have an IPSec VPN grabbing that traffic.
Steve
-
Ah.......it wasn't DNS this time, but it was IPSEC. It was some pesky IPSEC that wasn't fully enabled, only enabled on one side.
A few months ago I was going to setup some IPSEC stuff but gave up on it' I guess I never disabled it. I went ahead and disabled my half-assed IPSEC and now it's working.
I want to thank everyone for their help. Appreciate it. This was a tough one for me.
