communicating via vswitch from vms in bridges

compuser

I have two dedicated hosts on hetzner. My end goal is to be able to have devices on bridges on the hosts communicate with each other via vswitch. pfsense has a dhcp server. vm1 gets a dhcp id of 172.16.6.16. From host1 and vm1, I can ping 172.16.6.3 (host2 bridge). From host2, I can ping 172.16.6.16 (vm1). However, when I bring up vm2, it cant seem to pick up a dhcp lease. Would someone know what config I am missing that prevents vm2 from picking up the lease. Thanks!

{{ vm1, pfsense } host1 } - Vswitch - {{ vm2 } host2 }

Host1:
pfSense - vmbr6
vm1 - vmbr6

/etc/network/interfaces

auto vmbr6
iface vmbr6 inet static
        address 172.16.6.1/24
        bridge-ports enp5s0.4006
        bridge-stp off
        bridge-fd 0
        mtu 1400

auto enp5s0.4006
iface enp5s0.4006 inet manual
        vlan_raw_device enp5s0
        mtu 1400

Host 2:
vm2 - vmbr6

/etc/network/interfaces

auto vmbr6
iface vmbr6 inet static
        address 172.16.6.3/24
        bridge-ports enp5s0.4006
        bridge-stp off
        bridge-fd 0
        mtu 1400

auto enp5s0.4006
iface enp5s0.4006 inet manual
        vlan_raw_device enp5s0
        mtu 1400

vswitch:
vlanid 4006 - host1, host2

compuser

I created a vm3 on host2 with static IP 172.16.6.20 and turns out it started off with some packet loss but got better with the second and third attempt.

host1 ~ # ping 172.16.6.20
PING 172.16.6.20 (172.16.6.20) 56(84) bytes of data.
^C
--- 172.16.6.20 ping statistics ---
8 packets transmitted, 0 received, 100% packet loss, time 7180ms

host1 ~ # ping 172.16.6.20
PING 172.16.6.20 (172.16.6.20) 56(84) bytes of data.
64 bytes from 172.16.6.20: icmp_seq=7 ttl=64 time=0.911 ms
64 bytes from 172.16.6.20: icmp_seq=8 ttl=64 time=0.861 ms
64 bytes from 172.16.6.20: icmp_seq=9 ttl=64 time=0.875 ms
64 bytes from 172.16.6.20: icmp_seq=10 ttl=64 time=0.888 ms
64 bytes from 172.16.6.20: icmp_seq=11 ttl=64 time=0.915 ms
64 bytes from 172.16.6.20: icmp_seq=12 ttl=64 time=0.878 ms
64 bytes from 172.16.6.20: icmp_seq=13 ttl=64 time=0.877 ms
64 bytes from 172.16.6.20: icmp_seq=14 ttl=64 time=0.864 ms
^C
--- 172.16.6.20 ping statistics ---
14 packets transmitted, 8 received, 42.8571% packet loss, time 13333ms
rtt min/avg/max/mdev = 0.861/0.883/0.915/0.018 ms

host1 ~ # ping 172.16.6.20
PING 172.16.6.20 (172.16.6.20) 56(84) bytes of data.
64 bytes from 172.16.6.20: icmp_seq=1 ttl=64 time=0.827 ms
64 bytes from 172.16.6.20: icmp_seq=2 ttl=64 time=1.08 ms
64 bytes from 172.16.6.20: icmp_seq=3 ttl=64 time=0.543 ms
64 bytes from 172.16.6.20: icmp_seq=4 ttl=64 time=1.03 ms
64 bytes from 172.16.6.20: icmp_seq=5 ttl=64 time=0.990 ms
^C
--- 172.16.6.20 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4097ms
rtt min/avg/max/mdev = 0.543/0.893/1.080/0.194 ms

patient0

@compuser said in communicating via vswitch from vms in bridges:

Would someone know what config I am missing that prevents vm2 from picking up the lease

You could do a packet capture on pfSense to see the DHCP requests get to it. Since it's all on layer 2 it should "just work". The only unusual being the vSwitch.

Did you have you set the MTU 1400 or lower on the vmX?
Any firewall rules on the Hetzner Robot level?

compuser

@patient0

I ran tcpdump -i vmbr6 on host2. I see BOOTP requests from VM2 mac but no response from a DHCP server. I cant see any BOOTP broadcasts when I run tcpdump on host1. Could it be that the requests are not traversing the switch? On robot server, I have
ipv4 - * - src/172.16.6.0/24 - dest/172.16.6.0/24 - accept on both hosts and no deny rules.

08:06:02.274090 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from bc:24:11:4a:07:7a (oui Unknown), length 293
08:07:06.457056 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from bc:24:11:4a:07:7a (oui Unknown), length 293
08:07:26.689884 ARP, Request who-has 172.16.6.3 tell 0.0.0.0, length 42
08:07:26.689888 ARP, Reply 172.16.6.3 is-at 10:7c:61:4e:78:e5 (oui Unknown), length 28
08:07:26.689904 ARP, Reply 172.16.6.3 is-at 12:3d:86:98:ce:d9 (oui Unknown), length 28

I have set MTU of 1400 in 1) network device of VM3 in proxmox 2) for the bridge and vlan interface on both hosts and 3) on the vmbr6 interface in pfsense

patient0

@compuser said in communicating via vswitch from vms in bridges:

Could it be that the requests are not traversing the switch?

I don't have experience with vSwitch between dedicated servers, according to the internet it should work just fine.

What virtualisation software do you use for the VMs? Maybe it's more a bridge issue.

Are there any firewall rules on the hosts itself? And can see the package arrive on the enp5s0 on host2 and enp5s0 on host1?

compuser

@patient0

I am using proxmox. I am running ubuntu 24.04. I created a bridge in /etc/network/interfaces like this.

on host 1

auto vmbr6
iface vmbr6 inet static
        address 172.16.6.1/24
        bridge-ports enp5s0.4006
        #vlan-raw-device enp5s0
        bridge-stp off
        bridge-fd 0
        mtu 1400

auto enp5s0.4006
iface enp5s0.4006 inet manual
        vlan_raw_device enp5s0
        mtu 1400

and on host 2

auto vmbr6
iface vmbr6 inet static
        address 172.16.6.3/24
        bridge-ports enp5s0.4006
        #vlan-raw-device enp5s0
        bridge-stp off
        bridge-fd 0
        mtu 1400

auto enp5s0.4006
iface enp5s0.4006 inet manual
        vlan_raw_device enp5s0
        mtu 1400

No I dont have any firewall rules on the hosts. Just the robot server rules. The VMs are stock cloud images from ubuntu that I spun up just for this testing. I havent enabled ufw or anything else on them.

compuser

Another datapoint. dhcp server is reachable from host1 but not from host2

host2 ~ # nmap --script broadcast-dhcp-discover
Starting Nmap 7.93 ( https://nmap.org ) at 2025-06-15 21:32 CEST
WARNING: No targets were specified, so 0 hosts scanned.
Nmap done: 0 IP addresses (0 hosts up) scanned in 10.29 seconds

At the same time I can ping from host1 to bridge in host2
host1 ~ # ping 172.16.6.3
PING 172.16.6.3 (172.16.6.3) 56(84) bytes of data.
64 bytes from 172.16.6.3: icmp_seq=1 ttl=64 time=1.44 ms
64 bytes from 172.16.6.3: icmp_seq=2 ttl=64 time=0.718 ms
64 bytes from 172.16.6.3: icmp_seq=3 ttl=64 time=0.696 ms
^C
--- 172.16.6.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2033ms
rtt min/avg/max/mdev = 0.696/0.951/1.439/0.345 ms

and host2 is able to ping bridge in host1
host2 ~ # ping 172.16.6.1
PING 172.16.6.1 (172.16.6.1) 56(84) bytes of data.
64 bytes from 172.16.6.1: icmp_seq=1 ttl=64 time=0.350 ms
64 bytes from 172.16.6.1: icmp_seq=2 ttl=64 time=0.630 ms
64 bytes from 172.16.6.1: icmp_seq=3 ttl=64 time=0.413 ms
64 bytes from 172.16.6.1: icmp_seq=4 ttl=64 time=0.648 ms
^C
--- 172.16.6.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3096ms
rtt min/avg/max/mdev = 0.350/0.510/0.648/0.130 ms

Any ideas on how I can troubleshoot this?

compuser

@patient0 Turns out I need to add a firewall rule on proxmox1 that allows udp requests from 0.0.0.0/32 to 255.255.255.255/32. Now VM2 is able to get the dhcp lease. But is this safe? What will prevent anyone outside from reaching my dhcp server.

patient0

@compuser

@compuser said in communicating via vswitch from vms in bridges:

Turns out I need to add a firewall rule on proxmox1 that allows udp requests from 0.0.0.0/32 to 255.255.255.255/32

That is news to me, never seen these two. 0.0.0.0/32 matches only 0.0.0.0 but that is not a thing (I thought).
Edit: Unless that's another way to write 0.0.0.0, which for services and apps means listen to every interface.

But: does that mean you have firewall rules on Proxmox (see my earlier question)?

In general: if your hosts are accessible from the outside, you will need firewall rule.

compuser

@patient0 Sorry. I used the wrong terminology. I had to apply the firewall rule on hetzner firewall for host 1 to allow the broadcast traffic for dhcp request from vm on host2. I am not using the firewall within proxmox app. I am using firewall rules in pfsense which is my dhcp server. Does this answer your question?

Regarding the IP address itself, the src and destination matches with what you would see on the packet capture though.

However, I am noticing that the problem is not entirely resolved. The VM is getting its lease but for some reason the dns is not getting registered in pfsense. VM1 on host1 has its lease and dns registered automatically. I can also do a nslookup from pfsense diagnostics to retrieve the IP for VM1 but I cant do the same for VM2. Any idea on what I might be missing. From host2, I can see that port 53 is open too. What could be preventing pfsense from registering the dns of VM2 even though it is leasing the IP.

nmap -sU 172.16.6.2
Starting Nmap 7.93 ( https://nmap.org ) at 2025-06-16 07:22 CEST
Nmap scan report for 172.16.6.2
Host is up (0.00066s latency).
Not shown: 998 open|filtered udp ports (no-response)
PORT    STATE SERVICE
53/udp  open  domain
123/udp open  ntp
MAC Address: BC:24:11:61:53:E6 (Unknown)

Thanks a bunch for your time!

patient0

@compuser registering the client is thing between the DHCP server and the unbound server on pfSense.

Do you see the DHCP lease for VMs on host two on the pfSense? And are the VMs on host2 using pfSense for DNS queries?

compuser

@patient0
Yes, I can see the DHCP lease but the DNS lookup fails. The host is using the dns server correctly. Not sure what prevented the dns registration in pfsense.

https://imgur.com/a/fegBppv

compuser

Hi @patient0 . would you have any thoughts on why the dns registration might not be happening. Shouldnt pfsense be doing it behind the scenes automatically. Does it do any verification that might be failing. I did try opening up the hetzner firewall on host2, hoping that any host verification might be getting blocked, and that didnt help either.

patient0

@compuser if host test can't even be resolved on pfSense then it won't work on the client.

Which pfSense version and which DHCP server are you using?

https://imgur.com/a/fegBppv

Can you insert picture directly?

compuser

I got it to work. It had to do with not setting mtu of 1400. I can now do dns lookup and it works! Thank you for your suggestions.