DHCP Server provider the same IP for two different VMs

aloisiobilck · Oct 10, 2023, 2:59 PM

What is happening is the following:
I clone two VMs.

VM1 starts and gets an IP
VM2 starts and gets the same IP as VM1.
If I go through the VM2 terminal and give a dhclient <interface> it gets a new IP.

johnpoz · Oct 10, 2023, 3:04 PM

@aloisiobilck I would think that duplicate ip detection would prevent that.. Did you disable that on pfsense dhcp server?

The client itself should also be doing duplication checks as well.. What specific client OS is this VM?

Is VM1 still on when you start up VM2?

aloisiobilck · Oct 10, 2023, 4:08 PM

@johnpoz I will install tcpdump to see the DHCP packets.

SteveITS · Oct 10, 2023, 4:11 PM

@aloisiobilck no need:
https://docs.netgate.com/pfsense/en/latest/diagnostics/packetcapture/index.html

aloisiobilck · Oct 10, 2023, 4:30 PM

@johnpoz
I didn't disable it, it's the way you sent the image.
VM1 is up

aloisiobilck · Oct 10, 2023, 4:33 PM

@SteveITS
I was talking about the Linux VM, sorry. In Pfsense I am aware that I must use "Packet Capture"

Thanks

aloisiobilck · Oct 10, 2023, 4:36 PM

@johnpoz I'll test it with "Disable ping check"

SteveITS · Oct 10, 2023, 4:42 PM

@aloisiobilck said in DHCP Server provider the same IP for two different VMs:

@johnpoz I'll test it with "Disable ping check"

You want the ping check enabled. That tells pfSense to not assign an IP if anything responds to a ping. Does the firewall on these VMs allow ping?

johnpoz · Oct 10, 2023, 5:01 PM

@aloisiobilck where did I say to do that?

Even if this first vm doesn't answer ping, the arp probe that the 2nd vm sends out or should send out would be answered if its online.

Is the first vm still up when you turn on the 2nd vm??

What OS are you running on these VMs.. These are basic questions, that you didn't answer but jump straight to disabling the very thing that would help prevent duplicate IPs..

There are a billion flavors of linux - they can all have different dhcp clients, etc.. What distro are you running, so can look - and or even try duplicate exactly what your doing with the same OS..

aloisiobilck · Oct 10, 2023, 6:17 PM

@johnpoz

answering your questions:

OS is Ubuntu 22.04
VM1 is UP (running) when I turn on VM2.
"ping check" is enabled. I misunderstood, sorry!

johnpoz · Oct 10, 2023, 9:39 PM

@aloisiobilck so I have a ubuntu vm 22.04.. Let me try and duplicate your issue.

edit: take that back my clean vm I use for new vms is only 20.04, grabbing 22.04.3 now.. server - don't want/need desktop and will install it clean.. And then clone it and see what happens.

edit2: ok I have duplicated your problem..

I created a vm, ubuntu 22.04.3 LTS, min server install. updated it.. And installed the net-tools so I could use ifconfig.. Rebooted it, etc. Then shut it down.. Cloned it..

And yup duplicate IP.. 192.168.3.117.. This shouldn't happen I agree with you.. It is a different mac, and the duplication systems in both dhcpd and the dhcp client should prevent this.

Now I did start a packet capture before I booted the first vm, and left it running while booted the clone.. Let me look at the capture and see what is not happening.. The client for sure should of done an arp probe and not agreed to the IP that was in use..

edit3: ok yeah this weird.. So the clients are never sending an arp probe - which they should.. And I only see a ping test from pfsense once, when the first machine boots.. I don't see another one when the 2nd vm asks.. But there is no answer to the first one..

But since the client is a clone - it does have a duplicate client ID.. Hmmm

Going to have to look at this for a bit.. Going to look first to why ubuntu dhcp client isn't sending an arp probe before it accepts that IP..

edit4: Ok figured out what is going on.. stupid if you ask me.. No wonder I always have disabled netplan in the past.. Seems they use the duid and IAID to create the client identifier since prob back in version 18.. And since you cloned - its the same.. I changed my 2nd vm to use mac..

And now it gets a different IP.. .118

So yeah seems cloning could cause what your seeing for sure.. But there is a quick work around for you.. There prob a way to have it generate a different identifier when you clone the vm.. And want to look into having it send the arp probe as well.. I was thought from the rfc it was a "must" that client send arp probe for duplicate IP detection.. But seems its really only a "should" and ubuntu figures they shouldn't I guess ;)

The docs for netplan can be viewed here..
https://netplan.readthedocs.io/en/latest/netplan-yaml/

Not sure if that is the official place - but was first place I found that had the info about netplan I was looking for.

Glad you mentioned that you using 22.04, since then I was able to duplicate.. My linux vm isn't using netplan :) so it sends a mac as the identifier..

edit5: another way to fix it vs switching to mac, is change the machine-id.. You should be able to do that with deleting /etc/machine-id and then running systemd-machine-id-setup

Now the client ID sent via dhcp discover should be different than your other VM.. And shouldn't get a dupe.

edit6: just to validate.. I removed the mac setting from the yaml file from above.. And after I did that change in the machine-id, it did send a different identifier and did get a different IP as well..

This was fun! Still not a fan of netplan - but learned more about it.. I guess just old school ;)

aloisiobilck · Oct 11, 2023, 11:15 AM

Hi,
Packet capture in Pfsense. I started VM1 (00:50:56:89:41:b9) and then started VM2 (00:50:56:89:b0:76)

11:47:37.345649 00:50:56:89:41:b9 (oui Unknown) > Broadcast, ethertype IPv4 (0x0800), length 334: (tos 0xc0, ttl 64, id 0, offset 0, flags [none], proto UDP (17), length 320)
    0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 00:50:56:89:41:b9 (oui Unknown), length 292, xid 0x750b3c38, secs 1, Flags [none] (0x0000)
	  Client-Ethernet-Address 00:50:56:89:41:b9 (oui Unknown)
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Discover
	    Client-ID Option 61, length 19: hardware-type 255, 9f:6e:85:24:00:02:00:00:ab:11:20:ae:1c:a4:98:d7:de:c7
	    Parameter-Request Option 55, length 11: 
	      Subnet-Mask, Default-Gateway, Domain-Name-Server, Hostname
	      Domain-Name, MTU, Static-Route, NTP
	      Option 119, Option 120, Classless-Static-Route
	    MSZ Option 57, length 2: 576
	    Hostname Option 12, length 8: "z5zblig6"
11:47:37.345866 90:ec:77:74:ec:3f (oui Unknown) > 00:50:56:89:41:b9 (oui Unknown), ethertype IPv4 (0x0800), length 342: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    10.31.0.1.bootps > 10.31.1.121.bootpc: [udp sum ok] BOOTP/DHCP, Reply, length 300, xid 0x750b3c38, secs 1, Flags [none] (0x0000)
	  Your-IP 10.31.1.121
	  Client-Ethernet-Address 00:50:56:89:41:b9 (oui Unknown)
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Offer
	    Server-ID Option 54, length 4: 10.31.0.1
	    Lease-Time Option 51, length 4: 604800
	    Subnet-Mask Option 1, length 4: 255.255.254.0
	    Default-Gateway Option 3, length 4: 10.31.0.1
	    Domain-Name-Server Option 6, length 4: 10.2.0.2
	    Domain-Name Option 15, length 24: "mycompany.internal"
11:47:37.350266 00:50:56:89:41:b9 (oui Unknown) > Broadcast, ethertype IPv4 (0x0800), length 346: (tos 0xc0, ttl 64, id 0, offset 0, flags [none], proto UDP (17), length 332)
    0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 00:50:56:89:41:b9 (oui Unknown), length 304, xid 0x750b3c38, secs 1, Flags [none] (0x0000)
	  Client-Ethernet-Address 00:50:56:89:41:b9 (oui Unknown)
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Request
	    Client-ID Option 61, length 19: hardware-type 255, 9f:6e:85:24:00:02:00:00:ab:11:20:ae:1c:a4:98:d7:de:c7
	    Parameter-Request Option 55, length 11: 
	      Subnet-Mask, Default-Gateway, Domain-Name-Server, Hostname
	      Domain-Name, MTU, Static-Route, NTP
	      Option 119, Option 120, Classless-Static-Route
	    MSZ Option 57, length 2: 576
	    Server-ID Option 54, length 4: 10.31.0.1
	    Requested-IP Option 50, length 4: 10.31.1.121
	    Hostname Option 12, length 8: "z5zblig6"
11:47:37.351147 90:ec:77:74:ec:3f (oui Unknown) > 00:50:56:89:41:b9 (oui Unknown), ethertype IPv4 (0x0800), length 342: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    10.31.0.1.bootps > 10.31.1.121.bootpc: [udp sum ok] BOOTP/DHCP, Reply, length 300, xid 0x750b3c38, secs 1, Flags [none] (0x0000)
	  Your-IP 10.31.1.121
	  Client-Ethernet-Address 00:50:56:89:41:b9 (oui Unknown)
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: ACK
	    Server-ID Option 54, length 4: 10.31.0.1
	    Lease-Time Option 51, length 4: 604800
	    Subnet-Mask Option 1, length 4: 255.255.254.0
	    Default-Gateway Option 3, length 4: 10.31.0.1
	    Domain-Name-Server Option 6, length 4: 10.2.0.2
	    Domain-Name Option 15, length 24: "mycompany.internal"
11:47:43.750797 00:50:56:89:b0:76 (oui Unknown) > Broadcast, ethertype IPv4 (0x0800), length 334: (tos 0xc0, ttl 64, id 0, offset 0, flags [none], proto UDP (17), length 320)
    0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 00:50:56:89:b0:76 (oui Unknown), length 292, xid 0x2dfaf96c, secs 1, Flags [none] (0x0000)
	  Client-Ethernet-Address 00:50:56:89:b0:76 (oui Unknown)
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Discover
	    Client-ID Option 61, length 19: hardware-type 255, 9f:6e:85:24:00:02:00:00:ab:11:20:ae:1c:a4:98:d7:de:c7
	    Parameter-Request Option 55, length 11: 
	      Subnet-Mask, Default-Gateway, Domain-Name-Server, Hostname
	      Domain-Name, MTU, Static-Route, NTP
	      Option 119, Option 120, Classless-Static-Route
	    MSZ Option 57, length 2: 576
	    Hostname Option 12, length 8: "7o1umnqw"
11:47:43.750992 90:ec:77:74:ec:3f (oui Unknown) > 00:50:56:89:b0:76 (oui Unknown), ethertype IPv4 (0x0800), length 342: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    10.31.0.1.bootps > 10.31.1.121.bootpc: [udp sum ok] BOOTP/DHCP, Reply, length 300, xid 0x2dfaf96c, secs 1, Flags [none] (0x0000)
	  Your-IP 10.31.1.121
	  Client-Ethernet-Address 00:50:56:89:b0:76 (oui Unknown)
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Offer
	    Server-ID Option 54, length 4: 10.31.0.1
	    Lease-Time Option 51, length 4: 604800
	    Subnet-Mask Option 1, length 4: 255.255.254.0
	    Default-Gateway Option 3, length 4: 10.31.0.1
	    Domain-Name-Server Option 6, length 4: 10.2.0.2
	    Domain-Name Option 15, length 24: "mycompany.internal"
11:47:43.751674 00:50:56:89:b0:76 (oui Unknown) > Broadcast, ethertype IPv4 (0x0800), length 346: (tos 0xc0, ttl 64, id 0, offset 0, flags [none], proto UDP (17), length 332)
    0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 00:50:56:89:b0:76 (oui Unknown), length 304, xid 0x2dfaf96c, secs 1, Flags [none] (0x0000)
	  Client-Ethernet-Address 00:50:56:89:b0:76 (oui Unknown)
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Request
	    Client-ID Option 61, length 19: hardware-type 255, 9f:6e:85:24:00:02:00:00:ab:11:20:ae:1c:a4:98:d7:de:c7
	    Parameter-Request Option 55, length 11: 
	      Subnet-Mask, Default-Gateway, Domain-Name-Server, Hostname
	      Domain-Name, MTU, Static-Route, NTP
	      Option 119, Option 120, Classless-Static-Route
	    MSZ Option 57, length 2: 576
	    Server-ID Option 54, length 4: 10.31.0.1
	    Requested-IP Option 50, length 4: 10.31.1.121
	    Hostname Option 12, length 8: "7o1umnqw"
11:47:43.752086 90:ec:77:74:ec:3f (oui Unknown) > 00:50:56:89:b0:76 (oui Unknown), ethertype IPv4 (0x0800), length 342: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    10.31.0.1.bootps > 10.31.1.121.bootpc: [udp sum ok] BOOTP/DHCP, Reply, length 300, xid 0x2dfaf96c, secs 1, Flags [none] (0x0000)
	  Your-IP 10.31.1.121
	  Client-Ethernet-Address 00:50:56:89:b0:76 (oui Unknown)
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: ACK
	    Server-ID Option 54, length 4: 10.31.0.1
	    Lease-Time Option 51, length 4: 604800
	    Subnet-Mask Option 1, length 4: 255.255.254.0
	    Default-Gateway Option 3, length 4: 10.31.0.1
	    Domain-Name-Server Option 6, length 4: 10.2.0.2
	    Domain-Name Option 15, length 24: "mycompany.internal"

cat /etc/netplan/00-installer-config.yaml

# This is the network config written by 'subiquity'
network:
  ethernets:
    ens160:
      dhcp4: true
  version: 2

johnpoz · Oct 11, 2023, 11:16 AM

@aloisiobilck change your yaml file like I posted and it will use mac, and the issue will go away. Or change your machine-id file so the client id is different.

aloisiobilck · Oct 11, 2023, 12:11 PM

@johnpoz
Thank you very much, it is now working!
I don't believe the problem was with the Ubuntu netplan

johnpoz · Oct 11, 2023, 12:36 PM

@aloisiobilck the problem is you cloned it and the client ID being sent to the dhcp server is the same..

If netplan would use mac vs client ID, you would of never seen the issue. Or if netplan would use duplicate IP detection, ie arp probe before using an ip offered by a dhcpd you wouldn't of seen the issue.

This has been a known issue for some time if you google duplicate IP vm clone, etc. After I re-invented the wheel it seems by looking at the captures and what exactly what was going on. I started running into lots of threads about cloned vms and duplicated IP.. Solution given was either my yaml edit or the machine id change..

The dhcp server is not to blame - because the identifier sent matches an IP already given out, so sure it would send that back - hey guy I know you, here is the IP you had last time, etc.

Why go to client ID vs mac - not sure why netplan using that.. Why no arp probe for duplicate detection, not sure - but detection can slow down acquisition of IP from dhcp..

Depending on your vm software and how your creating your copy/new/clone vm - there can be ways you can setup in that vm software to generate different machine id when the vm is created.

I can not really think of anything could do on pfsense in preventing such a scenario.. Per the client ID sent, it was the same box - so yeah going to send the same IP.. Now maybe there is something in the dhcpd software that could check.. Hey wait this client ID is the same but the mac is different. But off the top I am not aware of any dhcpd that has such an option. Then again haven't looked too hard for such an option..

I do remember way back in the day when disk duplication was new, and cloning disks for windows.. Would need to generate a new guid in windows after you deployed the new disk.. Or all kinds of weird stuff could happen. I don't recall ever seeing duplicate IP issues from dhcp.. But that was using mac, and windows machine send out the arp probe for duplicate detection, etc. But other odd stuff with the AD, and permissions etc would come up if you didn't generate the new guid. If I recall mind you this like 30 years ago or something that when we would join the clone disks to the AD it would generate new guid. But if you cloned a machine that was already in the domain, you had all kinds of problems.. But again that was many many years ago.. So bit hazy on all the details.