Duplicate DHCP leases

swinster · Aug 24, 2022, 6:01 PM

Hi,

I seem to have an odd issue with my DHCP configuration. I have just deployed two identical VMware appliances from Bitnami. VMware has the sense to randomise the MAC address assigned to the vNICs, but both VMs ended up with the same IP address. I can only see one MAC entry on the GUI, but there are multiple lease entries in the DHCP lease file (now 6 entries, although this was 3 when I last looked). The really odd thing is that when looking at the GUI, sometimes the MAC address shows as from VM 1, and other times it shows as from VM 2 (I guess the last one that requested the lease), but they do not both appear at the same time (as obviously, that would likely offer two unique IP addresses). Initially, I only saw one MAC in the lease file, but now both seem to appear.

For example, I have just booted the VMs in order (waiting around 1 minute between each boot), and here is the latest DHCP lease file entries:

lease 172.16.199.17 {
  starts 3 2022/08/24 16:56:11;
  ends 3 2022/08/24 18:56:11;
  cltt 3 2022/08/24 16:56:11;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet 00:0c:29:85:95:6a;
  uid "\377\237n\205$\000\002\000\000\253\021\226\031\230\020\326;\364\205";
  client-hostname "debian";
}

lease 172.16.199.17 {
  starts 3 2022/08/24 16:56:11;
  ends 3 2022/08/24 17:07:18;
  tstp 3 2022/08/24 17:07:18;
  cltt 3 2022/08/24 16:56:11;
  binding state free;
  hardware ethernet 00:0c:29:85:95:6a;
  uid "\377\237n\205$\000\002\000\000\253\021\226\031\230\020\326;\364\205";
}
lease 172.16.199.17 {
  starts 3 2022/08/24 17:08:55;
  ends 3 2022/08/24 19:08:55;
  cltt 3 2022/08/24 17:08:55;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet 00:0c:29:85:95:6a;
  uid "\377\237n\205$\000\002\000\000\253\021\226\031\230\020\326;\364\205";
  client-hostname "debian";
}
lease 172.16.199.17 {
  starts 3 2022/08/24 17:08:55;
  ends 3 2022/08/24 17:10:03;
  tstp 3 2022/08/24 17:10:03;
  cltt 3 2022/08/24 17:08:55;
  binding state free;
  hardware ethernet 00:0c:29:85:95:6a;
  uid "\377\237n\205$\000\002\000\000\253\021\226\031\230\020\326;\364\205";
}
lease 172.16.199.17 {
  starts 3 2022/08/24 17:10:46;
  ends 3 2022/08/24 19:10:46;
  cltt 3 2022/08/24 17:10:46;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet 00:0c:29:77:9e:3a;
  uid "\377\237n\205$\000\002\000\000\253\021\226\031\230\020\326;\364\205";
  client-hostname "debian";
}

lease 172.16.199.17 {
  starts 3 2022/08/24 17:32:59;
  ends 3 2022/08/24 19:32:59;
  cltt 3 2022/08/24 17:32:59;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet 00:0c:29:85:95:6a;
  uid "\377\237n\205$\000\002\000\000\253\021\226\031\230\020\326;\364\205";
  client-hostname "debian";
}

This is an image of the GUI now, showing the MAC of VM2:

However, here is an image of the GUI a short while ago, showing the MAC or VM 1

I guess I could shut down the VMs and manually remove these entries from the lease file, but I'm wondering why this might be occurring.

I ran a PCAP on PfSense and noted that the DHCP discover from each includes their unique MAC address but Pfsense issues the same IP address to both. Here are a couple of images from the same PCAP that show the different DHCP dialogues, just highlighting the different DCP Discover packets from both VMs

In addition to all of this, I noted that there are other multiple active DHCP leases contained in the leases file, but for the same MAC - is this normal? Given these leases are for the same thing, I don't think there are any major issues, but again, it doesn't quite feel right.

For example:

lease 192.168.199.108 {
  starts 3 2022/08/24 16:15:04;
  ends 3 2022/08/24 18:15:04;
  cltt 3 2022/08/24 16:15:04;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet bc:dd:c2:21:db:ac;
  client-hostname "ESP_21DBAC";
}

lease 192.168.199.108 {
  starts 3 2022/08/24 17:04:18;
  ends 3 2022/08/24 19:04:18;
  cltt 3 2022/08/24 17:04:18;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet bc:dd:c2:21:db:ac;
  client-hostname "ESP_21DBAC";
}

Cheers

swinster · Aug 25, 2022, 9:38 AM

FWIW, I left both VMs off overnight just to let the leases expire fully. This morning, the DHCP lease file showed only 1 entry for 172.16.199.17. I decided to remove that entry manually and resave the file then reboot one of the devices.

That device obtained a lease immediately for 172.16.199.17 (as expected) but interestingly in the pfSense GUI, it shows as offline, which it very much isn't.

I then rebooted the second VM, and it immediately was issued the same IP address. Now the GUI has been updated with the MAC address of the second VM and it is now denoted as online.

I am confused.

swinster · Aug 25, 2022, 9:55 AM

And I'm back to multiple entries in the DHCP lease file, two different MACs but the same IP.

lease 172.16.199.17 {
  starts 4 2022/08/25 09:25:22;
  ends 4 2022/08/25 11:25:22;
  cltt 4 2022/08/25 09:25:22;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet 00:0c:29:77:9e:3a;
  uid "\377\237n\205$\000\002\000\000\253\021\226\031\230\020\326;\364\205";
  client-hostname "debian";
}

lease 172.16.199.17 {
  starts 4 2022/08/25 09:33:34;
  ends 4 2022/08/25 11:33:34;
  cltt 4 2022/08/25 09:33:34;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet 00:0c:29:85:95:6a;
  uid "\377\237n\205$\000\002\000\000\253\021\226\031\230\020\326;\364\205";
  client-hostname "debian";
}

swinster · Aug 25, 2022, 9:54 AM

Looks like the UID is the same for both VMs which I guess is somewhat understandable as they are built from the same OVA file. Obviously, the hostname is also identicle as well. Not sure if the UID should somehow be manipulated to be unique when the OVA is deployed though, or if this has an impact on how Pfsense issue a DHCP lease.

swinster · Aug 25, 2022, 11:28 AM

OK, I switched off the VMs, then checked the "Ignore client identifiers" box for the DHCP server for this interface, removed the leases from the DHCP file, and restarted the VMs. Now, the VMs have been assigned unique IP addresses However, I do wonder if this is the way it should be - goese to read https://www.rfc-editor.org/rfc/rfc2131#page-26

TBH, I was unaware that, whatever these UIDs are, they play such a crucial role in DHCP address allocation. I wonder if this is something specific to the way Bitnami create their VMware appliances and if this could be modified by them. I have not seen this with any other VMs created from the same OVA previously.

swinster · Aug 25, 2022, 10:47 AM

So, I guess the last thing to figure out is why there are duplicate DHCP lease entries for other devices, and if these would be harmful or how they can be overcome (if needs be).

swinster · Aug 25, 2022, 12:16 PM

Back to the original issue, I found this. From https://www.rfc-editor.org/rfc/rfc2131#section-2, we see some stuff about client identifiers:

°The 'client identifier' chosen by a
DHCP client MUST be unique to that client within the subnet to which
the client is attached. If the client uses a 'client identifier' in
one message, it MUST use that same identifier in all subsequent
messages, to ensure that all servers correctly identify the client.°(information text)

So, it would appear as if the Bitnami VMs are being a bit naughty here . I also noted https://www.rfc-editor.org/rfc/rfc2131#section-4.2:

°A DHCP server needs to use some unique identifier to associate a
client with its lease. The client MAY choose to explicitly provide
the identifier through the 'client identifier' option. If the client
supplies a 'client identifier', the client MUST use the same 'client
identifier' in all subsequent messages, and the server MUST use that
identifier to identify the client.°(information text)

So, it is entirely optional for a client to send the client identifier, but f they do, the server MUST use it to identify the client.

swinster · Aug 26, 2022, 1:05 PM

This post is deleted!

swinster · Aug 26, 2022, 4:06 PM

So, it turn out that the Bitnami boys and girls had left the file /etc/machine-id baked into the OVA containing a machine ID . They use systemd to manage their networking and by default, systemd and networkd uses a hash of this machine ID to generate a UID that the DHCP client will use (if not otherwise defined) - or so I am lead to believe ;)

Derelict · Aug 28, 2022, 12:26 PM

@swinster said in Duplicate DHCP leases:

That device obtained a lease immediately for 172.16.199.17 (as expected) but interestingly in the pfSense GUI, it shows as offline, which it very much isn't.

That Online/Offline status is determined by whether pfSense itself has an ARP entry for that host. It will only have an ARP entry if it has passed unicast traffic with that host from the firewall itself in the recent past. It is perfectly normal for that to show offline. Ping it from the firewall and it will show online if ARP is resolved.

I believe the more-recent linux DHCP clients are doing more with the machine-id than other clients do. I have seen that behavior before as well.