DHCP weirdness after 23.09 upgrade

[mathiasringhof]

Looks like we found something, see the Redmine issue. The binary shipped in 23.09 causes an issue. Running assumption is that it responds on random ports to DHCPREQUEST which seems to cause issues with some clients if they reject anything not coming from port 67.

Replacing the binary with the one from 23.05.1 resolved the issue for me.

[johnpoz]

@mathiasringhof yeah just did a quick sniff.. and see this

15:39:55.151294 IP 192.168.4.81.68 > 192.168.4.253.67: UDP, length 304
15:39:55.151592 IP 192.168.4.253.18184 > 192.168.4.81.68: UDP, length 300

That doesn't seem right, the reply should be from 67.. not some random port 18184..

The client sure doesn't seem to like it

15:39:55.151294 IP 192.168.4.81.68 > 192.168.4.253.67: UDP, length 304
15:39:55.151592 IP 192.168.4.253.18184 > 192.168.4.81.68: UDP, length 300
15:40:58.682252 IP 192.168.4.81.68 > 192.168.4.253.67: UDP, length 304
15:40:58.682557 IP 192.168.4.253.18184 > 192.168.4.81.68: UDP, length 300
15:41:04.499847 IP 192.168.4.65.68 > 192.168.4.253.67: UDP, length 308
15:41:04.500249 IP 192.168.4.253.18184 > 192.168.4.65.68: UDP, length 300
15:42:03.221195 IP 192.168.4.81.68 > 192.168.4.253.67: UDP, length 304
15:42:03.221489 IP 192.168.4.253.18184 > 192.168.4.81.68: UDP, length 300
15:42:04.507527 IP 192.168.4.65.68 > 192.168.4.253.67: UDP, length 308
15:42:04.507775 IP 192.168.4.253.18184 > 192.168.4.65.68: UDP, length 300

[SteveITS]

Tagging @stephenw10.

The extra logging isn’t really a breaking change but clients not getting a lease is.

Most people, if they even notice this, can use Kea, but not everyone since it’s not feature complete/ready yet.

[johnpoz]

@SteveITS I just switched to binary from 23.05.1 just waiting for something to renew.. But log already looks better.. 3 clients renewed before I got the sniff going..

2 of the problem childred I was seeing from 4.81 and 4.63 have now renewed it seems

Nov 19 10:12:37 	dhcpd 	36201 	DHCPACK on 192.168.4.63 to 68:57:2d:98:28:22 via igb2.4
Nov 19 10:12:37 	dhcpd 	36201 	DHCPREQUEST for 192.168.4.63 from 68:57:2d:98:28:22 via igb2.4
Nov 19 10:12:03 	dhcpd 	36201 	DHCPACK on 192.168.4.81 to a8:1b:6a:24:ec:26 via igb2.4
Nov 19 10:12:03 	dhcpd 	36201 	DHCPREQUEST for 192.168.4.81 from a8:1b:6a:24:ec:26 via igb2.4 

edit: ok couple of devices renewed while I was sniffing - this looks more normal

16:34:01.236228 IP 192.168.4.73.68 > 192.168.4.253.67: UDP, length 308
16:34:01.236487 IP 192.168.4.253.67 > 192.168.4.73.68: UDP, length 300
16:34:53.173058 IP 192.168.4.49.68 > 192.168.4.253.67: UDP, length 302
16:34:53.173367 IP 192.168.4.253.67 > 192.168.4.49.68: UDP, length 300

[Phizix]

@johnpoz & @mathiasringhof,

How does one replace the binary with the one from 23.05.1?

Phizix

[mathiasringhof]

@Phizix The file is located here:

/usr/local/sbin/dhcpd

Boot into a 23.05.1 environment and backup that file to a local share (e.g. scp or an SFTP client), then boot into 23.09, backup the old file just in case and replace.

You need to have SSH enabled (see here) for that to work.

[Phizix]

@mathiasringhof,

I assume this file is the same across most of the various hardware.

I have a SG-1100 backup that is currently on 23.05.1, If the file is the same I could pull it from it. If it is hardware specific, then I will have to jump back to 23.05.1 on my SG-5100 to pull it.

Do you happen to know?

Phizix

[SteveITS]

@Phizix the 1100 and 2100 and 3100 are Arm CPUs so that won’t work. If you had reinstalled at some point and have ZFS you could revert to an old boot environment temporarily.

[johnpoz]

@Phizix said in DHCP weirdness after 23.09 upgrade:

How does one replace the binary with the one from 23.05.1?

For me I just booted the zfs 23.05.1 on my VM, grabbed the file via sftp.. Then stopped dhcpd on my sg4860, copied the current dhcpd to dhcpd.old and then via sftp put the dhcpd from my vm on my 4860 and then restarted it..

I am going to have to do a bit of research - but I could understand why some clients might not like an answer from some random port when it sent to 67.. Have to fire up the rfc and see if client is just not up to date on what is should do.. It is odd, and if doesn't like it renew why does it take it on discover. Or is the discover coming from 67? Very interesting situation for sure.

edit:
It was actually a nice test of booting to an old zfs.. which worked out great.

[Phizix]

@SteveITS,

My SG-5100 is on ZFS, so I should be able to boot to the previous 23.05.1 environment and grab it.

I did not know if all the files were hardware specific or not.

I don't think it will be too hard, but if it had not been hardware specific then it would have almost been trivial to grab it.

Thanks for the response!

Phizix

[michmoor]

@johnpoz said in DHCP weirdness after 23.09 upgrade:

It was actually a nice test of booting to an old zfs.. which worked out great.

The boot environment feature is so clutch. Just awesome

[beatvjiking]

I'm seeing similar problems while applying "DHCP enforcement" through my wifi infrastructure - the controllers aren't recognizing the DHCP transactions and are kicking clients off even after they receive DHCP leases. Switching to Kea isn't an option for me yet since I'm running redundant pairs. The behavior is reliably reproducible on 23.09 and doesn't exist on 23.05.1.

[Phizix]

I may have to do this soon. It appears my garage door opener is not getting an IP. DOH!!

[beatvjiking]

If anyone has a non-ZFS-based way to patch/work around the issue at scale (not all my affected boxes have been moved to ZFS yet, and there are a lot, so a binary swap would be challenging) I'd be super amped to hear about it.

[mathiasringhof]

@beatvjiking As long as you pay attention to architecture you can just use a random 23.05.1 install, get the file and then deploy to all your boxes. Anything below the 4000 series is ARM though, so be careful not to deploy x86 binaries to those.

[Phizix]

OK, looking back at the Garage Door opener logs, it lost IP long before the 23.09 upgrade.

Looks like I have some troubleshooting to do.

Phizix

[lopezi]

Where can we download 23.05 to get the previous binary?

[SteveITS]

@lopezi If you have ZFS you can revert to an earlier boot environment.
https://docs.netgate.com/pfsense/en/latest/backup/zfsbe/index.html

If not and you have Netgate hardware you can request an old image and install it. (Also gets you ZFS)

If you don’t have Netgate hardware then one way might be to install 2.6, then upgrade to 23.05? If it lets you choose Previous Stable while upgrading…?

[lopezi]

@SteveITS No, not on Netgate hardware but I did upgrade from CE to Plus a few versions back but my install is still using UFS so no ZFS boot for me.

If I go to System -> Update -> Update Settings I can select 23.05.1 but I don't want to revert my whole system. However, if I can get the endpoints that this is pulling from, I could download the package in theory. Anyone know the endpoints?

[dem]

@johnpoz said in DHCP weirdness after 23.09 upgrade:

For me I just booted the zfs 23.05.1 on my VM, grabbed the file via sftp.

You might not need to reboot to fetch a file from a previous ZFS Boot Environment. Try looking for the ZFS snapshot of your previous root in /.zfs/snapshot. It will be named after the date and time it was created.

For example, on my 23.09 system:

ls -l /usr/local/sbin/dhcpd
-r-xr-xr-x  1 root wheel 3084656 Oct  6 00:44 /usr/local/sbin/dhcpd

ls -l /.zfs/snapshot/2023-11-09-11:07:11-0/usr/local/sbin/dhcpd
-r-xr-xr-x  1 root wheel 3111960 Feb 17  2023 /.zfs/snapshot/2023-11-09-11:07:11-0/usr/local/sbin/dhcpd

The /.zfs directory won't appear in a regular ls listing.