PSA: If you are using DHCP options with Windows 11 and DHCP/networking ceased to function after upgrading to Windows 11 24H2 ...

[chamilton_ccn]

Check the type in your option configuration.

Backstory: Since it's inception, one of our sites has had the following DHCP configuration:

Note the type parameter. Now regardless of the fact that this configuration is incorrect is irrelevant, it has been working (that is, correctly distributing routing information) for at least a year, right up until machines began automatically updating to Windows 11 24H2.

Suddenly none of the machines at this site that upgraded to 24H2 were able to connect to the Internet. They weren't even getting an IP address -- instead, APIPA addresses were present on the interfaces, indicating that DHCP "wasn't working".

If you Google "24H2 DHCP" or "24H2 no network" you'll get a bunch of results but this discussion on Microsoft's answers forum is the most active and up to date. One commenter was able to resolve the issue on their network by disabling DHCP option 43, but they didn't say much more than that. This, along with some other DHCP-24H2-related posts nudged me in the right direction.

After two days of troubleshooting, I was able to resolve the issue by changing the option type from text to string:

After changing that option, endpoints running 24H2 were able to acquire IP addresses via DHCP once again, and the routing information supplied by option 249 was configured correctly on the endpoint as well. I.e., The issue was totally resolved.

Summary: Microsoft changed something related to DHCP in their 24H2 update and it seems to have caused a lot of problems for a lot of people, albeit for a multitude of different reasons.

I'm sharing this information for anyone else who might encounter this incredibly frustrating problem.

[johnpoz]

@chamilton_ccn option 43 is normally used to let AP know where to find the controller - seems odd that would stop some windows client from getting an address at all. Normally a dhcp server shouldn't even offer such options unless the client specifically asks for it.

Isn't 249 one of the options to push routes to clients, 33, 121 and 249 pretty much do the same thing right.. You are using that in your network, curious how/what for.

Seems odd that any of those options would be your typical/common network - but yeah its not like MS hasn't made changes before that have caused issues before ;)

[chamilton_ccn]

@johnpoz

Isn't 249 one of the options to push routes to clients, 33, 121 and 249 pretty much do the same thing right..

Correct, however 249 is the preferred option.

You are using that in your network, curious how/what for.

Our sites have direct VTI connections to our datacenters. Endpoints (laptops) have "always on VPN" (via OpenVPN) to the same datacenters. OpenVPN seems to like for its routes to take priority over all others. To make matters worse, setting the route priority on the server (e.g. push "route-metric 300") isn't reliably honored by Windows OSs. When a laptop is at one of our sites attempting to connect to resources at one of our datacenters, we'd rather for it to use the "site to site" VPN since the datacenters are only ever one hop away from each office, instead of routing traffic to the OpenVPN server, then to wherever which is always at least two hops. To summarize: At our offices we push routes to our endpoints via option 249, which results in more efficient, consistent, and predictable routing of traffic.

Seems odd that any of those options would be your typical/common network -

Sure, but my point wasn't that this issue is specific to option 249 (or any other option). For example, option 242 is required by certain phone systems -- Avaya is one of them and that's a fairly common vendor of the types of systems that require this option, especially in older networks with older phone systems.

but yeah its not like MS hasn't made changes before that have caused issues before ;)

Yep! And again, if you Google "24H2 network issues" you'll see a ton of results. It seems to be the case that in the 24H2 update, Microsoft has "done something" with DHCP and it's causing trouble for quite a lot of people. My intent with this post was to maybe solve an issue for someone else, or at the least, contribute to their efforts.

[johnpoz]

@chamilton_ccn I just looked on my work laptop - its running 23H2 ;)

Hope you didn't take my post the wrong way - I see nothing wrong with your post bringing attention to a possible issue people could run into.

Was just wanting to have a bit of discussion is all.

For example the option 43 is odd to me since I wouldn't think that option would ever be on a dhcp scope that users machines are on, that sort of option would exist on your infrastructure network, why would windows machine ever even be on it.. Other than maybe some admin laptop or something now and then. Sure that would and could exist on some smb network maybe, but then again that dhcp server really shouldn't hand it out unless the dhcp client actually ask for it. So its just weird that could actually cause an issue with whatever they did with windows..

As to an always on vpn - I worked for a very large msp for many years with many customers and never ran across such a setup. My previous gig was a fortune 100 company and we were not setup like that. You could enable or disable the vpn on your laptop. And while my current gig isn't a huge company - we again are not setup like that, you enable your vpn when you want. We do have multiple DCs and locations across the globe that you can connect to via vpn. But systems are not set to auto login to the vpn, and sure doesn't limit you to which one you connect too, etc.

I could see such a setup having its own set of problems, maybe advantages but having a hard time of thinking of any off the top of my head ;) Other than a setup where your users don't have to think at all - it should just work and the laptop should have access to company resources all the time. But sure it could lead to odd routing at times. If the user was on a company site would seem odd that their machine should connect to the company vpn unless they were say on a locations guest wifi network or something.

Not saying they are not correct sort of setups that shouldn't work, etc. Just seem like one offs to me with some 30+ years in the business is all.. So yeah it is curious what MS might have done - but I could see maybe these 2 scenarios got overlooked somehow??

I have not been in that space for many moons, but next time talking to one our server/client guys that are responsible for the deployment of OS to the users devices and they run the dhcp servers etc.. I will ask them if they have seen any of this. Or if they are even thinking of pushing 24h2 any time soon ;)

The other thing that is curious is to me, other than some smaller smb is why would this sort of problem with 24h2 not be caught in testing before the update is pushed to user machines, etc.

I will for sure have to look into it a bit, just because I am a curious sort of guy ;) Thanks for posting the info.

[chamilton_ccn]

@johnpoz

Hope you didn't take my post the wrong way

Oh not at all!

I see nothing wrong with your post bringing attention to a possible issue people could run into.

Yeah, there's nothing more frustrating than seeing a comment on a discussion thread involving a problem you're experiencing that simply says "Disregard, I fixed it" or "resolved", but no further details so I generally try to share the details about problems I've run into, especially when I see others experiencing the same thing. It makes the Internet a better place!

For example the option 43 is odd to me since I wouldn't think that option would ever be on a dhcp scope that users machines are on, that sort of option would exist on your infrastructure network, why would windows machine ever even be on it.. Other than maybe some admin laptop or something now and then. Sure that would and could exist on some smb network maybe, but then again that dhcp server really shouldn't hand it out unless the dhcp client actually ask for it. So its just weird that could actually cause an issue with whatever they did with windows..

So the reason #43 made it into this discussion is due to the top comment on page #2 of the Microsoft answers thread I posted earlier. I'm mostly in agreement that #43 has no business on a client access network. I say mostly because only Siths deal in absolutes.

As to an always on vpn - I worked for a very large msp for many years with many customers and never ran across such a setup. My previous gig was a fortune 100 company and we were not setup like that. You could enable or disable the vpn on your laptop. And while my current gig isn't a huge company - we again are not setup like that, you enable your vpn when you want. We do have multiple DCs and locations across the globe that you can connect to via vpn. But systems are not set to auto login to the vpn, and sure doesn't limit you to which one you connect too, etc.

Yeah, I mean... it's definitely not common, but it's perfect for our environment.

I could see such a setup having its own set of problems, maybe advantages but having a hard time of thinking of any off the top of my head ;) Other than a setup where your users don't have to think at all - it should just work and the laptop should have access to company resources all the time. But sure it could lead to odd routing at times. If the user was on a company site would seem odd that their machine should connect to the company vpn unless they were say on a locations guest wifi network or something.

The main issue we've seen is inefficient routing, which is solved by telling our endpoints "When you're at an office, use the office's gateway as your route to the various datacenters instead of your AAVPN" ... because again, when all traffic destined for one of our datacenters traverses the AAVPN, that's not efficient when you've got a perfectly good (and shorter) path to the datacenter if your traffic goes through the VTI insead. Unfortunately I can't post a diagram because I don't have a sufficiently sanitized one; if I have some time I'll post one because I think using option 249 to resolve this is actually an underrated, elegant solution to the problem.

Not saying they are not correct sort of setups that shouldn't work, etc. Just seem like one offs to me

Yeah, I mean... The option exists for the very reason we're using it . And isn't that part of the beauty of what we do? That most of the time, someone found themselves in the exact same situation and created a solution that was good enough to become part of the protocol/spec/RFC/whatever, in turn, ensuring future humans in the same situation already have a path forward? And conversely, if you find yourself in a situation that nobody else has ever encountered, then either your approach to the problem you're trying to solve is incorrect or you've been presented with the opportunity to be that person who creates a solution that becomes a standard. We all stand on the shoulders of giants.

with some 30+ years in the business is all..

I've been in the industry since '98 so I'm not far behind ya.

So yeah it is curious what MS might have done - but I could see maybe these 2 scenarios got overlooked somehow??

Well, text is the wrong type for this particular DHCP option, so I'm inclined to believe that whatever change they made fixed or tightened up something in their OS that worked but wasn't technically correct (which, as we all know, is the best kind of correct).

I have not been in that space for many moons, but next time talking to one our server/client guys that are responsible for the deployment of OS to the users devices and they run the dhcp servers etc.. I will ask them if they have seen any of this. Or if they are even thinking of pushing 24h2 any time soon ;)

It might be worth the conversation. ¯\_(ツ)_/¯

The other thing that is curious is to me, other than some smaller smb is why would this sort of problem with 24h2 not be caught in testing before the update is pushed to user machines, etc.

Ya know, that's not an uncommon question -- in fact, you can always find some population of users asking it once a month, right after every Patch Tuesday. I've admin'd Windows for a long time and I can't recall a time when routine updates were so reliably broken. In this case however, I'm not pointing the finger because obviously the problem was on my end, but I can point to several incredibly frustrating situations that have occurred over the last few years that weren't.

I will for sure have to look into it a bit, just because I am a curious sort of guy ;) Thanks for posting the info.

Of course, enjoy your evening, and happy holidays to you and yours!

[johnpoz]

@chamilton_ccn said in PSA: If you are using DHCP options with Windows 11 and DHCP/networking ceased to function after upgrading to Windows 11 24H2 ...:

I've been in the industry since '98 so I'm not far behind ya

I just say now 30+ because I am getting close to 40.. Uggh getting old and don't always want to admit it to myself.. But retirement is that light I can start to see at the end of the tunnel.

I got a few gray years as well where I wasn't actually in the IT dept, but was doing lots of IT work but it wasn't my official job.. But those gray years end in 92 when I officially moved over to the IT dept.. But if I think back, I redid a db for the security office while I was doing temp duty in the navy awaiting orders back in 85.. So yeah man its pushing 40.. If I recall it was on old Zenith system ;)

You have a great holiday as well..

[Gertjan]

@johnpoz said in PSA: If you are using DHCP options with Windows 11 and DHCP/networking ceased to function after upgrading to Windows 11 24H2 ...:

For example the option 43 is odd to me since I wouldn't think that option would ever be on a dhcp scope that users machines are on

Then same '43' that Unifi devices can use so they can find the controller IP ? See here.
I know, a DNS method can also be used.

If a DHCP server has this DHCP option, or any option actually, it will only add it in the lease if the DHCP client was asking for it.

Or Am I mistaken somewhere ?

[johnpoz]

@Gertjan yeah that is what is weird.. Hope I can find some RCA on the problem at some point because it makes no sense.

Why would having an option that shouldn't wouldn't even be handed to the client cause an issue?

Ah.. ok that makes more sense now - just did a sniff to verify - seems windows does ask for those

So if there was something different with how those are handed out be it text/string or whatever that the client expects - then yeah could cause an issue..

But if they are not there - then even when the client asks for them, they are not handed out - they couldn't be because there is no info to send.

So that makes sense then, but still think that those options really shouldn't even be available in your common network. I mean why would I use 43 to send out the location of some wifi controller on my user network where there are no AP that would need that. But 43 guess could be sending out more than that info - and AP would just use it to find controller. But 249 typically wouldn't be even set I wouldn't think - not unless you were doing something like what @chamilton_ccn is doing.