DHCP client unable to get lease from cable provider [solved]

itpp21 · Mar 10, 2021, 6:27 PM

@melslenstra Only the renew request, it does not do discover ever unless it times out with the renew or dhclient is restarted.
I did end up here after massive amounts of searches which led to no solution as the problem is inside dhclient, your patching approach looked exactly like the behavior I am seeing (ignoring settings) where I have chosen a script to solve this.
This thread may not look the same but it sounds the same (it sounds like a duck ) with the core reason&effects.

tsimmons · Mar 10, 2021, 6:47 PM

I have subscribed to this same thread for years because I had very similar issues where DHCP client requests from my pfsense box didn't work and a client request from my Mac did; I have also seen upstream dhcp server addresses in the 10.x.x.x range; I did some pretty exhaustive research into the DHCP options but soon discovered there wasn't any effective way to modify them in pfsense so I ended up going with a big hammer: requested a static IP from my ISP and just hard-coded it in pfsense.

This is what I saw in the two requests:

PFSENSE DCHP options (didn't work):
	bootp: 0x0000
	option 12: hostname (pf_name)
	option 50: ip (192.168.100.10)
	-option 51: ** NOT SET, MISSING **
	option 53: type (discover)
	option 55: subnet_mask, broadcast, time, classless, router, domain_name, dns, hostname, domain_search, mtu
	-option 57: ** NOT SET, MISSING **
	option 61: client identifier (mac_address)

MAC DHCP options (did work):
	bootp: 0x0000
	option 12: hostname (mac_name)
	option 50: ip (192.168.17.100)
	option 51: leasetime (90 days)
	option 53: type (request, 3)
	option 55: subnet_mask, classless, router, dns, domain_name, domain_search, private, ldap, netbios_name, netbios_node
	option 57: dhcp ()
	option 61: client identifier (dhcp_id)

If there were some way to customize the specific DHCP options in pfsense, that would be ideal but it would probably require compiling your own dhclient and I'm running an SG-3100 appliance from NetGate and don't care to modify that much out of official support.

Edit: You can see what the options are from IANA.

itpp21 · Mar 10, 2021, 7:05 PM

@tsimmons This is what I've seen in hundreds of posts over the past 5 years, including options set not being honored.
I've got a sg-5100 but this issue has been around for other pfSense based systems as well.
I don't mind changing things to solve issues but this one being around so long without any solution is a bit strange.

This may have started as a problem here when my ISP decided to handle routed and bridge mode from the same dhcp server as their modems first boot fully into routed mode and then switch to bridge (which makes sense of what I'm seeing) but then again I have been unable to convince them of this 'mistake' and pfSense can't handle this either as in bridge mode its getting invalid information.

Gertjan · Mar 11, 2021, 7:15 AM

@itpp21 said in DHCP client unable to get lease from cable provider [solved]:

as their modems first boot fully into routed mode and then switch to bridge (.... but then again I have been unable to convince them of this 'mistake' and pfSense can't handle this either as in bridge mode its getting invalid information.

What pfSense can do : with these two, you can instruct the DHCP client to wait after a (WAN) interface LINK UP, which happens when the modem reboots/reset/powers on. Now, the dhcp client wait "for the dust to settle" before it starts discovering.

@tsimmons said in DHCP client unable to get lease from cable provider [solved]:

from IANA.

Could find "dhcp-server-identifier" or "server-identifier" on that list. It's "54" ?

@melslenstra said in DHCP client unable to get lease from cable provider [solved]:

DHCP RENEW and/or DISCOVER packets going out to the wrong / non-existent IP address specifically (in unicast)?

I should have checked it, by now, but didi't : I really presume that DISCOVER will broadcast
Subsequent RENEWs will send direcly to "server-identifier" if it's known.

Btw, @itpp21 :

@itpp21 said in DHCP client unable to get lease from cable provider [solved]:

option dhcp-server-identifier 10.160.92.1; This is not a dhcp server! but pfSense keeps trying this one for hours
option dhcp-server-identifier 10.160.92.1; This is not a dhcp server! but pfSense keeps trying this one for hours
option tftp-server-name "212.142.63.132";

If the "dhcp-server-identifie" is set, the DHCP client should try that IP.
The tftp-server-name seems to be set correctly.

I put my money on "the dhcp server guy @ the ISP made an error".

@tsimmons said in DHCP client unable to get lease from cable provider [solved]:

If there were some way to customize the specific DHCP options in pfsense

Yes, as mentioned already above : build your own dhcp-client config file from scratch, and use that one.
Use the man page of that file to see all the commands.
BUT : there is no command option to discard "options" send by the DHCP server, as they are mandatory. And when they are wrong, they have to be corrected on that side.

itpp21 · Mar 11, 2021, 8:19 AM

@gertjan said in DHCP client unable to get lease from cable provider [solved]:

@itpp21 said in DHCP client unable to get lease from cable provider [solved]:

as their modems first boot fully into routed mode and then switch to bridge (.... but then again I have been unable to convince them of this 'mistake' and pfSense can't handle this either as in bridge mode its getting invalid information.

What pfSense can do : with these two, you can instruct the DHCP client to wait after a (WAN) interface LINK UP, which happens when the modem reboots/reset/powers on. Now, the dhcp client wait "for the dust to settle" before it starts discovering.

Your missing an important point, dhclient suffers from memory loss.
If it starts it works by design, once its running it does not work by design, an obvious hint at this can be found here;

Press save on Wan config page,
'interfaces.inc':
/* Make sure dhclient is not running */
kill_dhclient_process($realif);

/* fire up dhclient */

.......

I appreciate your suggestions but for years this is not doing anything for anyone if used from a running dhclient as it does not honor any setting until it restarts (and then forgets everything) and the whole shebang starts again, which basically proves the point of having to use my script on a timer.

Gertjan · Mar 11, 2021, 8:45 AM

@itpp21 said in DHCP client unable to get lease from cable provider [solved]:

Your missing an important point, dhclient suffers from memory loss.

Received data is kept in it's working memory -
and stored in /var/db/dhclient.leases.{interface}

When the process restarted, because of a IF DOWN / UP, it restarts, parses the files, and knows as much as before.
And because it doesn't loses it's memory, you have to wipe its memory by killing the process.
killing the file
Restart.

Because it doesn't loose its 'memory', it (dhcpclient) will wake up if it was restarted, knows from FreeBSD ( pfSense) what the WAN interface is, finds the file /var/db/dhclient.leases.{interface}, sees the {interface}, sees that the IP is the one in the file and that that IP is the WAN IP, concludes that is is a DHCP type of IP, knows when to start RENEW as the time info is also stored in the file, and - the issue : knows that the DHCP server said "option dhcp-server-identifier 10.160.92.1; " as it has received that data, so it addresses the RENEW to "10.160.92.1" as it was told to do.

Thats why your killing scripts works.

I might be missing something, please detail, as I miss to understand what I miss ;)

itpp21 · Mar 11, 2021, 9:43 AM

Received data is kept in it's working memory -
and stored in /var/db/dhclient.leases.{interface}

Yes but not settings.

issue : knows that the DHCP server said "option dhcp-server-identifier 10.160.92.1; " as it has received that data, so it addresses the RENEW to "10.160.92.1" as it was told to do.

"as it was told to do" is where it's going wrong.
When you set for example "supersede dhcp-server-identifier 255.255.255.255" either in the config or hardcoded, the running version of dhclient does not ever use this resulting in above behavior polling the wrong server (which you know is happening so you set supersede).
Such values are only honored at starttime, at runtime they are not used.

itpp21 · Mar 11, 2021, 10:51 AM

Some interesting read why my ISP is causing a problem,

https://tools.ietf.org/html/rfc2132
9.7. Server Identifier

https://lists.isc.org/pipermail/dhcp-users/2016-October/020348.html

There appears this space allows 4 addresses (a1-a4) and that including this option is not really recommended and that there are ways to offer different addresses for different subnets.

Gertjan · Mar 11, 2021, 12:16 PM

@itpp21 said in DHCP client unable to get lease from cable provider [solved]:

this space allows 4 addresses (a1-a4)

Yeah, saw that post.
a1, ...a4 where each octets.
So, basicly, an IPv4addres.
Guess there is no such thing as IPv6 when that text was written ^^

cat chello.nl > " The use of the server-identifier statement is not recommended ".
You know it.
I know it.

itpp21 · Mar 11, 2021, 1:55 PM

@gertjan 4 years ago,
https://redmine.pfsense.org/issues/7416
But I can find the same thing beyond 4 years and it looks like its still not implemented... if anyone is willing to supply me a binary build for dhclient in pfSense FreeBSD 11.3-STABLE with this patch I'm willing to pay for it.

In fact what annoys me the most is you have full control on the LAN side but just about nothing on the WAN side for dhcp, a simple --dhcp-server-ignore xyz or --dhcp-server-replace xyz,abc(regex) would give anyone control over what an ISP is doing, a clear google shows ISP's don't always know what their doing and without 3th line access as a customer your f***d.

Gertjan · Mar 11, 2021, 1:55 PM

@itpp21 said in DHCP client unable to get lease from cable provider [solved]:

https://redmine.pfsense.org/issues/7416

The last line might contain a solution.
Check if the used FreeBSD version is the same.
Install the other one.
Retrieve the dhpc client binairies
Copy them over to pfSense......

I can't believe I just wrote that :(

Btw : do not forget the first and best option : Chose an ISP on what they offer.
As far as I know, we are not in some country where the government decides what ISP you should use. A couple of decades ago we earned the right to chose the product we like best.

@itpp21 said in DHCP client unable to get lease from cable provider [solved]:

In fact what annoys me the most is you have full control on the LAN side but just about nothing on the WAN side for dhcp, a simple --dhcp-server-ignore xyz or --dhcp-server-replace xyz,abc(regex) would give anyone control over what an ISP is doing, a clear google shows ISP's don't always know what their doing and without 3th line access as a customer your f***d.

Yep, if the admin of the server f*cks up, the client has no choice but leaving.
That goes for every server-client concept.

itpp21 · Mar 11, 2021, 3:29 PM

I can't believe I just wrote that :(

Hahahaha, well I got and placed dhclient_opnsense_19.1, I'll test it tomorrow to see what happens. I've grown up with hex-editing or swapping binaries to solve issues (pre 1980).

Btw : do not forget the first and best option : Chose an ISP on what they offer.

Not much to choose here, ziggo ain't bad (business user) I just don't want to be stuck with a problem I could solve, regardless if this is their mistake I want full control.

Yep, if the admin of the server f*cks up, the client has no choice but leaving.
That goes for every server-client concept.

Unless we give users more control, I am still open for anyone writing better control code for dhclient for a fee as long as I get the binary.

itpp21 · Mar 12, 2021, 8:21 AM

Well I've done it,

Fri Mar 12 08:57:31 2021;192.168.238.1; <30>Mar 12 08:57:31 dhclient_opnsense_19.1[66714]: DHCPDISCOVER on igb0 to 255.255.255.255 port 67 interval 7
Fri Mar 12 08:57:31 2021;192.168.238.1; <30>Mar 12 08:57:31 dhclient_opnsense_19.1[66714]: DHCPOFFER from 212.142.*
Fri Mar 12 08:57:33 2021;192.168.238.1; <30>Mar 12 08:57:33 dhclient_opnsense_19.1[66714]: DHCPREQUEST on igb0 to 255.255.255.255 port 67
Fri Mar 12 08:57:33 2021;192.168.238.1; <30>Mar 12 08:57:33 dhclient_opnsense_19.1[66714]: DHCPACK from 212.142.*
Fri Mar 12 08:57:33 2021;192.168.238.1; <13>Mar 12 08:57:33 dhclient: New IP Address (igb0): 62.194.*
Fri Mar 12 08:57:33 2021;192.168.238.1; <13>Mar 12 08:57:33 dhclient: New Subnet Mask (igb0): 255.255.255.0
Fri Mar 12 08:57:33 2021;192.168.238.1; <13>Mar 12 08:57:33 dhclient: New Broadcast Address (igb0): 255.255.255.255
Fri Mar 12 08:57:33 2021;192.168.238.1; <13>Mar 12 08:57:33 dhclient: New Routers (igb0): 62.194.*
Fri Mar 12 08:57:33 2021;192.168.238.1; <30>Mar 12 08:57:33 dhclient_opnsense_19.1[66714]: bound to 62.194.* -- renewal in 101534 seconds.

This works, so killed the processes, swapped the binaries and restarted the WAN interface, so far it works, lets see in the next days if a renew works as it should respecting the supersede values or not (atm. hardcoded in /etc/inc/interfaces.inc).

Fri Mar 12 09:00:30 2021;192.168.238.1; <13>Mar 12 09:00:30 dhclient: PREINIT
Fri Mar 12 09:00:30 2021;192.168.238.1; <30>Mar 12 09:00:30 dhclient[63500]: DHCPREQUEST on igb0 to 255.255.255.255 port 67
Fri Mar 12 09:00:30 2021;192.168.238.1; <30>Mar 12 09:00:30 dhclient[63500]: DHCPACK from 212.142.*
Fri Mar 12 09:00:30 2021;192.168.238.1; <13>Mar 12 09:00:30 dhclient: REBOOT
Fri Mar 12 09:00:30 2021;192.168.238.1; <13>Mar 12 09:00:30 dhclient: Starting add_new_address()
.......
[2.4.5-RELEASE][xxx@xxx.localdomain]/root: ps -aux|grep dhclient
root 63952 0.0 0.1 533620 2872 - Is 09:00 0:00.00 dhclient: igb0 [priv] (dhclient)
_dhcp 67622 0.0 0.1 533620 2984 - SCs 09:00 0:00.01 dhclient: igb0 (dhclient)
root 45844 0.0 0.1 6560 2336 0 S+ 09:17 0:00.00 grep dhclient
.......
Lets also see if it crashes via its timestamp.

And yep, /var/db/dhclient.leases.igb0 still contains wrong info.

itpp21 · Mar 16, 2021, 1:21 PM

I wasn't expecting a renew this soon but hey its all ok :)

Sat Mar 13 13:11:16 2021;192.168.238.1; <30>Mar 13 13:11:16 dhclient[67622]: DHCPREQUEST on igb0 to 255.255.255.255 port 67
Sat Mar 13 13:11:16 2021;192.168.238.1; <30>Mar 13 13:11:16 dhclient[67622]: DHCPACK from 212.142.*
Sat Mar 13 13:11:16 2021;192.168.238.1; <13>Mar 13 13:11:16 dhclient: RENEW
Sat Mar 13 13:11:16 2021;192.168.238.1; <13>Mar 13 13:11:16 dhclient: Creating resolv.conf
Sat Mar 13 13:11:16 2021;192.168.238.1; <30>Mar 13 13:11:16 dhclient[67622]: bound to 62.194.* -- renewal in 129600 seconds.

[2.4.5-RELEASE][xxxx@xxxx.localdomain]/sbin: ps -aux | grep dhclient
root 63952 0.0 0.1 533620 2872 - Is Fri09 0:00.00 dhclient: igb0 [priv] (dhclient)
_dhcp 67622 0.0 0.1 533620 2984 - SCs Fri09 0:00.15 dhclient: igb0 (dhclient)
root 31399 0.0 0.1 6560 2336 0 S+ 13:39 0:00.00 grep dhclient

No dhclient crashes.

[2.4.5-RELEASE][xxxx@xxxx.localdomain]/sbin: procstat -f 67622
PID COMM FD T V FLAGS REF OFFSET PRO NAME
67622 dhclient text v r r------- - - - /sbin/dhclient
67622 dhclient cwd v d r------- - - - /var/empty
67622 dhclient root v d r------- - - - /var/empty
67622 dhclient jail v d r------- - - - /var/empty
67622 dhclient 0 v c rw------ 6 0 - /dev/null
67622 dhclient 1 v c rw------ 6 0 - /dev/null
67622 dhclient 2 v c rw------ 6 0 - /dev/null
67622 dhclient 3 s - rw------ 2 0 UDD /var/run/logpriv
67622 dhclient 4 v r -w---n-l 2 0 - /var/run/dhclient.igb0.pid
67622 dhclient 6 v c r------- 1 2348 - /dev/bpf
67622 dhclient 7 s - rw------ 1 0 ?
67622 dhclient 8 v r -w------ 1 1893 - /var/db/dhclient.leases.igb0
67622 dhclient 10 p - rw------ 1 0 - -
67622 dhclient 12 s - rw------ 5 0 UDS /var/run/php-fpm.socket
67622 dhclient 13 s - rw------ 17 0 UDS /var/run/php-fpm.socket
67622 dhclient 14 v c r------- 6 460 - /dev/random

Proof the new binary is running from sbin

/var/db/dhclient.leases.igb0 still contains the wrong info but it surely looks like the supersede is honored !
Lets give a few more days, then remove the hardcoded values and use the advanced options in the GUI.

Attached the binary I am using, feel free to scan (I've done this with >100 engines) and try it.
dhclient_opnsense_19.1.zip

itpp21 · Mar 16, 2021, 1:21 PM

3 successful updates now using supersede hardcoded.
Restored interfaces.inc and entered the hardcoded values+supersede in the advanced settings WAN interface, restart WAN.
Give it 3 more days, if this also works we have a winner.

itpp21 · Mar 19, 2021, 1:24 PM

So far 2 renewals without issues, all respecting supersede (and other) setting via GUI config, Ymmv. but I will consider this a solution to a broken pfsense dhclient.

pablot · Nov 12, 2021, 6:45 PM

So it seems that dhclient from OPNSense fixes the problem, so, can someone give a little receipe of what had to be done to solve this issue?

Is it just coping the binary provided here, dhclient_opnsense_19.1 over /sbin/dhclient?

bmeeks · Nov 13, 2021, 1:14 PM

@pablot said in DHCP client unable to get lease from cable provider [solved]:

So it seems that dhclient from OPNSense fixes the problem, so, can someone give a little receipe of what had to be done to solve this issue?

Is it just coping the binary provided here, dhclient_opnsense_19.1 over /sbin/dhclient?

You can find the list of changes they made to dhclient here on GitHub: https://github.com/opnsense/src/commits/master/sbin/dhclient. You would need to compile your own copy of the source on a separate FreeBSD 12.2-STABLE machine.

These guys, like the Netgate team, eventually submit fixes back upstream into FreeBSD. From there they trickle out to the various projects that are based on FreeBSD. But that can sometimes take a bit of time, and also it may be that a particular fix makes it into one FreeBSD version and not another. For example, it may go into 13-CURRENT but not into 12.2-STABLE.

itpp21 · Dec 1, 2021, 2:05 PM

@pablot said in DHCP client unable to get lease from cable provider [solved]:

So it seems that dhclient from OPNSense fixes the problem, so, can someone give a little receipe of what had to be done to solve this issue?

Is it just coping the binary provided here, dhclient_opnsense_19.1 over /sbin/dhclient?

Yep, thats all it took. Still running it and still zero problems.

pablot · Dec 5, 2021, 9:44 AM

@itpp21 unfortunately it's not working for me. Perhaps I have another problem...