DHCP lease screen not loading

ANRt

@chance Can you manage a rollback? It's a bit nuclear, but if everything was working on 2.45 then maybe sitting on that until the current development version is stable would be the best option.

chance

@anrt

I probably can, but it's been a little while since the upgrade, and I "think" I can live with this for the time being... Here's hoping it makes 2.5.1...

plague311

@chance I deleted the files again, restarted the DHCP service through the DHCP leases screen and it is backup and running. So I can confirm that deleting those files and restarting the service will bring up that page again.

That being said, after restarting the service the only leases that showed up in that screen were the static IPs. I didn't get time to test if I had to renew each dynamic device to see if it would show up, but when I loaded up my test VM it grabbed a DHCP address and then showed up in the list.

I hope this helps a bit.

plague311

I'll explain what I did in case someone stumbles across this thread looking for a work around.

I run pfsense as a VM on my home server (just an old dell R810, nothing special), running ESXi. I opened up the console through the ESXi webGUi (SSH'ing in would do the same thing), pressed 8 (for shell) on the machine, typed

cd /var/dhcpd/var/db

Just to make sure I was in the right location I performed the

ls -lh

It's all lowercase but for clarification I typed LS -LH just so there's no confusion. After I confirmed that the files were in that directory I typed:

rm dhcpd.leases dhcpd.leases~

The one with the ~ is the active leases file. I deleted both just to be safe. I then opened up the webGUI for pfsense. I navigated to status > dhcp leases and restarted the service. Those showed all of my static IPs, and then the dynamics started filling in as my leases expired.

I hope this helps someone.

Gertjan

@plague311 said in DHCP lease screen not loading:

The one with the ~ is the active leases file.

The one with the ~ is always older, this the backup.

Instead of deleting file, you might try this :

Go here :
/usr/local/www/

Make a backup of this file :
/usr/local/www/status_dhcp_leases.php

Edit the file - using a text edit like Notepad++ - or use nano on pfSense itself :
/usr/local/www/status_dhcp_leases.php

Example :

Yo see the

echo "1 ";

echo "2 ";

etc.
I inserted ?

here they are :

add some of these "echo" lines, and look at the DHCP status page in the GUI.

If your browser reaches (shows) echo "1" but not echo "2", you know thing start to go wrong here :

chance

@gertjan

If the page doesn't load the echo's will never display.

I think you would have to wrap each relevant block in try/catch to see if you could get it to skip the part that errors and print the debug statement (echo "block A skipped").

I don't really know how php behaves though, so take that FWIW.

plague311

@gertjan When those lease files get deleted the screen works, but as chance mentioned when it's not loading, it's not loading anything. It just gives that nginx error and there's nothing. I don't know that this really helps track down an issue.

Gertjan

@chance said in DHCP lease screen not loading:

If the page doesn't load the echo's will never display.

Like rename or delete this file :
/usr/local/www/status_dhcp_leases.php
?
We all ought to have the same file, and believe me : it loads.

@chance said in DHCP lease screen not loading:

try/catch

Ho, ho, .... just keep it easy ;)
I advise to use 'echo x";' which needs just some edit / cut paste skills. Try/catch needs some real PHP skills.

Another option : some one send me over - with a protected paste.bin page - his dhcpdleases file.
Then I should be able to read it ? not be able to read it ? = seeing it as this file is used to create the DHCP Status page.
The thing is : no one communicates these files that "don't work".
If the issue is related to these files ....

edit : but ok, go : implement try catch blocks and tell us where things go wrong.

chance

@gertjan

I may just build a test PHP file from a copy of the existing one that I can call directly from command line and figure out where it's stopping. I do have fairly extensive development skills, just not with PHP (which I haven't used in the last 15 years since I was working with PBX systems)

When mine starts failing again, I will do something as described above.

EDIT: I made a copy of the dhcp leases php file and executed it with

php status_dhcp_leases2.php

It returns a whole html doc to the terminal, so I should be able to figure out which call is causing the issue.

krinn

Still Facing the issue after upgrade to 2.5.1, tried clearing the dhcp leases file and still the problem persist, able to view the dhcp leases through command prompt but GUI is not available. It would be very helpful if the someone who has sorted this issue can help.

Gertjan

@krinn said in DHCP lease screen not loading:

tried clearing the dhcp leases file and still the problem persist, able to view the dhcp leases through command prompt but GUI is not available.

What does show up when you visit that page ?

"Status > DHCP Leases" or https://............/status_dhcp_leases.php

Go to your browser dev mode ( Firefox : Ctrl-Shift - i and then select Network - and hit F5 )

DrQuinn24

@gertjan Here is what is seen when using the instruction for Firefox when choosing Status/DHCP leases:

I have had the same issue since upgrading to 2.5 ( currently on 2.5.1). The only major change prior to 2.5 that I can recall on my system was installing the NextDNS cli.

chance

@drquinn24 It's timing out just like it was for everyone above. the solution is to delete your leases database and restart the dhcp service. That "solves" the problem for an undetermined amount of time, but at the cost of you now don't know what leases are out.

Nothing in 2.5.1 is related to this issue, and I'm not even sure it's going to be a priority for netgate anytime soon.

karlfife

@chance
Has anyone reported this as a bug?

This problem appears to be a function of how many entries are in the dhcp.leases file which DO NOT contain a client-hostname field. MANY offending entries may cause a delay long enough to create a timeout condition.

Reproducable like this:

With zero entries in the dhcp.leases file, on my hardware it takes LESS THAN one second to load the dhcp leases page. Note that my page also renders 36 additional static mappings, even with an empty leases file.

The following entry appears in my dhcp.leases file after some period of time.

lease 10.179.11.33 {
  starts 5 2021/04/16 02:21:58;
  ends 5 2021/04/16 04:21:58;
  tstp 5 2021/04/16 04:21:58;
  cltt 5 2021/04/16 02:21:58;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet e8:4e:06:7a:b1:e2;
  set vendor-class-identifier = "android-dhcp-10";
  client-hostname "Atari5200";
}

Page load time is still under 1 sec. After some time, the next entry appears in dhcp.leases. Note that the new one has no name!

lease 10.179.11.70 {
  starts 5 2021/04/16 02:44:01;
  ends 5 2021/04/16 04:44:01;
  tstp 5 2021/04/16 04:44:01;
  cltt 5 2021/04/16 02:44:01;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet 34:af:b3:c5:f2:d2;
  set vendor-class-identifier = "android-dhcp-9";
}

In this state, on my specific hardware, it takes about 6 seconds to load the dhcp leases page. I'm suggesting that the second entry is causing the problem. To fix the issue, open the dhcp.leases file in vi, and add the following line to the second lease.

  client-hostname "Commodore64-10GBase-LR";

If you figure out how to save/exit from vi, the file looks like this:

lease 10.179.11.33 {
  starts 5 2021/04/16 02:21:58;
  ends 5 2021/04/16 04:21:58;
  tstp 5 2021/04/16 04:21:58;
  cltt 5 2021/04/16 02:21:58;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet e8:4e:06:7a:b1:e2;
  set vendor-class-identifier = "android-dhcp-10";
  client-hostname "Atari5200";
}
lease 10.179.11.70 {
  starts 5 2021/04/16 02:44:01;
  ends 5 2021/04/16 04:44:01;
  tstp 5 2021/04/16 04:44:01;
  cltt 5 2021/04/16 02:44:01;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet 34:af:b3:c5:f2:d2;
  set vendor-class-identifier = "android-dhcp-9";
  client-hostname "Commodore64-10GBase-LR";
}

Without doing anything else (no confounding dhcp service restarts etc.) the dhcp leases page again loads in under 1 second.

I have tested other variations of this test involving an increased number of rogue unnamed entries. More rogue entries appears to cause longer page load times. In every case, opening the dhcp.leases file in vi, and assigning the rogue entries a name appears to eliminate the page load delays all of the time.

If others can reproduce this, I wills file a bug report.

Gertjan

Great write-up.
This is what is needed to find the issue.

I just looked am my /var/dhcpd/var/db/dhcpd.leases file.
About 72 Kbytes - 196 leases, and (only) 19 leases have a "client-hostname".
177 lease entries have no client-hostname at all.
Still : I never had any issues looking at Status > DHCP leases.

So, is it ok to think that the presence of absence of a "client-hostname" line in a lease is not the issue ?

This file : /usr/local/www/status_dhcp_leases.php

When you place a // in front line 91, like this :

// $leases = system_get_dhcpleases();

( do the same thing for the next line 95 - just to be sure )

Status >> DHCP should show an empty page like this :

If still nothing shows up, the issue is not "DHCP" related at all.

chance

I've not reported this, I've actually abandoned ship after 2.5.0 gave me fits with IPSec and GRE tunnels.

I'm glad you guys are making progress though.

I would recommend going to the spot where system_get_dhcpleases() is and find the DNS lookup lines. Add an IF the hostname is null (in php obviously) then skip it.

That should solve the issue.

I checked in the source on github and this is the file.

https://github.com/pfsense/pfsense/blob/0f03681f9b16583ac0f6a6b98272ab9b2d11d79e/src/etc/inc/system.inc#L675

It's already checking for empty, so I assume it's getting a false positive. (or maybe it's not the current version.)

I think it should be something like this:

if (preg_match('/}$/', $line)) {
			if ($lease) {
				if (empty($item['hostname'])) {
					$hostname = gethostbyaddr($item['ip']);
					if (!empty($hostname) &$ empty($hostname) != "") {
						$item['hostname'] = $hostname;
					}
				}
				$leases['lease'][] = $item;
				$lease = false;
				$dedup_lease = true;
			} else if ($failover) {
				$leases['failover'][] = $item;
				$failover = false;
				$dedup_failover = true;
			}
			continue;
		}

Relevant line I added (&& $hostname != "")
if (!empty($hostname) && $hostname != "") {

~~No idea if it will work or not.~~

Nevermind, just comment out the gethostbyaddr line and live without it I guess...

karlfife

@chance
I believe this is covered in this bug report:
https://redmine.pfsense.org/issues/11512

Parallel discussion can be found here:
https://forum.netgate.com/topic/161121/after-upgrade-to-2-5-status_dhcp_leases-php-nor-diag_arp-php-will-load/13

zarteg

Turns out I had the same issue.
Kill your DNS Resolver and try again.

Gertjan

@chance said in DHCP lease screen not loading:

Nevermind, just comment out the gethostbyaddr line and live without it I guess...

The definition of gethostbyaddr is https://www.php.net/manual/fr/function.gethostbyaddr.php

Look somewhat down n on that page, you find :

The problem of broken DNS servers.....

So is this a 'broken DNS issue' after all ?

drumnbisco

Can confirm, had my primary dns set to 9.9.9.9 for dnssec, switched it to googles and clicked save, dhcp leases status page loaded fine. Switched it back and pages loads fine a 2nd time.