DHCP lease screen not loading

[plague311]

Hello,

Long time listener, first time caller. The main reason being I run a pretty simple network and pfsense has always been great.

I did the 2.5 update a few days ago and now the status > DHCP lease screen won't pull up. It gives me a 504 gateway NGINX error after a few minutes. Scrolled through the logs and found the below error:

Feb 25 11:53:40 nginx 2021/02/25 11:53:40 [error] 35428#100132: *2 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.41, server: , request: "GET /status_dhcp_leases.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "192.168.1.1", referrer: "https://192.168.1.1/services_dhcp.php"

I was able to replicate this error every time I had the issue. I'm a networking guy by trade, but admittedly am not great at figuring this stuff out when it comes to the background programs. Any help would be great, thanks in advance.

[VA3DPrinter]

Similar issue. I just updated to 2.5.0 and the login screen appears but once authenticated, it just hangs and I see these in the logs:

SYSTEM.LOG:
Feb 25 23:42:57 gw nginx: 2021/02/25 23:42:57 [error] 51612#100143: *8 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.100, server: , request: "GET / HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "192.168.1.254", referrer: "https://192.168.1.254/"

NGINX.LOG
Feb 25 23:49:20 gw nginx: 2021/02/25 23:49:20 [error] 51612#100143: *13 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.100, server: , request: "GET / HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "192.168.1.254", referrer: "https://192.168.1.254/"

Tried restarting PHP-FPM and the web configurator already with no luck.

[Gertjan]

@plague311 :

Something like https://forum.netgate.com/topic/160956/sg-1100-dhcp-lease-status-page-load-time?_=1614319231879 ?

@VA3DPrinter : loading the "DHCP lease screen not loading ", right ?

[VA3DPrinter]

@va3dprinter

So based on another post that discussed your DHCP issue and the volume of data required to process/display the page, and another post talking about the widgets on the front page I found my issue was a combination of the two. I had enable displaying of the Firewall logs and had a refresh of 5 seconds. If you have a large firewall log file, it is processing that file every five seconds. This of course kills your performance and the server's ability to render the page, so nginx gets a timeout. I removed the widget and the system is performing much better now. Will have to look into off-boxing logs and a more appropriate place to process/display that info. Thanks for the pieces that solved the puzzle...

[plague311]

@gertjan said in DHCP lease screen not loading:

@plague311 :

Something like https://forum.netgate.com/topic/160956/sg-1100-dhcp-lease-status-page-load-time?_=1614319231879 ?

@VA3DPrinter : loading the "DHCP lease screen not loading ", right ?

Nope, it's not similar to that really. The page doesn't load at all. It throws those errors and gives a "504 Gateway Timeout nginx" error. No matter how long I wait. It's not that big of a deal, I can just move the DHCP off of pfsense. I flushed all of the logs just in case, but it didn't make a difference. I can't find any storage or log size issues. It looks like this only started due to something on 2.5. It's not that big of a deal though, I can either move my DHCP over to a server or revert to 2.4.5.

[Gertjan]

@plague311

So, as asked in the other - mentioned -thread : what is the file size /var/dhcpd/var/db/dhcpd.leases ?

@plague311 said in DHCP lease screen not loading:

gives a "504 Gateway Timeout nginx" erro

That's an error that gets shown if the info to prepare (PHP makes html for nginx so it gets send to your browser) is to big.
Like a huge leases file.

Maybe you're right, but I cant' exclude that the file size issue as the origin of your problem.

Is your /var/dhcpd/var/db/dhcpd.leases like :

-rw-r--r--  1 dhcpd  _dhcp  22321 Mar  4 13:38 /var/dhcpd/var/db/dhcpd.leases

close to 23k bits.

My leases show up in the GUI right away.

[plague311]

@gertjan said in DHCP lease screen not loading:

@plague311

So, as asked in the other - mentioned -thread : what is the file size /var/dhcpd/var/db/dhcpd.leases ?

@plague311 said in DHCP lease screen not loading:

gives a "504 Gateway Timeout nginx" erro

That's an error that gets shown if the info to prepare (PHP makes html for nginx so it gets send to your browser) is to big.
Like a huge leases file.

Maybe you're right, but I cant' exclude that the file size issue as the origin of your problem.

Is your /var/dhcpd/var/db/dhcpd.leases like :

-rw-r--r--  1 dhcpd  _dhcp  22321 Mar  4 13:38 /var/dhcpd/var/db/dhcpd.leases

close to 23k bits.

My leases show up in the GUI right away.

Here is a screenshot of what you requested. It looks like there are a few outdated files in there. Perhaps those could be causing an issue. The files size for the current file is close to what you're showing though.

[pette_rsson]

I just updated and got the same problem. Did you fix this somehow?

[plague311]

@pette_rsson Nope, not as of yet. I'll probably just end up moving my DHCP to another server until netgate figures out a fix for it. I'm not seeing much online for a resolution so it must not be overly common.

[pette_rsson]

@plague311 hmm, ok.

I don't want to move it, so I need to fix it... :(

[plague311]

@pette_rsson said in DHCP lease screen not loading:

@plague311 hmm, ok.

I don't want to move it, so I need to fix it... :(

I hear ya, I don't want to move it either. I tried everything I could think of and got no where really.

[pette_rsson]

@plague311 Then it should be reported as a bug. Is there no possibility to raise a bug ?

[plague311]

@pette_rsson said in DHCP lease screen not loading:

@plague311 Then it should be reported as a bug. Is there no possibility to raise a bug ?

I don't know that I've ever reported one. In all honesty this is the first issue I've ever had with pfsense. I have some time later today though, and I can dig around to submit a report.

[ANRt]

Running into a similar issue on my system -- pfSense on bare metal (i5-4570, 8GB RAM).

I absolutely cannot get the Status: DHCP Leases page to load. Most pages in the GUI load normally; however other pages seem to load slower than they should and have historically (the Dashboard and the Update page, for example), others will time out occasionally and load normally at other times.

There is a similar Reddit thread about this issue at: https://old.reddit.com/r/PFSENSE/comments/lzt9qu/504_gateway_timeout_when_opening_status_dhcp/

[Gertjan]

@anrt Then tell us why the GUI => NGINX => PHP has so much trouble shwoing that file.

A 30 kbytes files takes some miliseconds to show.
How big is your lease files ?

I never saw these dhcpd.leases.xxxxxxxx files.
I guess these files are made when dhcpd re creates the leases files, and is faulting during so.
dhpcd is made by https://www.isc.org/dhcp/ and very solid. Still, it only works when you throw real RFC [DHCP] defied traffic on it. Other traffic, generated by the device that cost less then 1 $, could break everything. Even dhcpd. The golden law always applies : BS in = BS out.

So, what about : when are these dhcpd.leases.xxxxxxxx files created ? What happend on that moment with the system ? A dhcpd segfault ? Some one was ripping the power ones again ? Something else ?
What was logged at that moment ?
A host name with binary chars ?
Etc.

I'll agree with you both : some exceptional is happening. So, go hunt for it.

A sledge hammer solution : remove every device from the LANs. There will be no more issues.
Right ?
No, add one device at the time. And wait. No issues ? Add another device, etc.
When the issue pops up, you know what device to check.

Can you read that file yourself (use Notepad++ and open it) ?
Are there strange, binary ? characters ?

And what about your network ?
Is there a device that want a new lease every xx seconds ?
Is there a device that changes it 'name' on every lease ?
Etc.

There is a device that makes your live so much easier : an UPS.

Btw : Widget and massive firewall log rules.
The widget is there exist if there are really a few log lines to show. Just for the kicks. As you said yourself, the thing updates itself every 5 seconds, using world's most ineffective language on the planet : PHP (just a bit faster as Basic).

What is dropped on the WAN : you don't care / you don't log. "Many have tried, all have died".
LAN : same thing : no need to see the kids running in the wall every 5 seconds.

Huge logs files are ok of course. You should deal with them using the ancient tools like vi, nano, Notepad++, tail, logwatch (used on a remote syslog server) etc.

Firewall wall rules are there to be tested, and ones proven ok, make them silent. This way, only the exceptions will pop up, and draw your attention.

Remember this : to much info kills the info.

[plague311]

@gertjan No idea why you're so aggressively going after people just asking questions, but it appears that this is something to do with 2.5, not with any of our configurations themselves.

I run <30 devices on my network, 15 of which are statically addressed. I have no firewall rules because I don't need them. My DHCP scope allows about 40 dhcp addresses, and I'm sure I've restarted it a few times, which has created those numbered files. That being said, I can't imagine any of those causing an issue because they're extremely small, and it doesn't appear they're used for anything. Also, the DHCP scope itself is working absolutely fine. The router is handing out addresses properly. The only issue is loading that screen. Also, those numbered files are from months ago and have caused no issues in the time frame.

We hunted, we brought error messages, and we asked on the forum, in the subsection designed for such questions. Do I know how to resolve this? No, it doesn't make any sense because NOTHING on my network has changed, sans the update to 2.5.

Seriously though, if you're that frustrated by us asking questions, just don't answer. I'd rather work with someone from netgate than be quasi scolded by someone for using a forum for its designated purpose. Thank you for your help to this point but it's not the dhcp file size.

[ANRt]

@gertjan

• The network settings are unchanged for months.
• The hardware connected to the network has not changed for a similar period of time.
• There are approximately 15-30 devices connected to the network at any given time, this is not a different number that was the case some months ago.
• This system was updated to 2.5.0-RELEASE approximately two week ago. This issue began appearing over the weekend or possibly Monday.
• The DHCP log does not show a significant number of requests in a short period (the 50 most recent entries in the log took place over the course of slightly more than an hour).
• The issue persists with the primary switch disconnected and only a single system connected to the network (aside from the pfSense box)

[pette_rsson]

The page worked perfectly just before the update, and right after the update it was broken. I didn’t change a thing, did not even leave my chair....

[Gertjan]

@pette_rsson said in DHCP lease screen not loading:

The page worked perfectly just before the update

When the dust settled, did you executed a clean GUI reboot ?

Time to inspect the dhcp leases files ?
Like :

[2.5.0-RELEASE][admin@pfsense.local.net]/root: ls -al /var/dhcpd/var/                                                                                                             db
total 88
drwxr-xr-x  2 dhcpd  _dhcp    512 Mar 10 07:52 .
drwxr-xr-x  4 dhcpd  _dhcp    512 Jan 29  2015 ..
-rw-r--r--  1 dhcpd  _dhcp  33690 Mar 10 08:04 dhcpd.leases
-rw-r--r--  1 dhcpd  _dhcp  34410 Mar 10 07:52 dhcpd.leases~
-rw-r--r--  1 dhcpd  _dhcp    527 Mar 10 06:03 dhcpd6.leases
-rw-r--r--  1 dhcpd  _dhcp    772 Mar 10 06:03 dhcpd6.leases~

The last two files are for the IPv6 DHCP leases and not concerned here.

What did you find in the file "dhcpd.leases" ? (the dhcpd.leases~ is just an unused copy).
Are there any "dhcpd.leases.xxxxxxx" where xxxxx is a number ?

You can stop the DHCP server(s), wipe the dhcpd.leases and dhcpd.leases~ files, and restart the DHCP server(s). That helps ?

Check the Status > System Logs > DHCP page.
Refresh a page a couple of times. is there a lot of activity ? Any 'strange' messages ?

@pette_rsson said in DHCP lease screen not loading:

did not even leave my chair....

Don't worry. It isn't about what you've done. It's more about what you (and we) don't know yet.

Plan Z : (something I would do ) :
Take a look at the /usr/local/www/status_dhcp_leases.php
It's setting up some stuff, reads the leases file, and loops over it.

I would place log lines so it drops an 'I'm here' every xx lines. They way, you'll see, when you look at the DHCP Lease files with the GUI, when it stops logging thus when it reached the point where it block. At that point, start logging the variables (their values) used just above, and you'll find out why an where it blocks.

[plague311]

@gertjan Thank you for this. After I stopped the server, deleted all of the files, and restarted the server the dhcp leases screen is now pulling up without an issue.

Genuinely confusing but as long as it works I'm happy.

Thank you again for your help, it's very appreciated.