DHCP - No Free Leases (pf_2.4.3-release-p1)
-
Hi there,
I'm having the following issue:
I've configured a /19 network, a pool of 8170 IPs (reserved 20 IPs), and started having "no free leases" messages after about 1500 leases or such...
Here's an example of that:
Aug 1 13:50:32 dhcpd DHCPDISCOVER from 8c:8e:f2:xx:xx:xx via em11: network 10.0.192.0/19: no free leases
Aug 1 13:50:31 dhcpd DHCPDISCOVER from 00:22:57:xx:xx:xx via em11: network 10.0.192.0/19: no free leases
Aug 1 13:50:27 dhcpd DHCPDISCOVER from 40:9c:28:xx:xx:xx via em11: network 10.0.192.0/19: no free leases
Aug 1 13:50:26 dhcpd DHCPDISCOVER from 40:9c:28:xx:xx:xx via em11: network 10.0.192.0/19: no free leasesConfigured the same scope on a Cisco router and had no problems, with currently 2500+ active leases.
Can't understand why pfsense DHCP is saying "no free leases", when it still has more than 3/4 IPs left.
Tried to reduce the lease time, from 3600 to 900, and max lease to 1800 , but it's still the same...At around 1500+ leases I start getting "no free leases" messages.
If I restart ou stop/start the service, it's only the time until the leases catch up and start seeing the same error again.
I read other users talking about file permissions and leases not being released, but that years ago.
Anyone has any ideas how to solve this?
Thank you!
Best regards,
Hugo -
What did you define as the DHCP pool?
-
could you post the output of the file /var/dhpd/etc/dhcpd.conf
maybe there is an issue with huge subnets? (biggest i have in production is a /22)also, have you tried enabling the rrd-graphs for dhcpd ? (you can check the usage over a period of time then)
-
-
Here's the output:
option domain-name "gestao"; option ldap-server code 95 = text; option domain-search-list code 119 = text; option arch code 93 = unsigned integer 16; # RFC4578 default-lease-time 900; max-lease-time 1800; log-facility local7; one-lease-per-client true; deny duplicates; ping-check true; update-conflict-detection false; authoritative; subnet 10.0.192.0 netmask 255.255.224.0 { pool { ignore bootp; ignore-client-uids true; range 10.0.192.20 10.0.223.240; } option routers 10.0.223.250; option domain-name-servers 10.0.223.250; default-lease-time 900; max-lease-time 1800; }
I've 5000+ users on-site, so I really needed the /19 subnet...:)
Although I've been lucky, since I've not seen more that 3000 connected at the same time (WiFi), but with IoT and everyone having 2 or 3 devices, it's just a question of time.The graphs are enabled:
And this is strange...
The DHCP is OFF and there are still 1519 leases, even tough they should have already been deleted...
Maybe that's the problem...leases not being deleted?But for that to be the issue, there should be more ppl complaining about it, no?
Thank you!
-
I don't think that graph will change until you reenable dhcpd and it actually expires the leases. You can see from the graph that they were increasing during peak and decreasing during off-peak.
I have run at least a /19. It might have been larger.
I have never seen no free leases unless there were actually no free leases.
Except on one occasion where someone enabled MAC Allow in the DHCP Server instead of creating a MAC Address Passthrough in the Captive Portal.
That was fun.
-
So it's strange...Even with just 1500+ leases it started complaining about no free leases...
I also tried to change range, stop/start and nothing, always the same problem.Really out of ideas on how to solve this...besides relying on an external dhcp server...:-/
-
I guess you should learn the format of the dhcp leases file and take a look there to see what you can see.
https://www.freebsd.org/cgi/man.cgi?query=dhcpd.leases&sektion=5&manpath=freebsd+ports
That file is located at
/var/dhcpd/var/db/dhcpd.leases
in pfSense. -
I've looked into the files.
It's pretty standard stuff and I've not spotted any problems.Here's a example of a lease:
lease 10.0.210.31 { starts 3 2018/08/01 12:28:38; ends 3 2018/08/01 12:43:38; tstp 3 2018/08/01 12:43:38; cltt 3 2018/08/01 12:28:38; binding state active; next binding state free; rewind binding state free; hardware ethernet 88:d7:f6:xx:xx:xx; set vendor-class-identifier = "android-dhcp-7.0"; client-hostname "android-xxxxxxxxxxxxxxxx";
The pool permits 8000+ IPs, but after 1500+ it starts giving "No Free Leases" error messages.
Any ideas?
Thank you!
-
Update:
I've deleted all the leases from dhcpd.leases and deleted the dhcpd.leases~ file and activated the dhcp server on pfsense again and until now, no problems...:-)
2039+ leases and going, without any issues...
Will let this run for a couple of days to see if the problem is resolved...
The 1st time, the system was already running for at least 1 week, before ppl started complaining. -
Same problem...:(
Just 1248 leases in use this morning, but already the message "network 10.0.192.0/19: no free leases"...
It must be some pfsense dhcpd issue...since I never had this problem with other dhcpds...
Anyone any ideas?
Thank you!
-
It's ISC dhcpd just like everyone else runs.
You have enough RAM?
-
That's the thing...it's pretty standard stuff...or it should be!
Never had this problem with any other dhcpd in the past...It's something on pfsense...what...I don't know...
If I switch to another dhcpd, on another *nix box...no problems, cisco router...no problems, windows server...no problems...
Just pfsense having issues...RAM is OK...
Memory usage
62% of 16325 MiB -
I found this on another forum...Makes any sense?
This and similar errors RECENTLY started happening to me too. It appears that dhcpd made a significant change in how it handles ranges assigned to a subnet. It appears that if you have any hosts with a fixed-address that exists in your range, the entire range is ignored for anything other than your host entries! This IMHO is a step backwards, but here is my workaround (declare every address in your range as a range itself)
From what I understand...if someone has a fixed address, the dhcpd goes crazy and ignores the range?! Is this correct?!
Thank you!
-
That's how it has been as long as I can remember, the dynamic range is for truly dynamic addresses and static mappings have to outside any dynamic range.
-
pfsense gui will not let you put in a reservation for an IP that is inside your pool.
Other than windows dhcp, which allows for reservations inside a pool.. Yeah your reservations are outside your pool range.. Is that what causing you problems?
-
What I make of it :
@ihugof said in DHCP - No Free Leases (pf_2.4.3-release-p1):
It appears that if you have any hosts with a fixed-address that exists in your range,
So : "If there is some device that has a fixed IP - and this IP is within a pool of the serving DHCP, dhcpd goes on strike".
It can be any client with a user (who doesn't like DHCP) and he assigned to his device a static IP, which brings down the whole cirque ?If this IP can be arpped down, that you should try to firewall it out, and restart dhcpd ...
-
The thing is...anyone can decide to try to fix their IP...and that seems to "crash" the dhcpd...
-
It seems that anyone that try to fix their own IP...can cause problems...Which is kind of dumb...dhcpd side anyway...
-
I have never seen that... I can try and duplicate here.
Just so everyone is clear your pool is say .100 to .200
Then some client sets itself static at .150 and your saying dhcpd stops handing out any IP/Leases in that pool.. Or fails to hand out to a client asking for .150?
That would be stupid and for sure an issue, because your saying any single client on a network could cause what amounts to a dos attack just by setting a static IP to some other IP in the range it got from the dhcpd at first.
-
Yes...it seems to be that...which is dumb, but it seems that if anyone fixes one IP, the dhcpd considers that subnet as "full"...
That would explain why some ppl still were able to get an IP and others didn't...Will need to test this someway...
-
I never had this issue on other *nix boxes or any other dhcp servers...
I still don't know if this is the issue, but it kind of makes sense...since it's the only explanation until now...:)
What I understand from some other post was:
- I have for example a /19 network that goes from 10.0.192.0 to 10.0.223.255
- If someones fixes the IP 10.0.194.10, for example...then all leases from that IP forward, will remain as "in use";
- Only IPs from 10.0.192.0 to 10.0.194.9 will be available for lease.
If it's not this...then I've no clue why a scope with 8000+ IPs, gets full after 1200+ leased IPs...
Thx!
-
And you tracked down this 10.0.194.10 IP and it was set static?
In all my years have not seen this.. What I have seen is duplicate IP issues where someone set IP in pool to static, then for whatever reason the IP detection mech fails.. client with dupe off for example during the leasing ie client/server send out out arp/ping looking to see if IP is in use before requested by client or handed out by dhcpd.
This should be easy enough to try and duplicate.. And I agree this could be a issue for sure since it would mean that any client could even without even trying on purpose create a dos.
https://www.isc.org/wp-content/uploads/2018/02/dhcp44.html#IP%20ADDRESS%20CONFLICT%20PREVENTION
IP ADDRESS CONFLICT PREVENTIONThe DHCP server checks IP addresses to see if they are in use before allocating them to clients. It does this by sending an ICMP Echo request message to the IP address being allocated. If no ICMP Echo reply is received within a second, the address is assumed to be free. This is only done for leases that have been specified in range statements, and only when the lease is thought by the DHCP server to be free - i.e., the DHCP server or its failover peer has not listed the lease as in use.
If a response is received to an ICMP Echo request, the DHCP server assumes that there is a configuration error - the IP address is in use by some host on the network that is not a DHCP client. It marks the address as abandoned, and will not assign it to clients. The lease will remain abandoned for a minimum of abandon-lease-time seconds.
If a DHCP client tries to get an IP address, but none are available, but there are abandoned IP addresses, then the DHCP server will attempt to reclaim an abandoned IP address. It marks one IP address as free, and then does the same ICMP Echo request check described previously. If there is no answer to the ICMP Echo request, the address is assigned to the client.
The DHCP server does not cycle through abandoned IP addresses if the first IP address it tries to reclaim is free. Rather, when the next DHCPDISCOVER comes in from the client, it will attempt a new allocation using the same method described here, and will typically try a new IP address.
-
I had some time this night, and did some tests.
My dhcpd pool range is 192.168.2.10 -> 192.168.2.254. Gateway and DNS == pfSense = 192.168.2.1 / 24
It concerns an interface running the captive portal.I used 2 windows PC's, and gave them static IP's right into the pool (and DNS / Gateway = 192.168.2.1 ). I checked if the IP that was assigning was used ones but expired some time ago.
I didn't notice any suspected behavior, and could use the portal as any other device.
Further DHCP clients could obtain a lease, the portal kept working for them.
No unusual lines in the dhcp log.Btw : Status => DHCP Leases and click on "Show all configured leases" at the bottom of the screen.
For me, this will show all leases, expired, or not. There are actually 254-10=244 IP listed, they all have been used ones in the past, but are recycled by dhcpd when needed.Leases are expired after 12 hours - captive portal hard time out is 6 hours.
Works for .... nearly ten years now.
-
No, it was just an example from what I got on some other post...
That was what someone said it tried and what he found to be the truth...:)I'll try to do some tests, but last time it took more than 24h for the problem to resurface...
Thx!
-
I've enabled the dhcp server on pfsense again, at 07:23am (UTC+1)...let's see how long it takes to start getting the "no free leases" message...
On Status/DHCP Leases I've 1718 leases in use.
If I press "Show all configures leases" button it shows 3489 in use.Most of these leases shown are "offline" and "expired"...since yesterday...
Will dhcpd use these "expired" ones when it needs a free lease?It looks to me that it's keeping them and not releasing them, even when they expired the day before...
It's 09:42am (UTC+1) and still no problems...:-)
-
I guess I've spotted the problem...
dhcpd is not deleting/reusing the "expired" leases...I've 3000+ leases right now...and it says this:
From these 7137, only 3088 are active leases...
Most of leases have expire time of yesterday...but they still exist, they were not deleted...:-/These should already been removed but they're still here...
From what I know from dhcpd, this is normal, since it's just keeping the expired leases, waiting for the same device to come back later...
dhcpd should clean theses leases after a while or when it needs leases, but that's not happening...
Any ideas?
-
@johnpoz said in DHCP - No Free Leases (pf_2.4.3-release-p1):
The DHCP server checks IP addresses to see if they are in use before allocating them to clients. It does this by sending an ICMP Echo request message to the IP address being allocated. If no ICMP Echo reply is received within a second, the address is assumed to be free.
I just checked that, using both Wireshark and Packet Capture. I did not see that ping. I did very that normal pings were captured. Duplicate address detection is mandatory with IPv6 and often used with IPv4. However, that uses arp requests, on IPv4, or neighbour solicitations, on IPv6, to see if the address is in use. This test is done by the device prior to using an assigned address and not the DHCP server.
BTW, I watched for both ICMP and to the broadcast MAC address. I didn't see the ping either way.
-
@ihugof said in DHCP - No Free Leases (pf_2.4.3-release-p1):
I guess I've spotted the problem...
dhcpd is not deleting/reusing the "expired" leases...From these 7137, only 3088 are active leases...
Most of leases have expire time of yesterday...but they still exist, they were not deleted...:-/Noop.
If a device (MAC) comes back, hours, days or weeks after expiration, dhcpd tries to give it back the same IP it gave it ones, in the past. If still available - and marked as such in the file. This file is the 'memory' of dhcpd.
"expired"means : reusable by the same MAC (device), if it comes back some time, or any other MAC (device) if it is needed.
"expired" == freely available IP for any device.I'm wondering if dhcpd is reading and writing this file as it should be on your system.
The file is actually a dump of the internal structures dhcpd uses. This way, when the process is restarted, it's read so it (dhcpd) has a notion of the past - and what is distributed what to who - and when.dhcpd often restarts on your system ?
File system problems ? -
So...I arrived this morning and...
When showing all leases...
And of course...on the logs:
So the idea that dhcpd on pfsense it's not releasing the IPs as it should...it's correct!
Really going nuts with this one!!! :-/
Makes no sense! -
That's what I thought I knew about dhcpd...:o)
Don't know why it's being different on pfsense...As you can see from my last post, all leases are used up and new clients can't get a lease...even when there are just a 300+ users at this time, from the 8000+ leases.
Aug 13 07:17:46 dhcpd DHCPDISCOVER from 6c:27:79:xx:xx:xx via em11: network 10.0.192.0/19: no free leases Aug 13 07:17:46 dhcpd DHCPDISCOVER from 94:0e:6b:xx:xx:xx via em11: network 10.0.192.0/19: no free leases Aug 13 07:17:45 dhcpd DHCPDISCOVER from 58:e2:8f:xx:xx:xx via em11: network 10.0.192.0/19: no free leases Aug 13 07:17:43 dhcpd DHCPDISCOVER from 00:22:57:xx:xx:xx via em11: network 10.0.192.0/19: no free leases Aug 13 07:17:43 dhcpd DHCPACK on 10.0.219.170 to 3c:fa:43:xx:xx:xx (android-2800483908312c30) via em11
I can't find these clients on the leases files, so they are new ones...
So why don't they get an expired lease?
There are like 7000+ expired leases...About the dhcpd process...that's a weird one...seems to be running from just today...
dhcpd 35140 0.0 0.1 20748 9724 - Ss 07:23 0:00.38 /usr/local/sbin/ 35140 dhcpd 07:23 10:37
-rw-r--r-- 1 dhcpd _dhcp 2.2M Aug 13 07:56 dhcpd.leases
But all pfsense services seem to be doing the same...
Last reboot was on 23 Jul'18.What kind of file system problems?
I could I check those?Thx!
-
This image shows the number of leases that are active and thus not in the expired state :
Next image : this shows merely all possible leases - the size of the pool : 8157 - expired, active; whatever - all of it. It's just a dump of the dhcpd leases file (I guess).
When showing all leases...
So the idea that dhcpd on pfsense it's not releasing the IPs as it should...it's correct!
There is not such thing as "dhpcd releasing the leases". They expire - or are renewed before the end of the leasey by the device that uses a lease.
When no renew happens, after the lease time, they just expire and are re distributable to any device asking (new) for one.
When the lease isn't expired, a lease can only be renew by the device "owning" the IP at that moment.For me it clear that, when you took the screen shots, 340 leases are open - out of the 8157 available.
Example :
My pool is 243 IP's : (250-8).
Still, my dhcpd is serving IP's - because most of these IP's (leases) are in the expired state, so given to any one who needs one.
Could you describe your dhcp server settings - what did you take out the the default state ?
Some smart switches or other devices down stream that could manipule the DHCP requets ?
Just an idea : check out the DHCP log : a device is hammering with changing MAC address thus depleting the pool rapidly ?
Whatever ...Remember : your DHCP server and mine are exactly the same. The only thing that is different are the settings.
edit Read : https://serverfault.com/questions/151224/dhcpd-wont-let-go-of-old-leases - an example how a roque client can eat all leases rapidly - and a method how to find that client.
Worse : "Client ignore packet or never receive it" : another DHCPDISCOVER will follow, with another DHCPOFFER with another IP .... -
Here are the settings:
option domain-name "gestao"; option ldap-server code 95 = text; option domain-search-list code 119 = text; option arch code 93 = unsigned integer 16; # RFC4578 default-lease-time 7200; max-lease-time 86400; log-facility local7; one-lease-per-client true; deny duplicates; ping-check true; update-conflict-detection false; authoritative; subnet 10.0.192.0 netmask 255.255.224.0 { pool { option domain-name-servers 10.0.223.250; ignore bootp; range 10.0.192.20 10.0.223.240; } option routers 10.0.223.250; option domain-name-servers 10.0.223.250; default-lease-time 3600; max-lease-time 3601; option ntp-servers pool.ntp.org; }
That's the thing...I don't think they are being marked as expired...
Maybe because of this on the .conf file...?default-lease-time 7200; max-lease-time 86400;
I say this because I noticed that most leases have 24h expiration time and that's is not what I configured...
I tried to manually alter this but it always reverts to the same...Help?! :)
Thx!
-
I tried to change file by shell and kill/start process, after deleting all leases and no luck...
To be "Expired" leases are marked with "24h"...even if the max-lease-time and default-lease-time is set to 3601 and 3600s.
If I restart process via GUI, file is always overwritten...Where is it getting the configs from?!
Even if I remove file permissions, it chowns and chmods the file again with the right ones...
-
If you want to edit something in a conf you have to edit the services.inc file since yes this is what creates conf on service start, etc.
https://github.com/pfsense/pfsense/blob/master/src/etc/inc/services.inc
$dhcpdconf = <<<EOD option domain-name "{$syscfg['domain']}"; option ldap-server code 95 = text; option domain-search-list code 119 = text; option arch code 93 = unsigned integer 16; # RFC4578 {$custoptions} default-lease-time 7200; max-lease-time 86400; log-facility local7; one-lease-per-client true; deny duplicates; ping-check true; update-conflict-detection false; EOD;
-
@ihugof said in DHCP - No Free Leases (pf_2.4.3-release-p1):
default-lease-time 7200;
max-lease-time 86400;default-lease-time 7200; max-lease-time 86400;
are global time out values.
Every pool has its own, pool-specific settings, like in your case :
default-lease-time 3600; max-lease-time 3601;
-
You're the best! :)
It's seems to be that!
I've changed the services.inc and now the expiration time of the leases make more sense!
It still went and got some for tomorrow, that I don't know how, since I deleted all the leases...but lets wait and see of this does the trick!
Many Thx! :)
-
I know, but for some weird reason, it is considering the global values...not the pool ones...
After editing the services.inc like @johnpoz said, now the leases will expire, as they should, in 3601s.
Before it was assuming the 24h of the global value, and for the number of users I have, it just doesn't do, since I've more than 8190 different users, in a 24h period.How come no one had this problem before?
Or maybe no one noticed it...:)I'll let this run for at least 24h, to see if the problem is solved.
Thx!
-
@ihugof said in DHCP - No Free Leases (pf_2.4.3-release-p1):
How come no one had this problem before?
Good question. Dono what the problem is.
My LAN :
Setting :
Or : default lease time one day, max lease time 4 days.
This is a device on LAN, an iPhone, came in this morning :
It has a lease of 4 days => ok.
On my captive portal - another LAN, another DHCP server - another pool:
default lease time = 6 hours - max lease time 12 hours.
Hummm. 12 hours, so ok.
dhcpd on my two interfaces respects de duration as I set it.
-
Maybe it's the number of different clients I have...
But it started happening again...
Some leases are being kept for 24h...:-/Most of the leases are like this...and they will not be reused until they expire, 24h later...:-/
Thx!