ISC DHCP Users

expired lease problem

Classic

List

Threaded

8 messages Options

project722

expired lease problem

We had an unusual problem last night where the server was not able to give out any more leases from a specific pool. The server logs showed " no more free leases". This is a RHEL 6.7 server running dhcp 4.1.1.

We have scripts that run which look for active leases. In the pool in question we had over 200 available leases but the server was unable to provide them to a client. I dug around in dhcpd.leases and found what I think is the problem. I found a ton of leases in the "expired" state with an end date of 11/6. Which, If I am correct, will never move into it next binding state of free so it can be used because its a date in the past.

Here is an example of one:

lease 172.210.141.159 {
starts 2 2017/11/07 11:40:37;
ends 2 2017/11/07 12:10:37;
tstp 2 2017/11/14 11:55:37;
cltt 2 2017/11/07 11:40:37;
binding state expired;
next binding state free;

Am I correct here? If so, what causes this problem and how can I fix it? I have restarted dhcpd, but that does not help. If I were to edit /var/lib/dhcp/dhcpd.leases manually and remove these entries what are some things I should take into consideration?

_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

Sten Carlsen

Re: expired lease problem

I believe those leases will be reused eventually but are kept in this state as long as possible, in the expectation that the previous owner comes back for it.

Based on previous discussions of this symptom, I would look for reasons this particular client does not match any pool.

--
Best regards
Sten Carlsen

No improvements come from shouting:
"MALE BOVINE MANURE!!!"

On 14 Nov 2017, at 18:07, project722 <[hidden email]> wrote:

We had an unusual problem last night where the server was not able to give out any more leases from a specific pool. The server logs showed " no more free leases". This is a RHEL 6.7 server running dhcp 4.1.1.

We have scripts that run which look for active leases. In the pool in question we had over 200 available leases but the server was unable to provide them to a client. I dug around in dhcpd.leases and found what I think is the problem. I found a ton of leases in the "expired" state with an end date of 11/6. Which, If I am correct, will never move into it next binding state of free so it can be used because its a date in the past.

Here is an example of one:

lease 172.210.141.159 {
starts 2 2017/11/07 11:40:37;
ends 2 2017/11/07 12:10:37;
tstp 2 2017/11/14 11:55:37;
cltt 2 2017/11/07 11:40:37;
binding state expired;
next binding state free;

Am I correct here? If so, what causes this problem and how can I fix it? I have restarted dhcpd, but that does not help. If I were to edit /var/lib/dhcp/dhcpd.leases manually and remove these entries what are some things I should take into consideration?
_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

project722

Re: expired lease problem

Your right, I just confirmed that it will stay in an expired state, even after the end date. However, I never said anything about clients not matching a pool. I was able to fix the "no more free leases" error by expanding the pool a bit. But, that is just a band-aid. My concern now is why is the server not doing what it needs to do in order to free up these expired leases to make room for new clients.

On Tue, Nov 14, 2017 at 12:12 PM, Sten Carlsen <[hidden email]> wrote:

I believe those leases will be reused eventually but are kept in this state as long as possible, in the expectation that the previous owner comes back for it.

Based on previous discussions of this symptom, I would look for reasons this particular client does not match any pool.

--
Best regards
Sten Carlsen

No improvements come from shouting:
"MALE BOVINE MANURE!!!"

On 14 Nov 2017, at 18:07, project722 <[hidden email]> wrote:

We had an unusual problem last night where the server was not able to give out any more leases from a specific pool. The server logs showed " no more free leases". This is a RHEL 6.7 server running dhcp 4.1.1.

We have scripts that run which look for active leases. In the pool in question we had over 200 available leases but the server was unable to provide them to a client. I dug around in dhcpd.leases and found what I think is the problem. I found a ton of leases in the "expired" state with an end date of 11/6. Which, If I am correct, will never move into it next binding state of free so it can be used because its a date in the past.

Here is an example of one:

lease 172.210.141.159 {
starts 2 2017/11/07 11:40:37;
ends 2 2017/11/07 12:10:37;
tstp 2 2017/11/14 11:55:37;
cltt 2 2017/11/07 11:40:37;
binding state expired;
next binding state free;

Am I correct here? If so, what causes this problem and how can I fix it? I have restarted dhcpd, but that does not help. If I were to edit /var/lib/dhcp/dhcpd.leases manually and remove these entries what are some things I should take into consideration?
_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

Bill Shirley-2

Re: expired lease problem

In reply to this post by project722

Post your pool definition and log file excerpts.

Bill

On 11/14/2017 12:07 PM, project722 wrote:

We had an unusual problem last night where the server was not able to give out any more leases from a specific pool. The server logs showed " no more free leases". This is a RHEL 6.7 server running dhcp 4.1.1.

We have scripts that run which look for active leases. In the pool in question we had over 200 available leases but the server was unable to provide them to a client. I dug around in dhcpd.leases and found what I think is the problem. I found a ton of leases in the "expired" state with an end date of 11/6. Which, If I am correct, will never move into it next binding state of free so it can be used because its a date in the past.

Here is an example of one:

lease 172.210.141.159 {
starts 2 2017/11/07 11:40:37;
ends 2 2017/11/07 12:10:37;
tstp 2 2017/11/14 11:55:37;
cltt 2 2017/11/07 11:40:37;
binding state expired;
next binding state free;

Am I correct here? If so, what causes this problem and how can I fix it? I have restarted dhcpd, but that does not help. If I were to edit /var/lib/dhcp/dhcpd.leases manually and remove these entries what are some things I should take into consideration?
_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

project722

Re: expired lease problem

shared-network "temp" {

subnet 172.210.140.0 netmask 255.255.252.0 {

option domain-name "example.net";

option routers 172.210.140.1;

## Define the DHCP pool and access list for this pool.

pool {

allow unknown-clients;

failover peer "dhcp-failover";

range 172.210.140.2 172.210.140.255;

range 172.210.141.1 172.210.141.255;

range 172.210.142.1 172.210.142.255;

range 172.210.143.1 172.210.143.254;

}

subnet 172.210.144.0 netmask 255.255.252.0 {

option routers 172.210.144.1;

}

I can't post log entries as we have expanded the pool to patch the issue, so currently there are no errors. However, this pool only shows 74% utilization. And all it takes is a few more turn-ups to cause this problem.

Subnet 172.210.140.0/22

--------------------------------------------------

Monitoring: ON

Warning limit: 80%

Critical limit: 90%

Active leases: 762/1018 (74.9%)

As I've mentioned before, the only thing that stands out to me in dhcpd.leases is the fact that we have a couple hundred of "expired" leases, which could and should be used, even though these are actually being held in the expectation that the previous client will return. (At least, that is my understanding)

But, what is expected behavior here? For example, if we have 500 leases, 250 of them have a binding state of active and the other 250 have expired if a new client comes along DHCP should free up one of the expired ones, correct?

On Tue, Nov 14, 2017 at 3:24 PM, Bill Shirley <[hidden email]> wrote:

Post your pool definition and log file excerpts.

Bill

On 11/14/2017 12:07 PM, project722 wrote:
We had an unusual problem last night where the server was not able to give out any more leases from a specific pool. The server logs showed " no more free leases". This is a RHEL 6.7 server running dhcp 4.1.1.

We have scripts that run which look for active leases. In the pool in question we had over 200 available leases but the server was unable to provide them to a client. I dug around in dhcpd.leases and found what I think is the problem. I found a ton of leases in the "expired" state with an end date of 11/6. Which, If I am correct, will never move into it next binding state of free so it can be used because its a date in the past.

Here is an example of one:

lease 172.210.141.159 {
starts 2 2017/11/07 11:40:37;
ends 2 2017/11/07 12:10:37;
tstp 2 2017/11/14 11:55:37;
cltt 2 2017/11/07 11:40:37;
binding state expired;
next binding state free;

Am I correct here? If so, what causes this problem and how can I fix it? I have restarted dhcpd, but that does not help. If I were to edit /var/lib/dhcp/dhcpd.leases manually and remove these entries what are some things I should take into consideration?

_______________________________________________ dhcp-users mailing list [hidden email] https://lists.isc.org/mailman/listinfo/dhcp-users
_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

Bill Shirley-2

Re: expired lease problem

Do you really need the allow unknown-clients in your pool definition?
If you define any device with a host statement then it won't be eligible
for the only pool in the 172.210.140.0 subnet.

As I understand it, DHCP will use all the available leases before it will
recycle any for a DHCPDISCOVER.

How does your DHCP know which subnet to use for a request? Do you
have host or class statements?

Bill

On 11/15/2017 9:49 AM, project722 wrote:

shared-network "temp" {

subnet 172.210.140.0 netmask 255.255.252.0 {

option domain-name "example.net";

option routers 172.210.140.1;

## Define the DHCP pool and access list for this pool.

pool {

allow unknown-clients;

failover peer "dhcp-failover";

range 172.210.140.2 172.210.140.255;

range 172.210.141.1 172.210.141.255;

range 172.210.142.1 172.210.142.255;

range 172.210.143.1 172.210.143.254;



}

}

subnet 172.210.144.0 netmask 255.255.252.0 {

option routers 172.210.144.1;



}

}

I can't post log entries as we have expanded the pool to patch the issue, so currently there are no errors. However, this pool only shows 74% utilization. And all it takes is a few more turn-ups to cause this problem.

Subnet 172.210.140.0/22

--------------------------------------------------

Monitoring: ON

Warning limit: 80%

Critical limit: 90%

Active leases: 762/1018 (74.9%)

As I've mentioned before, the only thing that stands out to me in dhcpd.leases is the fact that we have a couple hundred of "expired" leases, which could and should be used, even though these are actually being held in the expectation that the previous client will return. (At least, that is my understanding)

But, what is expected behavior here? For example, if we have 500 leases, 250 of them have a binding state of active and the other 250 have expired if a new client comes along DHCP should free up one of the expired ones, correct?
On Tue, Nov 14, 2017 at 3:24 PM, Bill Shirley <[hidden email]> wrote:
Post your pool definition and log file excerpts.

Bill

On 11/14/2017 12:07 PM, project722 wrote:
We had an unusual problem last night where the server was not able to give out any more leases from a specific pool. The server logs showed " no more free leases". This is a RHEL 6.7 server running dhcp 4.1.1.

We have scripts that run which look for active leases. In the pool in question we had over 200 available leases but the server was unable to provide them to a client. I dug around in dhcpd.leases and found what I think is the problem. I found a ton of leases in the "expired" state with an end date of 11/6. Which, If I am correct, will never move into it next binding state of free so it can be used because its a date in the past.

Here is an example of one:

lease 172.210.141.159 {
starts 2 2017/11/07 11:40:37;
ends 2 2017/11/07 12:10:37;
tstp 2 2017/11/14 11:55:37;
cltt 2 2017/11/07 11:40:37;
binding state expired;
next binding state free;

Am I correct here? If so, what causes this problem and how can I fix it? I have restarted dhcpd, but that does not help. If I were to edit /var/lib/dhcp/dhcpd.leases manually and remove these entries what are some things I should take into consideration?

_______________________________________________ dhcp-users mailing list [hidden email] https://lists.isc.org/mailman/listinfo/dhcp-users
_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users
_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

project722

Re: expired lease problem

Thanks Bill for the insight. I am happy to report that the problem has been found and corrected. We have a failover pair and in our last maintenance the failover config on the secondary was redone with a mistake. It was pointing to itself as the failover peer. This caused the two dhcpd.leases file to be out of sync, skewed the pool numbers and caused the out of leases problem. Once corrected, everything went back to normal. This is probably a one off type thing but I hope it helps someone out there down the road.

On Wed, Nov 15, 2017 at 3:10 PM, Bill Shirley <[hidden email]> wrote:

Do you really need the allow unknown-clients in your pool definition?
If you define any device with a host statement then it won't be eligible
for the only pool in the 172.210.140.0 subnet.

As I understand it, DHCP will use all the available leases before it will
recycle any for a DHCPDISCOVER.

How does your DHCP know which subnet to use for a request? Do you
have host or class statements?

Bill
On 11/15/2017 9:49 AM, project722 wrote:
shared-network "temp" {

subnet 172.210.140.0 netmask 255.255.252.0 {

option domain-name "example.net";

option routers 172.210.140.1;

## Define the DHCP pool and access list for this pool.

pool {

allow unknown-clients;

failover peer "dhcp-failover";

range 172.210.140.2 172.210.140.255;

range 172.210.141.1 172.210.141.255;

range 172.210.142.1 172.210.142.255;

range 172.210.143.1 172.210.143.254;



}

}

subnet 172.210.144.0 netmask 255.255.252.0 {

option routers 172.210.144.1;



}

}

I can't post log entries as we have expanded the pool to patch the issue, so currently there are no errors. However, this pool only shows 74% utilization. And all it takes is a few more turn-ups to cause this problem.

Subnet 172.210.140.0/22

--------------------------------------------------

Monitoring: ON

Warning limit: 80%

Critical limit: 90%

Active leases: 762/1018 (74.9%)

As I've mentioned before, the only thing that stands out to me in dhcpd.leases is the fact that we have a couple hundred of "expired" leases, which could and should be used, even though these are actually being held in the expectation that the previous client will return. (At least, that is my understanding)

But, what is expected behavior here? For example, if we have 500 leases, 250 of them have a binding state of active and the other 250 have expired if a new client comes along DHCP should free up one of the expired ones, correct?
On Tue, Nov 14, 2017 at 3:24 PM, Bill Shirley <[hidden email]> wrote:
Post your pool definition and log file excerpts.

Bill

On 11/14/2017 12:07 PM, project722 wrote:
We had an unusual problem last night where the server was not able to give out any more leases from a specific pool. The server logs showed " no more free leases". This is a RHEL 6.7 server running dhcp 4.1.1.

We have scripts that run which look for active leases. In the pool in question we had over 200 available leases but the server was unable to provide them to a client. I dug around in dhcpd.leases and found what I think is the problem. I found a ton of leases in the "expired" state with an end date of 11/6. Which, If I am correct, will never move into it next binding state of free so it can be used because its a date in the past.

Here is an example of one:

lease 172.210.141.159 {
starts 2 2017/11/07 11:40:37;
ends 2 2017/11/07 12:10:37;
tstp 2 2017/11/14 11:55:37;
cltt 2 2017/11/07 11:40:37;
binding state expired;
next binding state free;

Am I correct here? If so, what causes this problem and how can I fix it? I have restarted dhcpd, but that does not help. If I were to edit /var/lib/dhcp/dhcpd.leases manually and remove these entries what are some things I should take into consideration?

_______________________________________________ dhcp-users mailing list [hidden email] https://lists.isc.org/mailman/listinfo/dhcp-users
_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users
_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users
_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

Gregory Sloop

Re: expired lease problem

I'm glad you fixed it, but isn't that something that DHCPd should vigorously complain about in the logs?
*-It would seem that it should complain on startup [not sure if that's the case]
*-Also, it should complain that it can't talk to the peer - itself - and I'd guess the pool balance would be really crazy.
*-I'd imagine it would show communications interrupted etc.

Wasn't that all in the logs? [Not trying to beat you up here - we've all made mistakes that look nuts in retrospect..I'm just really curious/surprised if there were not a ton of indications in the logs...]

I'm perhaps most interested that DHCPd would even start with a peer def was pointing at itself.

I've started centralizing logs with RSync, and find that DHCPd problems, especially related to peer/failover functions, often become more clear if you can see the two sets of logs from both peers in one place - together. [And the obvious benefits of retaining/archiving logs for retrospective review...]

_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users