DHCP server balancing issue

[de Brouwer Tom (ST-CO/ENG5.1)]

Hi,

 

I’m experiencing some issues with my redundant ISC DHCP setup, I’m using version “dhcp-4.1-ESV-R4”, compiled with “--enable-delayed-ack”.

 

The issue I’m experiencing is that the balancing suddenly stops and never restores correctly, and a such only half of my clients get IP addresses assigned. I debugged this currently to the stage that one of the DHCP servers get’s into the “communications-interrupted” state and therefore does not allow to balance the pool (I think). Sometimes it takes 2 weeks, sometime a few days before the servers gets into this state. Restarting dhcpd resolves this issue.

 

In the slave log, I get the warning “Got POOLREQ, answering negatively!  Peer may be out of leases or database inconsistent.”

 

In the master log, I get the warning ”pool response: 0 leases”

 

Under normal operations I see the master logging “balancing pool 8fe8f68 192.168.6.0/24  total 241  free 20  backup 81  lts -30  max-own (+/-)10” every hour, once I start experiencing this issue the log is changed into “balancing pool 8fe8f68 192.168.6.0/24  total 241  free 19  backup 81  lts -31  max-own (+/-)10  (requesting peer rebalance!)” Which I think means it gets to a critical available lease threshold.

 

What could be causing this issue? I couldn’t find anything in the release notes of dhcp-4.1-ESV-R11 which explicitly points to this issue, so I’m not sure whether upgrading makes sense.

 

Thanks,

 

Tom

 

_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

[Gregory Sloop]

Hi, I’m experiencing some issues with my redundant ISC DHCP setup, I’m using version “dhcp-4.1-ESV-R4”, compiled with “--enable-delayed-ack”. The issue I’m experiencing is that the balancing suddenly stops and never restores correctly, and a such only half of my clients get IP addresses assigned. I debugged this currently to the stage that one of the DHCP servers get’s into the “communications-interrupted” state and therefore does not allow to balance the pool (I think). Sometimes it takes 2 weeks, sometime a few days before the servers gets into this state. Restarting dhcpd resolves this issue. In the slave log, I get the warning “Got POOLREQ, answering negatively!  Peer may be out of leases or database inconsistent.” In the master log, I get the warning ”pool response: 0 leases” Under normal operations I see the master logging “balancing pool 8fe8f68 192.168.6.0/24  total 241  free 20  backup 81  lts -30  max-own (+/-)10” every hour, once I start experiencing this issue the log is changed into “balancing pool 8fe8f68 192.168.6.0/24  total 241  free 19  backup 81  lts -31  max-own (+/-)10  (requesting peer rebalance!)” Which I think means it gets to a critical available lease threshold. What could be causing this issue? I couldn’t find anything in the release notes of dhcp-4.1-ESV-R11 which explicitly points to this issue, so I’m not sure whether upgrading makes sense. Thanks, Tom

I believe there are a number of load-balancing/fail-over bugs/corner-cases in 4.1 versions that are fixed in 4.2 and newer.

You might do a search of the archives and see if you can find someone talking about them, but IIRC, they exist and are hard to nail down and, again IIRC, the "solution" I've seen is "try something more current." (i.e. 4.2+)

HTH

-Greg


_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

[Shawn Routhier]

On Jun 29, 2015, at 8:46 AM, Gregory Sloop <[hidden email]> wrote:

Hi, I’m experiencing some issues with my redundant ISC DHCP setup, I’m using version “dhcp-4.1-ESV-R4”, compiled with “--enable-delayed-ack”. The issue I’m experiencing is that the balancing suddenly stops and never restores correctly, and a such only half of my clients get IP addresses assigned. I debugged this currently to the stage that one of the DHCP servers get’s into the “communications-interrupted” state and therefore does not allow to balance the pool (I think). Sometimes it takes 2 weeks, sometime a few days before the servers gets into this state. Restarting dhcpd resolves this issue. In the slave log, I get the warning “Got POOLREQ, answering negatively!  Peer may be out of leases or database inconsistent.” In the master log, I get the warning ”pool response: 0 leases” Under normal operations I see the master logging “balancing pool 8fe8f68 192.168.6.0/24  total 241  free 20  backup 81  lts -30  max-own (+/-)10” every hour, once I start experiencing this issue the log is changed into “balancing pool 8fe8f68 192.168.6.0/24  total 241  free 19  backup 81  lts -31  max-own (+/-)10  (requesting peer rebalance!)” Which I think means it gets to a critical available lease threshold. What could be causing this issue? I couldn’t find anything in the release notes of dhcp-4.1-ESV-R11 which explicitly points to this issue, so I’m not sure whether upgrading makes sense. Thanks, Tom

I believe there are a number of load-balancing/fail-over bugs/corner-cases in 4.1 versions that are fixed in 4.2 and newer.

Most bugs are generally fixed in all current releases.  We have fixed several failover issues in more recent 4.1 as well as 4.2 or 4.3.

Some of the items we fixed could have odd results, if you are having issues with failover upgrading to R11 is a good idea.

You might do a search of the archives and see if you can find someone talking about them, but IIRC, they exist and are hard to nail down and, again IIRC, the "solution" I've seen is "try something more current." (i.e. 4.2+)

HTH

-Greg

regards,

Shawn

_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

[de Brouwer Tom (ST-CO/ENG5.1)]

Thanks. I will do that.

 

Would these fixes from 4.2 / 4.3 have been merged into 4.1-R11, or is that something I should have a look at myself?

 

Thanks again!

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Shawn Routhier
Sent: maandag 29 juni 2015 18:30
To: Users of ISC DHCP
Subject: Re: DHCP server balancing issue

 

 

On Jun 29, 2015, at 8:46 AM, Gregory Sloop <[hidden email]> wrote:

 

Hi, I’m experiencing some issues with my redundant ISC DHCP setup, I’m using version “dhcp-4.1-ESV-R4”, compiled with “--enable-delayed-ack”. The issue I’m experiencing is that the balancing suddenly stops and never restores correctly, and a such only half of my clients get IP addresses assigned. I debugged this currently to the stage that one of the DHCP servers get’s into the “communications-interrupted” state and therefore does not allow to balance the pool (I think). Sometimes it takes 2 weeks, sometime a few days before the servers gets into this state. Restarting dhcpd resolves this issue. In the slave log, I get the warning “Got POOLREQ, answering negatively!  Peer may be out of leases or database inconsistent.” In the master log, I get the warning ”pool response: 0 leases” Under normal operations I see the master logging “balancing pool 8fe8f68 192.168.6.0/24  total 241  free 20  backup 81  lts -30  max-own (+/-)10” every hour, once I start experiencing this issue the log is changed into “balancing pool 8fe8f68 192.168.6.0/24  total 241  free 19  backup 81  lts -31  max-own (+/-)10  (requesting peer rebalance!)” Which I think means it gets to a critical available lease threshold. What could be causing this issue? I couldn’t find anything in the release notes of dhcp-4.1-ESV-R11 which explicitly points to this issue, so I’m not sure whether upgrading makes sense. Thanks, Tom

I believe there are a number of load-balancing/fail-over bugs/corner-cases in 4.1 versions that are fixed in 4.2 and newer.

 

Most bugs are generally fixed in all current releases.  We have fixed several failover issues in more recent 4.1 as well as 4.2 or 4.3.

Some of the items we fixed could have odd results, if you are having issues with failover upgrading to R11 is a good idea.

You might do a search of the archives and see if you can find someone talking about them, but IIRC, they exist and are hard to nail down and, again IIRC, the "solution" I've seen is "try something more current." (i.e. 4.2+)

HTH

-Greg

 

regards,

Shawn

 

_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

[de Brouwer Tom (ST-CO/ENG5.1)]

Sorry, didn’t read correct. You said you did fix them in 4.1.

 

Thanks.

 

Tom

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of de Brouwer Tom (ST-CO/ENG5.1)
Sent: maandag 29 juni 2015 19:17
To: Users of ISC DHCP
Subject: RE: DHCP server balancing issue

 

Thanks. I will do that.

 

Would these fixes from 4.2 / 4.3 have been merged into 4.1-R11, or is that something I should have a look at myself?

 

Thanks again!

 

From: [hidden email] [[hidden email]] On Behalf Of Shawn Routhier
Sent: maandag 29 juni 2015 18:30
To: Users of ISC DHCP
Subject: Re: DHCP server balancing issue

 

 

On Jun 29, 2015, at 8:46 AM, Gregory Sloop <[hidden email]> wrote:

 

 

Hi, I’m experiencing some issues with my redundant ISC DHCP setup, I’m using version “dhcp-4.1-ESV-R4”, compiled with “--enable-delayed-ack”. The issue I’m experiencing is that the balancing suddenly stops and never restores correctly, and a such only half of my clients get IP addresses assigned. I debugged this currently to the stage that one of the DHCP servers get’s into the “communications-interrupted” state and therefore does not allow to balance the pool (I think). Sometimes it takes 2 weeks, sometime a few days before the servers gets into this state. Restarting dhcpd resolves this issue. In the slave log, I get the warning “Got POOLREQ, answering negatively!  Peer may be out of leases or database inconsistent.” In the master log, I get the warning ”pool response: 0 leases” Under normal operations I see the master logging “balancing pool 8fe8f68 192.168.6.0/24  total 241  free 20  backup 81  lts -30  max-own (+/-)10” every hour, once I start experiencing this issue the log is changed into “balancing pool 8fe8f68 192.168.6.0/24  total 241  free 19  backup 81  lts -31  max-own (+/-)10  (requesting peer rebalance!)” Which I think means it gets to a critical available lease threshold. What could be causing this issue? I couldn’t find anything in the release notes of dhcp-4.1-ESV-R11 which explicitly points to this issue, so I’m not sure whether upgrading makes sense. Thanks, Tom

I believe there are a number of load-balancing/fail-over bugs/corner-cases in 4.1 versions that are fixed in 4.2 and newer.

 

Most bugs are generally fixed in all current releases.  We have fixed several failover issues in more recent 4.1 as well as 4.2 or 4.3.

Some of the items we fixed could have odd results, if you are having issues with failover upgrading to R11 is a good idea.

 

You might do a search of the archives and see if you can find someone talking about them, but IIRC, they exist and are hard to nail down and, again IIRC, the "solution" I've seen is "try something more current." (i.e. 4.2+)

HTH

-Greg

 

regards,

Shawn

 

_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

[Shawn Routhier]

I have been reminded by a co-worker that we also found and fixed an issue 

between delayed-ack and failover recently.  This patch has not been released yet.

So in addition to migrating to a more recent version you may want to disable

delayed-ack for the time being.

regards,

Shawn

On Jun 29, 2015, at 10:17 AM, de Brouwer Tom (ST-CO/ENG5.1) <[hidden email]> wrote:

Thanks. I will do that.

 

Would these fixes from 4.2 / 4.3 have been merged into 4.1-R11, or is that something I should have a look at myself?

 

Thanks again!

 

From: [hidden email] [[hidden email]] On Behalf Of Shawn Routhier
Sent: maandag 29 juni 2015 18:30
To: Users of ISC DHCP
Subject: Re: DHCP server balancing issue

 

 

On Jun 29, 2015, at 8:46 AM, Gregory Sloop <[hidden email]> wrote:

 

Hi, I’m experiencing some issues with my redundant ISC DHCP setup, I’m using version “dhcp-4.1-ESV-R4”, compiled with “--enable-delayed-ack”. The issue I’m experiencing is that the balancing suddenly stops and never restores correctly, and a such only half of my clients get IP addresses assigned. I debugged this currently to the stage that one of the DHCP servers get’s into the “communications-interrupted” state and therefore does not allow to balance the pool (I think). Sometimes it takes 2 weeks, sometime a few days before the servers gets into this state. Restarting dhcpd resolves this issue. In the slave log, I get the warning “Got POOLREQ, answering negatively!  Peer may be out of leases or database inconsistent.” In the master log, I get the warning ”pool response: 0 leases” Under normal operations I see the master logging “balancing pool 8fe8f68 192.168.6.0/24  total 241  free 20  backup 81  lts -30  max-own (+/-)10” every hour, once I start experiencing this issue the log is changed into “balancing pool 8fe8f68 192.168.6.0/24  total 241  free 19  backup 81  lts -31  max-own (+/-)10  (requesting peer rebalance!)” Which I think means it gets to a critical available lease threshold.  What could be causing this issue? I couldn’t find anything in the release notes of dhcp-4.1-ESV-R11 which explicitly points to this issue, so I’m not sure whether upgrading makes sense. Thanks, Tom

I believe there are a number of load-balancing/fail-over bugs/corner-cases in 4.1 versions that are fixed in 4.2 and newer.

 

Most bugs are generally fixed in all current releases.  We have fixed several failover issues in more recent 4.1 as well as 4.2 or 4.3.

Some of the items we fixed could have odd results, if you are having issues with failover upgrading to R11 is a good idea.

You might do a search of the archives and see if you can find someone talking about them, but IIRC, they exist and are hard to nail down and, again IIRC, the "solution" I've seen is "try something more current." (i.e. 4.2+)

HTH

-Greg

 

regards,

Shawn

 

_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

[de Brouwer Tom (ST-CO/ENG5.1)]

Are you able to release this patch on an as-is basis? Or do you have any indication when this would roll into an release?

 

Thanks,

 

Tom

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Shawn Routhier
Sent: maandag 29 juni 2015 19:30
To: Users of ISC DHCP
Subject: Re: DHCP server balancing issue

 

I have been reminded by a co-worker that we also found and fixed an issue 

between delayed-ack and failover recently.  This patch has not been released yet.

So in addition to migrating to a more recent version you may want to disable

delayed-ack for the time being.

 

regards,

Shawn

 

On Jun 29, 2015, at 10:17 AM, de Brouwer Tom (ST-CO/ENG5.1) <[hidden email]> wrote:

 

Thanks. I will do that.

 

Would these fixes from 4.2 / 4.3 have been merged into 4.1-R11, or is that something I should have a look at myself?

 

Thanks again!

 

From: [hidden email] [[hidden email]] On Behalf Of Shawn Routhier
Sent: maandag 29 juni 2015 18:30
To: Users of ISC DHCP
Subject: Re: DHCP server balancing issue

 

 

On Jun 29, 2015, at 8:46 AM, Gregory Sloop <[hidden email]> wrote:

 

Hi, I’m experiencing some issues with my redundant ISC DHCP setup, I’m using version “dhcp-4.1-ESV-R4”, compiled with “--enable-delayed-ack”. The issue I’m experiencing is that the balancing suddenly stops and never restores correctly, and a such only half of my clients get IP addresses assigned. I debugged this currently to the stage that one of the DHCP servers get’s into the “communications-interrupted” state and therefore does not allow to balance the pool (I think). Sometimes it takes 2 weeks, sometime a few days before the servers gets into this state. Restarting dhcpd resolves this issue. In the slave log, I get the warning “Got POOLREQ, answering negatively!  Peer may be out of leases or database inconsistent.” In the master log, I get the warning ”pool response: 0 leases” Under normal operations I see the master logging “balancing pool 8fe8f68 192.168.6.0/24  total 241  free 20  backup 81  lts -30  max-own (+/-)10” every hour, once I start experiencing this issue the log is changed into “balancing pool 8fe8f68 192.168.6.0/24  total 241  free 19  backup 81  lts -31  max-own (+/-)10  (requesting peer rebalance!)” Which I think means it gets to a critical available lease threshold.  What could be causing this issue? I couldn’t find anything in the release notes of dhcp-4.1-ESV-R11 which explicitly points to this issue, so I’m not sure whether upgrading makes sense. Thanks, Tom

I believe there are a number of load-balancing/fail-over bugs/corner-cases in 4.1 versions that are fixed in 4.2 and newer.

 

Most bugs are generally fixed in all current releases.  We have fixed several failover issues in more recent 4.1 as well as 4.2 or 4.3.

Some of the items we fixed could have odd results, if you are having issues with failover upgrading to R11 is a good idea.

You might do a search of the archives and see if you can find someone talking about them, but IIRC, they exist and are hard to nail down and, again IIRC, the "solution" I've seen is "try something more current." (i.e. 4.2+)

HTH

-Greg

 

regards,

Shawn

 

_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

 

_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

[Shawn Routhier]

In general we don’t create patches on a one-off basis.  You can see the changes via the public

git repository.  Go to the main isc page (www.isc.org) on the community tab select source repository,

on that page select ISC DHCP, or look at how to clone it if you’d prefer that style.

We are nearing the end of the current release cycle and hope to have a beta out in August.

Shawn

On Jun 29, 2015, at 10:33 AM, de Brouwer Tom (ST-CO/ENG5.1) <[hidden email]> wrote:

Are you able to release this patch on an as-is basis? Or do you have any indication when this would roll into an release?

 

Thanks,

 

Tom

 

From: [hidden email] [[hidden email]] On Behalf Of Shawn Routhier
Sent: maandag 29 juni 2015 19:30
To: Users of ISC DHCP
Subject: Re: DHCP server balancing issue

 

I have been reminded by a co-worker that we also found and fixed an issue 

between delayed-ack and failover recently.  This patch has not been released yet.

So in addition to migrating to a more recent version you may want to disable

delayed-ack for the time being.

 

regards,

Shawn

 

On Jun 29, 2015, at 10:17 AM, de Brouwer Tom (ST-CO/ENG5.1) <[hidden email]> wrote:

 

Thanks. I will do that.

 

Would these fixes from 4.2 / 4.3 have been merged into 4.1-R11, or is that something I should have a look at myself?

 

Thanks again!

 

From: [hidden email] [[hidden email]] On Behalf Of Shawn Routhier
Sent: maandag 29 juni 2015 18:30
To: Users of ISC DHCP
Subject: Re: DHCP server balancing issue

 

 

On Jun 29, 2015, at 8:46 AM, Gregory Sloop <[hidden email]> wrote:

 

Hi, I’m experiencing some issues with my redundant ISC DHCP setup, I’m using version “dhcp-4.1-ESV-R4”, compiled with “--enable-delayed-ack”. The issue I’m experiencing is that the balancing suddenly stops and never restores correctly, and a such only half of my clients get IP addresses assigned. I debugged this currently to the stage that one of the DHCP servers get’s into the “communications-interrupted” state and therefore does not allow to balance the pool (I think). Sometimes it takes 2 weeks, sometime a few days before the servers gets into this state. Restarting dhcpd resolves this issue. In the slave log, I get the warning “Got POOLREQ, answering negatively!  Peer may be out of leases or database inconsistent.” In the master log, I get the warning ”pool response: 0 leases” Under normal operations I see the master logging “balancing pool 8fe8f68 192.168.6.0/24  total 241  free 20  backup 81  lts -30  max-own (+/-)10” every hour, once I start experiencing this issue the log is changed into “balancing pool 8fe8f68 192.168.6.0/24  total 241  free 19  backup 81  lts -31  max-own (+/-)10  (requesting peer rebalance!)” Which I think means it gets to a critical available lease threshold.  What could be causing this issue? I couldn’t find anything in the release notes of dhcp-4.1-ESV-R11 which explicitly points to this issue, so I’m not sure whether upgrading makes sense. Thanks, Tom

I believe there are a number of load-balancing/fail-over bugs/corner-cases in 4.1 versions that are fixed in 4.2 and newer.

 

Most bugs are generally fixed in all current releases.  We have fixed several failover issues in more recent 4.1 as well as 4.2 or 4.3.

Some of the items we fixed could have odd results, if you are having issues with failover upgrading to R11 is a good idea.

You might do a search of the archives and see if you can find someone talking about them, but IIRC, they exist and are hard to nail down and, again IIRC, the "solution" I've seen is "try something more current." (i.e. 4.2+)

HTH

-Greg

 

regards,

Shawn

 

_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

 

_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users