Re: Add scope without service restart, possible? (Follow-up)

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Add scope without service restart, possible? (Follow-up)

Cayab, Jefrey E.
Thank you all for the very informative replies.

I have to admit that I don't have experience with dual servers with failover setup so if you could share more info, that'll be really great.
Maybe not only for me but for others also.

But just wanted to get this off my mind - so if I have 2 servers in failover mode, I could update configs in the Primary server and restart dhcpd service on it but will not have service downtime to DHCP clients (existing and new)?

@Steve van der Burg, really interested to know more how you push new configs to both servers (manual or automated?) and then the servers test them and, if they pass, restart on alternate minutes.

@Patrick Trapp, I really think a lot of us here could benefit from your workflow.


Thank you

Jef

On Sat, May 23, 2020 at 5:44 AM <[hidden email]> wrote:
Send dhcp-users mailing list submissions to
        [hidden email]

To subscribe or unsubscribe via the World Wide Web, visit
        https://lists.isc.org/mailman/listinfo/dhcp-users
or, via email, send a message with subject or body 'help' to
        [hidden email]

You can reach the person managing the list at
        [hidden email]

When replying, please edit your Subject line so it is more specific
than "Re: Contents of dhcp-users digest..."


Today's Topics:

   1. Re: Add scope without service restart, possible? (Simon Hobson)
   2. RE: Add scope without service restart, possible?
      (Steve van der Burg)
   3. Re: Add scope without service restart, possible? (Patrick Trapp)


----------------------------------------------------------------------

Message: 1
Date: Fri, 22 May 2020 20:22:14 +0100
From: Simon Hobson <[hidden email]>
To: Users of ISC DHCP <[hidden email]>
Subject: Re: Add scope without service restart, possible?
Message-ID: <[hidden email]>
Content-Type: text/plain; charset=us-ascii

Cayab, Jefrey E. <[hidden email]> wrote:

> Wanted to check if it's possible to add new DHCP scope to a running DHCPd service and to take effect without restart the service? Is there a reload equivalent that can take in the updated config?

Not with the ISC server. However, a restart should be very quick for any reasonable configuration - and if you use dual servers with failover, then no interruption in service to end devices.

Simon



------------------------------

Message: 2
Date: Fri, 22 May 2020 19:30:10 +0000
From: Steve van der Burg <[hidden email]>
To: Users of ISC DHCP <[hidden email]>
Subject: RE: Add scope without service restart, possible?
Message-ID:
        <[hidden email]>

Content-Type: text/plain; charset="us-ascii"

I run a pair of servers with all leases (~40k active right now, out of ~370k defined) in failover pools and routinely change the configs and restart with no interruption to service, as Simon just indicated.

I push new configs to both servers and then the servers test them and, if they pass, restart on alternate minutes (primary on even minutes, for example, and secondary on odd minutes).

...Steve

-----Original Message-----
From: dhcp-users <[hidden email]> On Behalf Of Simon Hobson
Sent: Friday,May 22,2020 3:22 PM
To: Users of ISC DHCP <[hidden email]>
Subject: Re: Add scope without service restart, possible?

CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Cayab, Jefrey E. <[hidden email]> wrote:

> Wanted to check if it's possible to add new DHCP scope to a running DHCPd service and to take effect without restart the service? Is there a reload equivalent that can take in the updated config?

Not with the ISC server. However, a restart should be very quick for any reasonable configuration - and if you use dual servers with failover, then no interruption in service to end devices.

Simon

_______________________________________________
ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.

dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

This email is directed in confidence solely to the person named above and may contain confidential, privileged or personal health information. Please be aware that this email may also be released to members of the public under Ontario's Freedom of Information and Protection of Privacy Act if required. Review, distribution, or disclosure of this email by anyone other than the person(s) for whom it was originally intended is strictly prohibited. If you are not an intended recipient, please notify the sender immediately via a return email and destroy all copies of the original message. Thank you for your cooperation.


------------------------------

Message: 3
Date: Fri, 22 May 2020 21:44:15 +0000
From: Patrick Trapp <[hidden email]>
To: Users of ISC DHCP <[hidden email]>
Subject: Re: Add scope without service restart, possible?
Message-ID:
        <[hidden email]>

Content-Type: text/plain; charset="us-ascii"

I can second that. We run a pair of servers plus a staging server. We make changes and test the code on the staging server. If it passes, we sync to a git server. We run a script that causes the production servers to pull the new config from git and restart, one at a time.
________________________________
From: dhcp-users <[hidden email]> on behalf of Steve van der Burg <[hidden email]>
Sent: Friday, May 22, 2020 2:30 PM
To: Users of ISC DHCP <[hidden email]>
Subject: RE: Add scope without service restart, possible?

CAUTION: This email originated from outside of the company. Do not click links or open attachments unless you recognize the sender and know the content is safe.

I run a pair of servers with all leases (~40k active right now, out of ~370k defined) in failover pools and routinely change the configs and restart with no interruption to service, as Simon just indicated.

I push new configs to both servers and then the servers test them and, if they pass, restart on alternate minutes (primary on even minutes, for example, and secondary on odd minutes).

...Steve

-----Original Message-----
From: dhcp-users <[hidden email]> On Behalf Of Simon Hobson
Sent: Friday,May 22,2020 3:22 PM
To: Users of ISC DHCP <[hidden email]>
Subject: Re: Add scope without service restart, possible?

CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Cayab, Jefrey E. <[hidden email]> wrote:

> Wanted to check if it's possible to add new DHCP scope to a running DHCPd service and to take effect without restart the service? Is there a reload equivalent that can take in the updated config?

Not with the ISC server. However, a restart should be very quick for any reasonable configuration - and if you use dual servers with failover, then no interruption in service to end devices.

Simon

_______________________________________________
ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.

dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

This email is directed in confidence solely to the person named above and may contain confidential, privileged or personal health information. Please be aware that this email may also be released to members of the public under Ontario's Freedom of Information and Protection of Privacy Act if required. Review, distribution, or disclosure of this email by anyone other than the person(s) for whom it was originally intended is strictly prohibited. If you are not an intended recipient, please notify the sender immediately via a return email and destroy all copies of the original message. Thank you for your cooperation.
_______________________________________________
ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.

dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/dhcp-users/attachments/20200522/56f5cb30/attachment.htm>

------------------------------

Subject: Digest Footer

_______________________________________________
ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.

dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users


------------------------------

End of dhcp-users Digest, Vol 139, Issue 8
******************************************

_______________________________________________
ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.

dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users
Reply | Threaded
Open this post in threaded view
|

RE: Add scope without service restart, possible? (Follow-up)

Steve van der Burg

(Warning, long message ahead)

 

We have a large network here, connected together with Cisco gear that I don’t run or have access to.  The network folks have configured it to forward DHCP broadcast messages from clients to our two DHCP servers (the failover pair).  It’s my understanding that anything that the clients do by unicast (ie. where they send a request (an INFORM, renew, etc) to the server that granted them the lease) they will fall back to broadcast for.  So that covers anything urgent that might happen while one server is down.  And most things aren’t urgent – clients start trying to renew halfway through their leases, so they’ll just try again a few times if a server doesn’t answer.

 

Anyway, I reviewed my code (that I developed about 15 years ago and have treaked a few times since) and I’m not actually pushing.  Each DHCP server polls my central admin server (another linux server) once a minute.  I could certainly push the configs, though.  It’s a pretty minor detail, really.

 

For the polling I even pull the configs off of a web server running on the admin server.  If I were starting over today, I would just use sftp or scp with key auth and pull (or push from the admin server).

 

Anyway, my ‘download config and maybe restart’ code is all perl.  The config gets tarred up on the admin server and includes a small ‘main’ config, two peer configs (things specific to each peer) and one large, included config (generated by code out of another system that I set up).

 

On each DHCP server, this runs once a minute by cron.  The test for which minute to run on is simple:

 

# To ensure that both servers aren't down at the same time, we run on odd minutes on the secondary

# and even minutes on the primary.

#

if ( -e "$spath/dhcpd.i.am.secondary" ) {

   exit if (localtime)[1] % 2 == 0;

   $me   = "dhcp2";

   $peer = "dhcp1";

}

else {

   exit if (localtime)[1] % 2 == 1;

   $me   = "dhcp1";

   $peer = "dhcp2";

}

 

The secondary server just has one file on it (“dhcpd.i.am.seconday”) that the primary doesn’t.  I could also test for the hostname or something just as easily.  The $me/$peer stuff is there to let the script manipulate the config later.

 

I grab the config from the web server (not shown here – a 1-liner using perl’s LWP package) and then do a check to ensure that something weird hasn’t happened that shrank the config a lot, and also see if the new tar file is different than the last one that I saved a copy of (the one that became the current running config):

 

exit unless -e $cf;                   # get out if there's no new file

unless ( -e $of ) {

   `/bin/cp $cf $of`;                 # create the older version if it never existed

}

exit if (-s $cf) / (-s $of) < 0.5;    # get out if the file has gotten a lot smaller

 

# Do MD5 hashes on both tar files (new one, and the one that holds the current production files):

#

my ($od,$nd);

if ( open(FO,$of) && open(FN,$cf) ) {

   $od = Digest::MD5->new->addfile(*FO)->hexdigest;

   $nd = Digest::MD5->new->addfile(*FN)->hexdigest;

   close FO;

   close FN;

}

exit if $nd eq $od && (-s $of) == (-s $cf);       # if the MD5 hashes and sizes are the same, we can quit now.

 

If what was sent is different, I untar it (and for some reason run MD5 hashes on the pieces against the current pieces – no idea why, really) – not shown here.

 

After untarring, the $me/$peer stuff gets used to select the correct small config piece specific to this server and trash the other one:

 

rename "$me.peer.conf","dhcpd.peer.conf";

unlink "$peer.peer.conf";

 

The small ‘main’ config contains the include statements for the other pieces, but I have untarred the new one, for testing, into /some/test/path, so I need to rewrite those includes:

 

# Create a copy of the root config that we can test with:

#

if ( open(MCF,"dhcpd.conf") && open(NCF,">dhcpd.test.conf") ) {

   while (<MCF>) {

      s#include\s+"/etc/dhcp#include "$extpath#;

      print NCF $_;

   }

   close MCF;

   close NCF;

}

 

Now I can test the new config:

 

my $testres = `/usr/sbin/dhcpd -t -cf $extpath/dhcpd.test.conf 2>&1`;

my $goodconf = 1;

$goodconf = 0 if $testres =~ /Configuration file errors encountered/;

 

If it passes, I copy it into the right place (not shown), wait until near the end of the current minute and then stop the server.  Another check, run by cron once a minute, will start it again a few seconds later.  And even with thousands of subnets and hundred of thousands of leasable addresses, ISC’s dhcpd still starts in under 5 seconds (probably way under) on my servers.  Again, a lot of this is from a long time ago.  I should just let systemd instantly restart dhcpd once I stop it.

 

while ( 1 ) {

   last if (localtime)[0] > 57;

   sleep 1;

}

`/usr/sbin/service isc-dhcp-server stop`;

 

And that’s it.  Over on the admin server, something else creates the huge set of subnet stanzas (99% of the overall config), tars it up with the static pieces and places it where this script can get it.  So a changed config written there is loaded and live on both servers within a couple of minutes.  And each one emails me (also in the script and not shown here) telling me about a successful restart or a failure of the config test shown above.

 

…Steve

 

From: dhcp-users <[hidden email]> On Behalf Of Cayab, Jefrey E.
Sent: Friday,May 29,2020 12:28 PM
To: [hidden email]
Subject: Re: Add scope without service restart, possible? (Follow-up)

 

CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Thank you all for the very informative replies.

I have to admit that I don't have experience with dual servers with failover setup so if you could share more info, that'll be really great.
Maybe not only for me but for others also.

But just wanted to get this off my mind - so if I have 2 servers in failover mode, I could update configs in the Primary server and restart dhcpd service on it but will not have service downtime to DHCP clients (existing and new)?

@Steve van der Burg, really interested to know more how you push new configs to both servers (manual or automated?) and then the servers test them and, if they pass, restart on alternate minutes.

@Patrick Trapp, I really think a lot of us here could benefit from your workflow.


Thank you

 

Jef

 

On Sat, May 23, 2020 at 5:44 AM <[hidden email]> wrote:

Send dhcp-users mailing list submissions to
        [hidden email]

To subscribe or unsubscribe via the World Wide Web, visit
        https://lists.isc.org/mailman/listinfo/dhcp-users
or, via email, send a message with subject or body 'help' to
        [hidden email]

You can reach the person managing the list at
        [hidden email]

When replying, please edit your Subject line so it is more specific
than "Re: Contents of dhcp-users digest..."


Today's Topics:

   1. Re: Add scope without service restart, possible? (Simon Hobson)
   2. RE: Add scope without service restart, possible?
      (Steve van der Burg)
   3. Re: Add scope without service restart, possible? (Patrick Trapp)


----------------------------------------------------------------------

Message: 1
Date: Fri, 22 May 2020 20:22:14 +0100
From: Simon Hobson <[hidden email]>
To: Users of ISC DHCP <[hidden email]>
Subject: Re: Add scope without service restart, possible?
Message-ID: <[hidden email]>
Content-Type: text/plain; charset=us-ascii

Cayab, Jefrey E. <[hidden email]> wrote:

> Wanted to check if it's possible to add new DHCP scope to a running DHCPd service and to take effect without restart the service? Is there a reload equivalent that can take in the updated config?

Not with the ISC server. However, a restart should be very quick for any reasonable configuration - and if you use dual servers with failover, then no interruption in service to end devices.

Simon



------------------------------

Message: 2
Date: Fri, 22 May 2020 19:30:10 +0000
From: Steve van der Burg <[hidden email]>
To: Users of ISC DHCP <[hidden email]>
Subject: RE: Add scope without service restart, possible?
Message-ID:
        <[hidden email]>

Content-Type: text/plain; charset="us-ascii"

I run a pair of servers with all leases (~40k active right now, out of ~370k defined) in failover pools and routinely change the configs and restart with no interruption to service, as Simon just indicated.

I push new configs to both servers and then the servers test them and, if they pass, restart on alternate minutes (primary on even minutes, for example, and secondary on odd minutes).

...Steve

-----Original Message-----
From: dhcp-users <[hidden email]> On Behalf Of Simon Hobson
Sent: Friday,May 22,2020 3:22 PM
To: Users of ISC DHCP <[hidden email]>
Subject: Re: Add scope without service restart, possible?

CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Cayab, Jefrey E. <[hidden email]> wrote:

> Wanted to check if it's possible to add new DHCP scope to a running DHCPd service and to take effect without restart the service? Is there a reload equivalent that can take in the updated config?

Not with the ISC server. However, a restart should be very quick for any reasonable configuration - and if you use dual servers with failover, then no interruption in service to end devices.

Simon

_______________________________________________
ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.

dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

This email is directed in confidence solely to the person named above and may contain confidential, privileged or personal health information. Please be aware that this email may also be released to members of the public under Ontario's Freedom of Information and Protection of Privacy Act if required. Review, distribution, or disclosure of this email by anyone other than the person(s) for whom it was originally intended is strictly prohibited. If you are not an intended recipient, please notify the sender immediately via a return email and destroy all copies of the original message. Thank you for your cooperation.


------------------------------

Message: 3
Date: Fri, 22 May 2020 21:44:15 +0000
From: Patrick Trapp <[hidden email]>
To: Users of ISC DHCP <[hidden email]>
Subject: Re: Add scope without service restart, possible?
Message-ID:
        <[hidden email]>

Content-Type: text/plain; charset="us-ascii"

I can second that. We run a pair of servers plus a staging server. We make changes and test the code on the staging server. If it passes, we sync to a git server. We run a script that causes the production servers to pull the new config from git and restart, one at a time.
________________________________
From: dhcp-users <[hidden email]> on behalf of Steve van der Burg <[hidden email]>
Sent: Friday, May 22, 2020 2:30 PM
To: Users of ISC DHCP <[hidden email]>
Subject: RE: Add scope without service restart, possible?

CAUTION: This email originated from outside of the company. Do not click links or open attachments unless you recognize the sender and know the content is safe.

I run a pair of servers with all leases (~40k active right now, out of ~370k defined) in failover pools and routinely change the configs and restart with no interruption to service, as Simon just indicated.

I push new configs to both servers and then the servers test them and, if they pass, restart on alternate minutes (primary on even minutes, for example, and secondary on odd minutes).

...Steve

-----Original Message-----
From: dhcp-users <[hidden email]> On Behalf Of Simon Hobson
Sent: Friday,May 22,2020 3:22 PM
To: Users of ISC DHCP <[hidden email]>
Subject: Re: Add scope without service restart, possible?

CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Cayab, Jefrey E. <[hidden email]> wrote:

> Wanted to check if it's possible to add new DHCP scope to a running DHCPd service and to take effect without restart the service? Is there a reload equivalent that can take in the updated config?

Not with the ISC server. However, a restart should be very quick for any reasonable configuration - and if you use dual servers with failover, then no interruption in service to end devices.

Simon

_______________________________________________
ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.

dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

This email is directed in confidence solely to the person named above and may contain confidential, privileged or personal health information. Please be aware that this email may also be released to members of the public under Ontario's Freedom of Information and Protection of Privacy Act if required. Review, distribution, or disclosure of this email by anyone other than the person(s) for whom it was originally intended is strictly prohibited. If you are not an intended recipient, please notify the sender immediately via a return email and destroy all copies of the original message. Thank you for your cooperation.
_______________________________________________
ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.

dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/dhcp-users/attachments/20200522/56f5cb30/attachment.htm>

------------------------------

Subject: Digest Footer

_______________________________________________
ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.

dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users


------------------------------

End of dhcp-users Digest, Vol 139, Issue 8
******************************************


This email is directed in confidence solely to the person named above and may contain confidential, privileged or personal health information. Please be aware that this email may also be released to members of the public under Ontario's Freedom of Information and Protection of Privacy Act if required. Review, distribution, or disclosure of this email by anyone other than the person(s) for whom it was originally intended is strictly prohibited. If you are not an intended recipient, please notify the sender immediately via a return email and destroy all copies of the original message. Thank you for your cooperation.

_______________________________________________
ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.

dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users