dhcpd losing hostnames

[Michael Vincent]

A few pfSense 2.3.1 (based on FreeBSB 10.3) users have noticed an
issue with the dhcp server (ISC DHCP Server 4.3.3-P1) losing client
hostnames regularly. Packet captures have proven that while hostnames
are always provided by clients and generally recorded in dhcpd.leases,
the hostnames often disappear from dhcpd.leases after some period of
time. We haven't been able to conclusively prove that dhcpd itself is
to blame, but investigations of packet captures and logs are pointing
in that direction.

Background is available in this thread:
https://forum.pfsense.org/index.php?topic=110011.0 and a bug ticket
has been opened here: https://redmine.pfsense.org/issues/6589

One user installed OPNSense 16.7 RC1 (a pfSense fork) that includes
ISC DHCP 4.3.4 and still experienced the issue. I'm not sure which
version of dhcp was used in pfSense 2.2.6 offhand, but this issue of
missing hostnames was not present there.

One theory I've come up with is that clients that request a lease too
soon after their last renewal (reuse_lease in the logs) cause dhcpd to
lose those clients' hostnames for some reason, but I haven't had a
chance to test that in a controlled way yet.

Is there any way to make the dhcpd logs more verbose? Any suggestions
on debugging techniques or things to try to narrow down this issue?

My dhcpd.conf:

    option domain-name "mydomain";
    option ldap-server code 95 = text;
    option domain-search-list code 119 = text;
    option arch code 93 = unsigned integer 16; # RFC4578

    default-lease-time 7200;
    max-lease-time 86400;
    log-facility local7;
    one-lease-per-client true;
    deny duplicates;
    ping-check true;
    update-conflict-detection false;
    authoritative;
    subnet 192.168.1.0 netmask 255.255.255.0 {
       pool {
           range 192.168.1.100 192.168.1.245;
       }

       option routers 192.168.1.1;
       option domain-name-servers 192.168.1.1;
       default-lease-time 259200;
       max-lease-time 518400;
       option ntp-servers 192.168.1.1;
    }
    host s_lan_0 {
           hardware ethernet 00:00:00:01:01:01;
       fixed-address 192.168.1.2;
       option host-name "StaticClientA";
    }
_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

[Chris Buechler]

On Mon, Jul 11, 2016 at 9:52 PM, Michael Vincent <[hidden email]> wrote:

> A few pfSense 2.3.1 (based on FreeBSB 10.3) users have noticed an
> issue with the dhcp server (ISC DHCP Server 4.3.3-P1) losing client
> hostnames regularly. Packet captures have proven that while hostnames
> are always provided by clients and generally recorded in dhcpd.leases,
> the hostnames often disappear from dhcpd.leases after some period of
> time. We haven't been able to conclusively prove that dhcpd itself is
> to blame, but investigations of packet captures and logs are pointing
> in that direction.
>
> Background is available in this thread:
> https://forum.pfsense.org/index.php?topic=110011.0 and a bug ticket
> has been opened here: https://redmine.pfsense.org/issues/6589
>

To elaborate on this a bit, the reason for the root problem is
dhcpd.leases file doesn't always have client-hostname specified,
though the client always sends it. We have a daemon that processes
dhcpd.leases and registers hostnames in dnsmasq and/or Unbound from
that (since they can't do proper DDNS), which is kind of a hack, but
the only way to accommodate those in that circumstance.

Prior to dhcpd 4.3, this was always fine, the client-hostname was
always recorded in the lease in dhcpd.leases going back a number of
versions across a decade or so. Now in 4.3.3 and 4.3.4, the presence
of client-hostname is hit and miss for some people. With no apparent
reason as to why some have the issue and others don't.

So it appears there's a regression in 4.3.x with recording
client-hostname to the leases file.
_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

[Shawn Routhier]

> On Jul 11, 2016, at 8:58 PM, Chris Buechler <[hidden email]> wrote:
>
> On Mon, Jul 11, 2016 at 9:52 PM, Michael Vincent <[hidden email]> wrote:
>> A few pfSense 2.3.1 (based on FreeBSB 10.3) users have noticed an
>> issue with the dhcp server (ISC DHCP Server 4.3.3-P1) losing client
>> hostnames regularly. Packet captures have proven that while hostnames
>> are always provided by clients and generally recorded in dhcpd.leases,
>> the hostnames often disappear from dhcpd.leases after some period of
>> time. We haven't been able to conclusively prove that dhcpd itself is
>> to blame, but investigations of packet captures and logs are pointing
>> in that direction.
>>
>> Background is available in this thread:
>> https://forum.pfsense.org/index.php?topic=110011.0 and a bug ticket
>> has been opened here: https://redmine.pfsense.org/issues/6589
>>
>
> To elaborate on this a bit, the reason for the root problem is
> dhcpd.leases file doesn't always have client-hostname specified,
> though the client always sends it. We have a daemon that processes
> dhcpd.leases and registers hostnames in dnsmasq and/or Unbound from
> that (since they can't do proper DDNS), which is kind of a hack, but
> the only way to accommodate those in that circumstance.

I hope that should be read that dnsmasq and Unbound have issues with
proper DDNS?  The current servers should be doing DDNS correctly
and if they aren’t we’d like to know about issues.

>
> Prior to dhcpd 4.3, this was always fine, the client-hostname was
> always recorded in the lease in dhcpd.leases going back a number of
> versions across a decade or so. Now in 4.3.3 and 4.3.4, the presence
> of client-hostname is hit and miss for some people. With no apparent
> reason as to why some have the issue and others don't.
>
> So it appears there's a regression in 4.3.x with recording
> client-hostname to the leases file.

Some questions to help with isolating the problem
1) Do you know if anybody has created a bug ticket for this in the past?

2) You mention 4.3.3 and 4.3.4, do you know if the problem started showing
up earlier than that for example in 4.3.0?

3) From some of the descriptions on the first of the two links above it sounds
like dhcp-cache-threshold may be causing issues.  This was added in 4.3.0
and tries to avoid updating a lease if that isn’t necessary.  Instead it re-uses
the lease information (expiration time and etc) from the current lease and
thus avoids doing an fsync.
You can try to disable it by setting it to 0, add the following line in your config:
dhcp-cache-threshold 0;
and see if that affects the problem.

4) Have people seen this that aren’t using pfsense?

5) From the description in the pfsense chain it sounds like the client hostname
is always being included in the DHCP messages - is that correct?

Shawn Routhier
ISC DHCP lead engineer

> _______________________________________________
> dhcp-users mailing list
> [hidden email]
> https://lists.isc.org/mailman/listinfo/dhcp-users

_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

[Michael Vincent]

On Tue, Jul 12, 2016 at 2:57 PM, Shawn Routhier <[hidden email]> wrote:
> 1) Do you know if anybody has created a bug ticket for this in the past?

I am not aware of anyone creating an isc-dhcpd bug ticket for this
issue so far. The first reference to this issue was in the pfSense 2.3
beta in February: https://forum.pfsense.org/index.php?topic=107235.0.
I'm fairly sure those betas were using isc-dhcpd 4.3.3 as well. Since
Chris just created a pfSense bug ticket a few days ago, I doubt any of
the pfSense devs were aware of this issue before then.

> 2) You mention 4.3.3 and 4.3.4, do you know if the problem started showing
> up earlier than that for example in 4.3.0?

I just booted a pfSense 2.2.6 livecd in a VM and it's using isc-dhcpd
4.2.8. This issue is not present in that version of pfSense.

I'm currently running an embedded install of pfSense, which makes it
difficult to roll back to earlier versions of isc-dhcpd. It looks like
pfSense jumped from 4.2.8 straight to 4.3.3. I'll see if I can maybe
just swap out the isc-dhcpd binary for an earlier version to test.

> 3) From some of the descriptions on the first of the two links above it sounds
> like dhcp-cache-threshold may be causing issues.  This was added in 4.3.0
> and tries to avoid updating a lease if that isn’t necessary.  Instead it re-uses
> the lease information (expiration time and etc) from the current lease and
> thus avoids doing an fsync.
> You can try to disable it by setting it to 0, add the following line in your config:
> dhcp-cache-threshold 0;
> and see if that affects the problem.

Thanks, I'll give that a shot!

> 4) Have people seen this that aren’t using pfsense?

I have done some searching around the internet and have not come
across any other reports of similar issues. The way that pfSense
registers the client hostnames in dnsmasq or Unbound by parsing the
dhcpd.leases file may be somewhat unique. We only noticed the issue
when we were no longer able to reliably resolve dhcp client hostnames
after upgrading from pfSense 2.2 to 2.3. I wouldn't be surprised if
this behavior is present outside of pfSense, but doesn't really affect
anyone that doesn't parse dhcpd.leases for client hostnames.

> 5) From the description in the pfsense chain it sounds like the client hostname
> is always being included in the DHCP messages - is that correct?

Yes, a couple of us have done packet captures over extended durations
and the client hostnames are *always* included in the DHCP request
messages.
_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

[Michael Vincent]

On Wed, Jul 13, 2016 at 1:31 AM, Michael Vincent <[hidden email]> wrote:
> On Tue, Jul 12, 2016 at 2:57 PM, Shawn Routhier <[hidden email]> wrote:
>> You can try to disable it by setting it to 0, add the following line in your config:
>> dhcp-cache-threshold 0;
>> and see if that affects the problem.
>
> Thanks, I'll give that a shot!

Setting dhcp-cache-threshold to 0 has completely resolved the issue
for two of us on three installations so far :-). That works great as a
workaround, but it would be nice to fix the root issue.

I've looked at the source a bit, but don't understand the whole flow
yet. Do you have any idea why isc-dhcpd might forget the client
hostname in the reuse_lease case? The section of code around
server/dhcp.c:3037 looks like it should handle populating
lt->client_hostname, but perhaps that's failing for some reason? Since
lt->cltt is getting updated, it seems unlikely that the new lease is
getting entirely lost or erased.

I got the unit tests building and running, but I don't see any
high-level tests that I could modify to reproduce this issue.

What are the next steps?
_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

[Shawn Routhier]

> On Jul 14, 2016, at 9:21 PM, Michael Vincent <[hidden email]> wrote:
>
> On Wed, Jul 13, 2016 at 1:31 AM, Michael Vincent <[hidden email]> wrote:
>> On Tue, Jul 12, 2016 at 2:57 PM, Shawn Routhier <[hidden email]> wrote:
>>> You can try to disable it by setting it to 0, add the following line in your config:
>>> dhcp-cache-threshold 0;
>>> and see if that affects the problem.
>>
>> Thanks, I'll give that a shot!
>
> Setting dhcp-cache-threshold to 0 has completely resolved the issue
> for two of us on three installations so far :-). That works great as a
> workaround, but it would be nice to fix the root issue.

yes - dhcp-cache-threshold should be a nice performance win if
you have a number of clients that renew early (or if you have a single
client that goes crazy and renews all the time).

>
> I've looked at the source a bit, but don't understand the whole flow
> yet. Do you have any idea why isc-dhcpd might forget the client
> hostname in the reuse_lease case? The section of code around
> server/dhcp.c:3037 looks like it should handle populating
> lt->client_hostname, but perhaps that's failing for some reason? Since
> lt->cltt is getting updated, it seems unlikely that the new lease is
> getting entirely lost or erased.
>
> I got the unit tests building and running, but I don't see any
> high-level tests that I could modify to reproduce this issue.
>
> What are the next steps?

The next step is to open a bug, which you can do if you’d
like or I’ll do tomorrow.  If you open it it’s a bit easier to
get the correct email address for you if you want to receive
updates.

The most likely candidate for failure is that we aren’t copying
the hostname as we move things around.  The server creates
a temp copy of the lease to do updates and merges information
back into the lease when it finishes.  There would appear to be
a path through the code where we either don’t copy the hostname
or we do copy it and remove it from the lease and then don’t
return it when we decide we don’t need to update the lease.

Now that I know what the basic cause of the problem is I can
try to figure out where the bug is.  I’ll have to see where it fits
in our schedule priority wise but I hope to get a patch into the
next release.  As I try and reproduce it I may have more questions
in case there is something else in your environments that triggers
the issue.

thanks for trying out disabling dhcp-cache-threshold

regards,
Shawn

_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

[Chris Buechler-2]

On Tue, Jul 12, 2016 at 2:57 PM, Shawn Routhier <[hidden email]> wrote:

>
>> On Jul 11, 2016, at 8:58 PM, Chris Buechler <[hidden email]> wrote:
>>
>> On Mon, Jul 11, 2016 at 9:52 PM, Michael Vincent <[hidden email]> wrote:
>>> A few pfSense 2.3.1 (based on FreeBSB 10.3) users have noticed an
>>> issue with the dhcp server (ISC DHCP Server 4.3.3-P1) losing client
>>> hostnames regularly. Packet captures have proven that while hostnames
>>> are always provided by clients and generally recorded in dhcpd.leases,
>>> the hostnames often disappear from dhcpd.leases after some period of
>>> time. We haven't been able to conclusively prove that dhcpd itself is
>>> to blame, but investigations of packet captures and logs are pointing
>>> in that direction.
>>>
>>> Background is available in this thread:
>>> https://forum.pfsense.org/index.php?topic=110011.0 and a bug ticket
>>> has been opened here: https://redmine.pfsense.org/issues/6589
>>>
>>
>> To elaborate on this a bit, the reason for the root problem is
>> dhcpd.leases file doesn't always have client-hostname specified,
>> though the client always sends it. We have a daemon that processes
>> dhcpd.leases and registers hostnames in dnsmasq and/or Unbound from
>> that (since they can't do proper DDNS), which is kind of a hack, but
>> the only way to accommodate those in that circumstance.
>
> I hope that should be read that dnsmasq and Unbound have issues with
> proper DDNS?  The current servers should be doing DDNS correctly
> and if they aren’t we’d like to know about issues.
>

Neither of them support RFC 2136, as they're not considered
authoritative DNS servers. In dnsmasq's case you have to populate the
hosts file, in Unbound's, hard code in its conf file.


>> So it appears there's a regression in 4.3.x with recording
>> client-hostname to the leases file.
>
> Some questions to help with isolating the problem
> 1) Do you know if anybody has created a bug ticket for this in the past?
>

Not that I'm aware of.


> 2) You mention 4.3.3 and 4.3.4, do you know if the problem started showing
> up earlier than that for example in 4.3.0?
>

4.3.2 was the first 4.3.x version we shipped as part of pfsense, not
sure on any 4.3.x prior to that.


> 3) From some of the descriptions on the first of the two links above it sounds
> like dhcp-cache-threshold may be causing issues.  This was added in 4.3.0
> and tries to avoid updating a lease if that isn’t necessary.  Instead it re-uses
> the lease information (expiration time and etc) from the current lease and
> thus avoids doing an fsync.
> You can try to disable it by setting it to 0, add the following line in your config:
> dhcp-cache-threshold 0;
> and see if that affects the problem.
>

Judging by the reports of Michael in this thread, and another of our
users on our forum, setting 'dhcp-cache-threshold 0' works around the
issue. I made that change so we have a workaround available for users.
https://redmine.pfsense.org/issues/6589
https://forum.pfsense.org/index.php?topic=110011.15

> 4) Have people seen this that aren’t using pfsense?
>

I doubt many, if any other people rely on client-hostname's presence
in the leases file to the extent they'd notice its absence.


> 5) From the description in the pfsense chain it sounds like the client hostname
> is always being included in the DHCP messages - is that correct?
>

Yes, I've also confirmed that to be the case. I've seen systems where
a pcap of the DHCP requests show the hostname, and then the leases
file ends up not having it.

Thanks for the help, we at least have a workaround available now. For
our users' use cases, disabling the dhcp-cache-threshold is an
acceptable workaround in the mean time until the root cause is fixed.
_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users