DHCPREQUEST flooding

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

DHCPREQUEST flooding

Alex Moen
I am running a 4.2.5 ISC DHCP server (up-to-date via Centos 7
repository) for our ISP business.  We have around 7000 subscribers; most
with an el-cheapo router, a few with no router at all.  Most of our
customers are using a variant of Linksys router (Linksys, Cisco-Linksys,
Belkin, etc) because that is what we provide if they ask for a router.
However, this issue is not only a Linksys issue, as we are also seeing
PCs exhibiting the same behavior.

The issue is that we have a fairly large number of devices (around 300)
that are issuing DHCPREQUESTs at extremely short intervals (the worst, a
few second apart).  In the last 6 hours, some of these devices have
REQUESTed over 2000 times.  They are all being ACKed.

Is this a common problem that everyone sees, or do I have a config
issue?  This has actually been going on for a long, long time, and I am
just tired of the large log file sizes.  Since we're an ISP, we have to
keep our logs for a few years time, so the log file size can become an
issue.

A typical network stanza looks like:

         subnet 76.10.94.0 netmask 255.255.254.0 {
         pool {
               authoritative;
               range 76.10.94.20 76.10.95.200;
               min-lease-time 129600;
               max-lease-time 259200;
               default-lease-time 259200;
               option subnet-mask 255.255.254.0;
               option broadcast-address 76.10.95.255;
               option routers 76.10.94.1;
               }
         }

Thanks for any input!!

Alex
_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users
Reply | Threaded
Open this post in threaded view
|

RE: DHCPREQUEST flooding

Patrick Trapp
Do the 300-ish devices share anything in particular in their configurations? Is the configuration you shared pertinent to some of your culprits?

Can you confirm that the ACK are reaching the devices? Do any of the devices lose their address entirely and have to be rebooted to get back on the network or is this issue literally only apparent to you and your logs?

Patrick

________________________________________
From: [hidden email] [[hidden email]] on behalf of Alex Moen [[hidden email]]
Sent: Thursday, May 05, 2016 9:40 AM
To: [hidden email]
Subject: DHCPREQUEST flooding

I am running a 4.2.5 ISC DHCP server (up-to-date via Centos 7
repository) for our ISP business.  We have around 7000 subscribers; most
with an el-cheapo router, a few with no router at all.  Most of our
customers are using a variant of Linksys router (Linksys, Cisco-Linksys,
Belkin, etc) because that is what we provide if they ask for a router.
However, this issue is not only a Linksys issue, as we are also seeing
PCs exhibiting the same behavior.

The issue is that we have a fairly large number of devices (around 300)
that are issuing DHCPREQUESTs at extremely short intervals (the worst, a
few second apart).  In the last 6 hours, some of these devices have
REQUESTed over 2000 times.  They are all being ACKed.

Is this a common problem that everyone sees, or do I have a config
issue?  This has actually been going on for a long, long time, and I am
just tired of the large log file sizes.  Since we're an ISP, we have to
keep our logs for a few years time, so the log file size can become an
issue.

A typical network stanza looks like:

         subnet 76.10.94.0 netmask 255.255.254.0 {
         pool {
               authoritative;
               range 76.10.94.20 76.10.95.200;
               min-lease-time 129600;
               max-lease-time 259200;
               default-lease-time 259200;
               option subnet-mask 255.255.254.0;
               option broadcast-address 76.10.95.255;
               option routers 76.10.94.1;
               }
         }

Thanks for any input!!

Alex
_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users
_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users
Reply | Threaded
Open this post in threaded view
|

Re: DHCPREQUEST flooding

Bob Harold

On Thu, May 5, 2016 at 10:51 AM, Patrick Trapp <[hidden email]> wrote:
Do the 300-ish devices share anything in particular in their configurations? Is the configuration you shared pertinent to some of your culprits?

Can you confirm that the ACK are reaching the devices? Do any of the devices lose their address entirely and have to be rebooted to get back on the network or is this issue literally only apparent to you and your logs?

Patrick

________________________________________
From: [hidden email] [[hidden email]] on behalf of Alex Moen [[hidden email]]
Sent: Thursday, May 05, 2016 9:40 AM
To: [hidden email]
Subject: DHCPREQUEST flooding

I am running a 4.2.5 ISC DHCP server (up-to-date via Centos 7
repository) for our ISP business.  We have around 7000 subscribers; most
with an el-cheapo router, a few with no router at all.  Most of our
customers are using a variant of Linksys router (Linksys, Cisco-Linksys,
Belkin, etc) because that is what we provide if they ask for a router.
However, this issue is not only a Linksys issue, as we are also seeing
PCs exhibiting the same behavior.

The issue is that we have a fairly large number of devices (around 300)
that are issuing DHCPREQUESTs at extremely short intervals (the worst, a
few second apart).  In the last 6 hours, some of these devices have
REQUESTed over 2000 times.  They are all being ACKed.

Is this a common problem that everyone sees, or do I have a config
issue?  This has actually been going on for a long, long time, and I am
just tired of the large log file sizes.  Since we're an ISP, we have to
keep our logs for a few years time, so the log file size can become an
issue.

A typical network stanza looks like:

         subnet 76.10.94.0 netmask 255.255.254.0 {
         pool {
               authoritative;
               range 76.10.94.20 76.10.95.200;
               min-lease-time 129600;
               max-lease-time 259200;
               default-lease-time 259200;
               option subnet-mask 255.255.254.0;
               option broadcast-address 76.10.95.255;
               option routers 76.10.94.1;
               }
         }

Thanks for any input!!

Alex
_______________________________________________

Are you using failover?  In failover mode, the first lease will be a shorter (mclt) time, but that should not affect renewals.

-- 
Bob Harold
 

_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users
Reply | Threaded
Open this post in threaded view
|

Re: DHCPREQUEST flooding

Alex Moen
On 05/05/2016 09:56 AM, Bob Harold wrote:

>
> On Thu, May 5, 2016 at 10:51 AM, Patrick Trapp <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Do the 300-ish devices share anything in particular in their
>     configurations? Is the configuration you shared pertinent to some of
>     your culprits?
>
>     Can you confirm that the ACK are reaching the devices? Do any of the
>     devices lose their address entirely and have to be rebooted to get
>     back on the network or is this issue literally only apparent to you
>     and your logs?
>
>     Patrick
>
>     ________________________________________
>     From: [hidden email]
>     <mailto:[hidden email]>
>     [[hidden email]
>     <mailto:[hidden email]>] on behalf of Alex Moen
>     [[hidden email] <mailto:[hidden email]>]
>     Sent: Thursday, May 05, 2016 9:40 AM
>     To: [hidden email] <mailto:[hidden email]>
>     Subject: DHCPREQUEST flooding
>
>     I am running a 4.2.5 ISC DHCP server (up-to-date via Centos 7
>     repository) for our ISP business.  We have around 7000 subscribers; most
>     with an el-cheapo router, a few with no router at all.  Most of our
>     customers are using a variant of Linksys router (Linksys, Cisco-Linksys,
>     Belkin, etc) because that is what we provide if they ask for a router.
>     However, this issue is not only a Linksys issue, as we are also seeing
>     PCs exhibiting the same behavior.
>
>     The issue is that we have a fairly large number of devices (around 300)
>     that are issuing DHCPREQUESTs at extremely short intervals (the worst, a
>     few second apart).  In the last 6 hours, some of these devices have
>     REQUESTed over 2000 times.  They are all being ACKed.
>
>     Is this a common problem that everyone sees, or do I have a config
>     issue?  This has actually been going on for a long, long time, and I am
>     just tired of the large log file sizes.  Since we're an ISP, we have to
>     keep our logs for a few years time, so the log file size can become an
>     issue.
>
>     A typical network stanza looks like:
>
>               subnet 76.10.94.0 netmask 255.255.254.0 {
>               pool {
>                     authoritative;
>                     range 76.10.94.20 76.10.95.200;
>                     min-lease-time 129600;
>                     max-lease-time 259200;
>                     default-lease-time 259200;
>                     option subnet-mask 255.255.254.0;
>                     option broadcast-address 76.10.95.255;
>                     option routers 76.10.94.1;
>                     }
>               }
>
>     Thanks for any input!!
>
>     Alex
>     _______________________________________________
>
>
> Are you using failover?  In failover mode, the first lease will be a
> shorter (mclt) time, but that should not affect renewals.
>
> --
> Bob Harold
>

No failover; standalone configuration.

--
Alex Moen
NSTII
North Dakota Telephone Company
701-662-6481
_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users
Reply | Threaded
Open this post in threaded view
|

Re: [Ext] DHCPREQUEST flooding

Jim Glassford
In reply to this post by Alex Moen
Hi Alex,

I blame poor IP stacks on the client device.
We get hammered by some wired game systems, mobile devices and once in a
while a printer. Printers I get them to review/update drivers or we do a
hard code on it, problem games systems and mobile devices are mostly
student devices so just live with the DHCPREQUESTS until they upgrade or
graduate. Not so easy for you.

best!
jim



On 5/5/2016 10:40 AM, Alex Moen wrote:

> I am running a 4.2.5 ISC DHCP server (up-to-date via Centos 7
> repository) for our ISP business.  We have around 7000 subscribers;
> most with an el-cheapo router, a few with no router at all.  Most of
> our customers are using a variant of Linksys router (Linksys,
> Cisco-Linksys, Belkin, etc) because that is what we provide if they
> ask for a router. However, this issue is not only a Linksys issue, as
> we are also seeing PCs exhibiting the same behavior.
>
> The issue is that we have a fairly large number of devices (around
> 300) that are issuing DHCPREQUESTs at extremely short intervals (the
> worst, a few second apart).  In the last 6 hours, some of these
> devices have REQUESTed over 2000 times.  They are all being ACKed.
>
> Is this a common problem that everyone sees, or do I have a config
> issue?  This has actually been going on for a long, long time, and I
> am just tired of the large log file sizes.  Since we're an ISP, we
> have to keep our logs for a few years time, so the log file size can
> become an issue.
>
> A typical network stanza looks like:
>
>         subnet 76.10.94.0 netmask 255.255.254.0 {
>         pool {
>               authoritative;
>               range 76.10.94.20 76.10.95.200;
>               min-lease-time 129600;
>               max-lease-time 259200;
>               default-lease-time 259200;
>               option subnet-mask 255.255.254.0;
>               option broadcast-address 76.10.95.255;
>               option routers 76.10.94.1;
>               }
>         }
>
> Thanks for any input!!
>
> Alex
> _______________________________________________
> dhcp-users mailing list
> [hidden email]
> https://lists.isc.org/mailman/listinfo/dhcp-users

_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users
Reply | Threaded
Open this post in threaded view
|

Re: DHCPREQUEST flooding

Alex Moen
In reply to this post by Patrick Trapp
On 05/05/2016 09:51 AM, Patrick Trapp wrote:
> Do the 300-ish devices share anything in particular in their configurations? Is the configuration you shared pertinent to some of your culprits?

Good question, and one that I did not fully address in my original
config. We're talking ISP customers here, in an aging, rural population.
I am certain that 99% of these devices are factory config (and probably
have never been updated) with the only change being a non-factory SSID
and possibly WPA config (although many don't want a password on their
wireless). As I did say, there are multiple generations of routers out
there (Linksys, Cisco, Cisco-Linksys, Belkin), so that doesn't seem to
indicate a particular model or firmware to target.

> Can you confirm that the ACK are reaching the devices?

We have confirmed that the ACK is being sent from the access gear out
the customer's interface to the customer's device.  I can't confirm any
further than that without actually going to the customer's premise and
performing some captures.

> Do any of the devices lose their address entirely and have to be rebooted to get back on the network or is this issue literally only apparent to you and your logs?

We haven't had any customer complaints indicating that they are needing
reboots.  It looks like it's only impacting the logs and not the
customer's experience.


> ________________________________________
> From: [hidden email] [[hidden email]] on behalf of Alex Moen [[hidden email]]
> Sent: Thursday, May 05, 2016 9:40 AM
> To: [hidden email]
> Subject: DHCPREQUEST flooding
>
> I am running a 4.2.5 ISC DHCP server (up-to-date via Centos 7
> repository) for our ISP business.  We have around 7000 subscribers; most
> with an el-cheapo router, a few with no router at all.  Most of our
> customers are using a variant of Linksys router (Linksys, Cisco-Linksys,
> Belkin, etc) because that is what we provide if they ask for a router.
> However, this issue is not only a Linksys issue, as we are also seeing
> PCs exhibiting the same behavior.
>
> The issue is that we have a fairly large number of devices (around 300)
> that are issuing DHCPREQUESTs at extremely short intervals (the worst, a
> few second apart).  In the last 6 hours, some of these devices have
> REQUESTed over 2000 times.  They are all being ACKed.
>
> Is this a common problem that everyone sees, or do I have a config
> issue?  This has actually been going on for a long, long time, and I am
> just tired of the large log file sizes.  Since we're an ISP, we have to
> keep our logs for a few years time, so the log file size can become an
> issue.
>
> A typical network stanza looks like:
>
>           subnet 76.10.94.0 netmask 255.255.254.0 {
>           pool {
>                 authoritative;
>                 range 76.10.94.20 76.10.95.200;
>                 min-lease-time 129600;
>                 max-lease-time 259200;
>                 default-lease-time 259200;
>                 option subnet-mask 255.255.254.0;
>                 option broadcast-address 76.10.95.255;
>                 option routers 76.10.94.1;
>                 }
>           }
>
> Thanks for any input!!
>
> Alex
> _______________________________________________
> dhcp-users mailing list
> [hidden email]
> https://lists.isc.org/mailman/listinfo/dhcp-users
> _______________________________________________
> dhcp-users mailing list
> [hidden email]
> https://lists.isc.org/mailman/listinfo/dhcp-users
>

--
Alex Moen
NSTII
North Dakota Telephone Company
701-662-6481
_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users
Reply | Threaded
Open this post in threaded view
|

Re: DHCPREQUEST flooding

José Queiroz
Hi Alex,

2016-05-05 12:57 GMT-03:00 Alex Moen <[hidden email]>:
On 05/05/2016 09:51 AM, Patrick Trapp wrote:
Do the 300-ish devices share anything in particular in their configurations? Is the configuration you shared pertinent to some of your culprits?

Good question, and one that I did not fully address in my original config. We're talking ISP customers here, in an aging, rural population. I am certain that 99% of these devices are factory config (and probably have never been updated) with the only change being a non-factory SSID and possibly WPA config (although many don't want a password on their wireless). As I did say, there are multiple generations of routers out there (Linksys, Cisco, Cisco-Linksys, Belkin), so that doesn't seem to indicate a particular model or firmware to target.

Can you confirm if this behaviour is not related to PC-devices? Because those devices could be infected by some malware.
You can see this looking in the OUI part of the MAC addresses on the requests.


_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users
Reply | Threaded
Open this post in threaded view
|

Re: DHCPREQUEST flooding

Alex Moen
The vast majority of the clients are el-cheapo routers.  There are a few
PCs (or, at least, non routers) in there: 5 out of the 300 or so.


On 05/05/2016 11:11 AM, José Queiroz wrote:

> Hi Alex,
>
> 2016-05-05 12:57 GMT-03:00 Alex Moen <[hidden email]
> <mailto:[hidden email]>>:
>
>     On 05/05/2016 09:51 AM, Patrick Trapp wrote:
>
>         Do the 300-ish devices share anything in particular in their
>         configurations? Is the configuration you shared pertinent to
>         some of your culprits?
>
>
>     Good question, and one that I did not fully address in my original
>     config. We're talking ISP customers here, in an aging, rural
>     population. I am certain that 99% of these devices are factory
>     config (and probably have never been updated) with the only change
>     being a non-factory SSID and possibly WPA config (although many
>     don't want a password on their wireless). As I did say, there are
>     multiple generations of routers out there (Linksys, Cisco,
>     Cisco-Linksys, Belkin), so that doesn't seem to indicate a
>     particular model or firmware to target.
>
>
> Can you confirm if this behaviour is not related to PC-devices? Because
> those devices could be infected by some malware.
> You can see this looking in the OUI part of the MAC addresses on the
> requests.
>
>
>
> _______________________________________________
> dhcp-users mailing list
> [hidden email]
> https://lists.isc.org/mailman/listinfo/dhcp-users
>

--
Alex Moen
NSTII
North Dakota Telephone Company
701-662-6481
_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users
Reply | Threaded
Open this post in threaded view
|

Re: DHCPREQUEST flooding

dave c
In reply to this post by Alex Moen
I've been seeing devices that behave as the original author described forever it seems. Since
way back when I ran the network at Vassar College at the turn of the millennium. We refer to
devices (or workstations when it's directly Windows doing it) as going into jealous girlfriend
mode, constant calls asking if the DHCP server still loves the CPE device etc... :)

It would be interesting, and I'll admit I've not taken the time myself to do this research, to
parse the logs, count the DHCPACKs to each particular mac address, sort by highest and then do
some research on the vendor portion of the highest of the MAC addresses found to be hyper
requesting. My gut tells me it will be mostly Belkin devices as they've been the root cause of
so many of my CPE device problems over the years, but it would be nice to know if it was indeed
limited to only one or two particular vendors responsible for it. It wouldn't mean I could ban
that equipment, but I could at least have the satisfaction of knowing who was responsible.

As for the logging issue, we have about 24k users under DHCP and send DHCP logs to twin syslog
servers (in the event one misses a log packet for whatever reason) and keep them for 400 days.
We rotate daily and immediately compress them and name the log files with a date/time sequence
for ease of our folks searching an old log file to answer an external inquiry. We seem to hover
at about 128-140mb per day after compression and the whole year or so is sitting at 41gb so not
an overly huge storage commitment. A pair of $49 SSDs would hold my logs and still have almost
80gb free for other storage and larger SSDs are not much more expensive than the 120gb versions.

Dave C

On 5/5/16 10:57, Alex Moen wrote:

> On 05/05/2016 09:51 AM, Patrick Trapp wrote:
>> Do the 300-ish devices share anything in particular in their configurations? Is the
>> configuration you shared pertinent to some of your culprits?
>
> Good question, and one that I did not fully address in my original config. We're talking ISP
> customers here, in an aging, rural population. I am certain that 99% of these devices are
> factory config (and probably have never been updated) with the only change being a non-factory
> SSID and possibly WPA config (although many don't want a password on their wireless). As I did
> say, there are multiple generations of routers out there (Linksys, Cisco, Cisco-Linksys,
> Belkin), so that doesn't seem to indicate a particular model or firmware to target.
>
>> Can you confirm that the ACK are reaching the devices?
>
> We have confirmed that the ACK is being sent from the access gear out the customer's interface
> to the customer's device.  I can't confirm any further than that without actually going to the
> customer's premise and performing some captures.
>
>> Do any of the devices lose their address entirely and have to be rebooted to get back on the
>> network or is this issue literally only apparent to you and your logs?
>
> We haven't had any customer complaints indicating that they are needing reboots.  It looks like
> it's only impacting the logs and not the customer's experience.
>
>
>> ________________________________________
>> From: [hidden email] [[hidden email]] on behalf of Alex
>> Moen [[hidden email]]
>> Sent: Thursday, May 05, 2016 9:40 AM
>> To: [hidden email]
>> Subject: DHCPREQUEST flooding
>>
>> I am running a 4.2.5 ISC DHCP server (up-to-date via Centos 7
>> repository) for our ISP business.  We have around 7000 subscribers; most
>> with an el-cheapo router, a few with no router at all.  Most of our
>> customers are using a variant of Linksys router (Linksys, Cisco-Linksys,
>> Belkin, etc) because that is what we provide if they ask for a router.
>> However, this issue is not only a Linksys issue, as we are also seeing
>> PCs exhibiting the same behavior.
>>
>> The issue is that we have a fairly large number of devices (around 300)
>> that are issuing DHCPREQUESTs at extremely short intervals (the worst, a
>> few second apart).  In the last 6 hours, some of these devices have
>> REQUESTed over 2000 times.  They are all being ACKed.
>>
>> Is this a common problem that everyone sees, or do I have a config
>> issue?  This has actually been going on for a long, long time, and I am
>> just tired of the large log file sizes.  Since we're an ISP, we have to
>> keep our logs for a few years time, so the log file size can become an
>> issue.
>>
>> A typical network stanza looks like:
>>
>>           subnet 76.10.94.0 netmask 255.255.254.0 {
>>           pool {
>>                 authoritative;
>>                 range 76.10.94.20 76.10.95.200;
>>                 min-lease-time 129600;
>>                 max-lease-time 259200;
>>                 default-lease-time 259200;
>>                 option subnet-mask 255.255.254.0;
>>                 option broadcast-address 76.10.95.255;
>>                 option routers 76.10.94.1;
>>                 }
>>           }
>>
>> Thanks for any input!!
>>
>> Alex
>> _______________________________________________
>> dhcp-users mailing list
>> [hidden email]
>> https://lists.isc.org/mailman/listinfo/dhcp-users
>> _______________________________________________
>> dhcp-users mailing list
>> [hidden email]
>> https://lists.isc.org/mailman/listinfo/dhcp-users
>>
>

--
Dave Calafrancesco
_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users
Reply | Threaded
Open this post in threaded view
|

Re: DHCPREQUEST flooding

A.L.M.Buxey
In reply to this post by Alex Moen
hi,

you know this rings some bells.... I'm sure we've faced an issue with dodgy clients
just not accepting reasonable lease times -  you've got 3 days...which is fine...
I'd suggets you try to bring that down to 86400 or 14400   - after all, these clients
are hitting your server frequently anyway so if they DO work with those values then
their half-life reconfirm will still be less than what you are facing.

...interestingly I've never thought about having authoritative within a pool scope....


alan
_______________________________________________
dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users