dhcpd and relayed responses from multi-interface Linux server

[Neufeld, Keith]

Does anyone have dhcpd (server, not client) successfully running on a Linux server built in the last few years (recent enough to use LPF, which you can see in the log file at server start) in an environment with relayed clients?

I'm surprised if this hasn't been covered before, but my search-fu isn't good enough to find it in the list archives.


We're prepping a pair of new ISC DHCP servers on RHEL 8.  We originally tried RH's bundled DHCP 4.1.5 and have since updated to 4.4.2, with the same results (described below)

The DHCP servers have two interfaces (I'm renaming them from their autogenerated names for conversational simplicity), one for administrative access and one for application service access:

eth0 -- 10.1.61.16 -- admin
eth1 -- 10.1.69.16 -- service

We have subnet declarations for each interface because we need to serve clients on each of those subnets; but dhcpd is being invoked with eth1 on its command line and appears to be listening only on eth1 as desired:

Jun  1 16:26:18 netsvc-516 dhcpd[704228]: Listening on LPF/eth1/00:50:56:be:43:d6/10.1.69.0/24
Jun  1 16:26:18 netsvc-516 dhcpd[704228]: Sending on   LPF/eth1/00:50:56:be:43:d6/10.1.69.0/24
Jun  1 16:26:18 netsvc-516 dhcpd[704228]: Sending on   Socket/fallback/fallback-net

The problem is with the "fallback" UDP socket used for transmitting unicast messages, when it needs to send responses to relayed DISCOVERs.  In common/socket.c, in the function maybe_setup_fallback(), a UDP INET socket is created but not bound to any specific interface.  Then when packets are transmitted out the fallback socket, it appears that the behavior is to use the server host's route table to decide which interface to send the packet out of, *and to use that interface's IP as the source IP*.

Since our servers are set up with the default route out eth0, eth0's IP gets used as the source IP on the relay responses (OFFERs).  It appears that the relay agent in our LAN routers keeps a state table and checks the source IP of relay responses.  Since the OFFER is coming from a different IP (10.1.61.16) than the DISCOVER was relayed to (10.1.69.16), the router doesn't relay the response back to the client and the client never completes the conversation to get a lease.

When we temporarily change the server host's default route to eth1, the client gets the relayed responses, continues the conversation (broadcast/relay REQUEST/ACK), and gets its lease as it normally would.


It seems to me that our scenario -- a multi-interface server running LPF plus fallback socket and needing to handle relayed communication -- is bound to fail under the current code.  (Therefore not many people must be doing it. :-) )

It seems to me that fixing it would require building an array of UDP INET sockets (for the sake of conversation, call it the fallback array, though maybe conceptually it's no longer fallback), one on each interface on which dhcpd is listening; and matching unicast transmissions, at least *relayed* ones, to send out the interface on which they were received in order to ensure that their source IP is correct.

It's been way too long since I studied sockets for me to remember whether it's possible that dhcpd could instead write the desired source IP into the response packet and it would be preserved by the server host even when sending it out a different interface.  If this would work, it would be a much less invasive code change.


Has this come up on the list before?  Anyone else run across it?  Am I overlooking a simple workaround (other than changing our server routing or running dhcpd in a single-interface virtual jail)?

--
Keith Neufeld
Director of Networking and Telecommunications
Wichita State University

_______________________________________________
ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.

dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

[Simon Hobson]

Neufeld, Keith <[hidden email]> wrote:

> The DHCP servers have two interfaces (I'm renaming them from their autogenerated names for conversational simplicity), one for administrative access and one for application service access:
>
> eth0 -- 10.1.61.16 -- admin
> eth1 -- 10.1.69.16 -- service
>
> We have subnet declarations for each interface because we need to serve clients on each of those subnets; but dhcpd is being invoked with eth1 on its command line

Is there a specific reason for doing it this way ? While it's easy to reply with what is in effect "you're building the network wrong", that does seem to be the fundamental issue here.

> and appears to be listening only on eth1 as desired:
>
> Jun  1 16:26:18 netsvc-516 dhcpd[704228]: Listening on LPF/eth1/00:50:56:be:43:d6/10.1.69.0/24
> Jun  1 16:26:18 netsvc-516 dhcpd[704228]: Sending on   LPF/eth1/00:50:56:be:43:d6/10.1.69.0/24
> Jun  1 16:26:18 netsvc-516 dhcpd[704228]: Sending on   Socket/fallback/fallback-net
>
> The problem is with the "fallback" UDP socket used for transmitting unicast messages, when it needs to send responses to relayed DISCOVERs.  In common/socket.c, in the function maybe_setup_fallback(), a UDP INET socket is created but not bound to any specific interface.  Then when packets are transmitted out the fallback socket, it appears that the behavior is to use the server host's route table to decide which interface to send the packet out of, *and to use that interface's IP as the source IP*.
...
> Has this come up on the list before?

I don't recall seeing this before on the list. But then it would be normal to serve the locally connected devices directly rather than via a relay agent - hence few would hit the problem.

I did start off setting out a couple of options that I thought might work, but while expanding my thoughts realised that they wouldn't achieve what's needed.

Policy based routing won't work because, I think, source address selection occurs before the routing could have an effect.

Disabling packet filter operations (needs a compile with an option set) which can deal with some classes of problem, wouldn't get round the issue.

It might, you'd need to do the homework, be possible to apply some mangle rules to change the source address - i.e. for UDP from port 67 on 10.1.61.16, change (mangle) the source address to 10.1.69.16. You'd then also need to apply a routing rule to force the packet to go out of eth1 rather than out of eth0.
I think it might be doable in the Output Mangle table looking at https://shorewall.org/NetfilterOverview.html

Hoep this is of some help, Simon

_______________________________________________
ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.

dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

[Neufeld, Keith]

> On Jun 3, 2020, at 09:44 , Simon Hobson <[hidden email]> wrote:
>
>> The DHCP servers have two interfaces ..., one for administrative access and one for application service access:
>>
>> eth0 -- 10.1.61.16 -- admin
>> eth1 -- 10.1.69.16 -- service
>>
>> We have subnet declarations for each interface because we need to serve clients on each of those subnets; but dhcpd is being invoked with eth1 on its command line
>
> Is there a specific reason for doing it this way ? While it's easy to reply with what is in effect "you're building the network wrong", that does seem to be the fundamental issue here.

Our server team operates many servers with multiple interfaces ... as have I over the last thirty years ... and I haven't regarded it as "building the network wrong."  Could you say more about how that's inappropriate?

>> and appears to be listening only on eth1 as desired:
>>
>> Jun  1 16:26:18 netsvc-516 dhcpd[704228]: Listening on LPF/eth1/00:50:56:be:43:d6/10.1.69.0/24
>> Jun  1 16:26:18 netsvc-516 dhcpd[704228]: Sending on   LPF/eth1/00:50:56:be:43:d6/10.1.69.0/24
>> Jun  1 16:26:18 netsvc-516 dhcpd[704228]: Sending on   Socket/fallback/fallback-net
>>
>> The problem is with the "fallback" UDP socket used for transmitting unicast messages, when it needs to send responses to relayed DISCOVERs.  In common/socket.c, in the function maybe_setup_fallback(), a UDP INET socket is created but not bound to any specific interface.  Then when packets are transmitted out the fallback socket, it appears that the behavior is to use the server host's route table to decide which interface to send the packet out of, *and to use that interface's IP as the source IP*.
> ...
>> Has this come up on the list before?
>
> I don't recall seeing this before on the list. But then it would be normal to serve the locally connected devices directly rather than via a relay agent - hence few would hit the problem.

Oh, you've mistaken my meaning.  We expect local devices to be served locally.

At least on this platform, dhcpd does not respond correctly to *any* relayed message to a multi-interface DHCP server unless the messages are relayed to the interface that is the server host's default route out.

This means, for example, that you cannot relay to multiple interfaces on the same DHCP server ... which I would like to be able to do temporarily while migrating the server from an old IP in an old subnet to a new IP in a new subnet, serving both at the same time while phasing the updates of the routers' IP helper / DHCP relay statements.

I'd expected others might have other use cases; but from the lack of prior conversation about this, apparently not.

--
Keith Neufeld
Director of Networking and Telecommunications
Wichita State University

_______________________________________________
ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.

dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

[Simon Hobson]

Neufeld, Keith <[hidden email]> wrote:

>>> The DHCP servers have two interfaces ..., one for administrative access and one for application service access:
>>>
>>> eth0 -- 10.1.61.16 -- admin
>>> eth1 -- 10.1.69.16 -- service
>>>
>>> We have subnet declarations for each interface because we need to serve clients on each of those subnets; but dhcpd is being invoked with eth1 on its command line
>>
>> Is there a specific reason for doing it this way ? While it's easy to reply with what is in effect "you're building the network wrong", that does seem to be the fundamental issue here.
>
> Our server team operates many servers with multiple interfaces ... as have I over the last thirty years ... and I haven't regarded it as "building the network wrong."  Could you say more about how that's inappropriate?

In this case I think there is an argument that you are building the network wrong. I read that you have devices that are connected to the network on eth0 but are being services via relay agents forwarding requests via the network on eth1. In your original message you wrote :
> We have subnet declarations for each interface because we need to serve clients on each of those subnets; but dhcpd is being invoked with eth1 on its command line

and so I assumed we were talking about clients in the eth0 network.

For these, relaying requests that could be handled directly could be considered "building the network wrong".


> Oh, you've mistaken my meaning.  We expect local devices to be served locally.
>
> At least on this platform, dhcpd does not respond correctly to *any* relayed message to a multi-interface DHCP server unless the messages are relayed to the interface that is the server host's default route out.
>
> This means, for example, that you cannot relay to multiple interfaces on the same DHCP server ... which I would like to be able to do temporarily while migrating the server from an old IP in an old subnet to a new IP in a new subnet, serving both at the same time while phasing the updates of the routers' IP helper / DHCP relay statements.

Ah, now I understand !

I think the simple answer is that it should not matter what source address is used - only the destination address which routes the packet back to the relay agent. In terms of protocol spec, that is the case - there's an implicit assumption that regardless of the source address, the routing will get it to where it needs to be.
Your problem is that your relay agents impose restrictions which are outside of the spec - and yes, I understand your problem now. It's neither the multi-homing nor the relay security that's causing the problem - but the combination of both of them. I suspect there will be many setups with one or the other in place - but few with both, and hence the problem you see.

The only solution I can think of is as I described in my earlier post - use the output mangle table to alter the source address. In a "stable" setup I think it should be as simple as "protocol=UDP and source port=67" -> "set source address to 10.1.69.16". As long as the server only has one address it uses for DHCP service then that ought to work.

Now, for your potential migration setup I think it gets a bit more complicated. You might need to expand that logic along the lines of :
"protocol=UDP and source port=67 and dest IP in 10.0.0.0/8" -> "set source address to 10.1.69.16"
"protocol=UDP and source port=67 and dest IP in 172.16.0.0/12" -> "set source address to 172.17.69.16"

Is that any help ?



There is one thing that also comes to mind. AIUI the ISC DHCP server is more or less out of active development, with Kia having taken over as the actively developed ISC DCHP server going forwards. TBH I think it's done marvellous duty seeing as it was originally intended as a reference implementation of the RFC.
I've not kept up with how Kia is doing - last I read it was ahead on some features and behind on others. It might be worth looking into whether Kia operates the same or differently in this respect, and if it has the same issue, whether ISC might take on the problem as a feature request (though their policy is AIUI, "you want it, you sponsor it"). If you have some bright CS people in the Uni, could it become part of a project for a small group of them ?

Just tossing ideas out there in case any of them are suitable for your situation.

_______________________________________________
ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.

dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users

[sthaug]

> At least on this platform, dhcpd does not respond correctly to *any* relayed message to a multi-interface DHCP server unless the messages are relayed to the interface that is the server host's default route out.

This is certainly not the case for *all* platforms that ISC DHCP runs
on. I don't have these problems with FreeBSD, for instance.

Steinar Haug, Nethelp consulting, [hidden email]
_______________________________________________
ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.

dhcp-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/dhcp-users