docker swarm: dns resolution fails

[sameo]

From @mcastelino on April 27, 2017 17:37

When running Clear Container based containers in a docker swarm, DNS resolution does not work both for internal and external DNS when the DNS resolution is performed from within the Clear Container.

This is due to the way the DNS resolution is implemented within docker swarm.

DNS Resolution in Swarm

All docker swarm containers have the DNS resolver set to 127.0.0.11:53

Docker swarm has an internal DNS based load balancer that RRs the DNS requests to spread load.
That runs on the localhost on the host bound to a host port specific to the container.
https://github.com/docker/libnetwork/blob/5ac04367ae7b0b12c33bed5f5b395bd4c104fff9/sandbox.go#L815

There is a iptables rule injected into the container namespace which is used to implement the docker DNS load balancer/resolver. That way 127.0.0.11:53 maps to a specific port on which the corresponding resolver is running.

	-A DOCKER_OUTPUT -d 127.0.0.11/32 -p tcp -m tcp --dport 53 -j DNAT --to-destination 127.0.0.11:41343
	-A DOCKER_OUTPUT -d 127.0.0.11/32 -p udp -m udp --dport 53 -j DNAT --to-destination 127.0.0.11:43411
	-A DOCKER_POSTROUTING -s 127.0.0.11/32 -p tcp -m tcp --sport 41343 -j SNAT --to-source :53
	-A DOCKER_POSTROUTING -s 127.0.0.11/32 -p udp -m udp --sport 43411 -j SNAT --to-source :53

Here the DNS request is NATed to a container specific TCP and UDP port.

The resolver in this case is dockerd

 netstat -plunt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.11:41343        0.0.0.0:*               LISTEN      14447/dockerd
udp        0      0 127.0.0.11:43411        0.0.0.0:*                           14447/dockerd

In the case of clear containers there is currently no way for the DNS request from within the VM to talk to the dockerd running on the host side. The only host connectivity that the VM has is via the docker_gwbridge. However the DNS resolver running within the network namespace is not reachable via the VM.

Network setup with Clear Containers

+---------------------------------+                +--------------------------------+
|   ingress sbox                  |                |                                |
|                    +            |                |                       +        |
|                    +-----------------------------------------------------+        |
|            I IP    |            |                |                       +--------------+
|                    +----+       |                |           +-----------+        |
|                    +    |       |                |       over|ay box     |        |
|                         |       |                |           |                    |
+---------------------------------+                +--------------------------------+
                          |                                    |
                          |                                    |
                          |                                    |            host container ns
                          |                   +--------------------------------------------+
                          |                   |        +-+     |                           |
                          |                   |        | +-----+     +-----------------+   |
                          |                   |        | |           |    IP           |   |
                          |                   |        | +--------------+ VIP          |   |
                          |         Resolver-----+     +-+           |                 |   |
     docker_gw_bridge     |       127.0.0.11  |                      |                 |   |
               +          |                   |       +-+R IP        |                 |   |
               +----------+                   |       | +---------------+ HIP          |   |
       H GW IP +--------------------------------------+ |            |                 |   |
               |                              |       +-+            +-----------------+   |
               +       default gw             |                    /etc/resolv.conf (127..)|
                                              +--------------------------------------------+

Internal DNS Resolution

Internal DNS resolution is handled completely by dockerd. So dockerd directly responds to the DNS request from the container process for any cluster local resource.

External DNS Resolution

External DNS resolution is not handled by dockerd. When dockerd is unable to resolve the name to a cluster local resource it will then perform a DNS resolution using the host's resolv.conf.

Hence the DNS resolution process for external name is

process -> dockerd:43411  -> host (via the namespace) -> external DNS -> host -> dockerd -> container.

Here you will notice, dockerd sends packets out from within the namespace to the host via the interface bound to the docker_gwbridge.

In the case of clear containers as there is network connectivity between the container network namespace and the host, this request can never be fulfilled.

Work around for External DNS

For external DNS resolution, the resolv.conf can be updated to point to a external DNS resolver. This will ensure that the external DNS resolution works

Fixing this issue in Clear Containers

The long term plan is to proxy the internal DNS requests from within the VM to dockerd. On failure of the DNS resolution the resolution has to be performed from within the VM to the host resolver. However assuming that the host resolver is the right resolver to use in the case of dockerd resolution failure may not be a correct assumption. Also this results in longer resolution times as dockerd takes a significant amount of time to fail the external DNS request.

Copied from original issue: intel/cc-oci-runtime#854

[sameo]

From @mcastelino on April 27, 2017 17:56

@devimc We need to add this to our release notes.

[sameo]

From @devimc on April 27, 2017 18:49

great! thanks @mcastelino and nice description

[mcastelino]

Initial proposal for support for DNS resolution for clear containers with docker swarm.

The internal DNS resolution can be supported by running a DNS proxy agent within each Clear Container VM, which then forwards the request to the host side network namespace to the docker DNS resolver. This can be done in one of two ways

• Assign a unique IP address to the integration bridge (which is not assigned an IP today), and use that to send the DNS requests to the host side namespace (this has been been verified to work using socat)
  • Note: Rancher VM also does something similar
• Send the packets using a custom protocol over a virtual serial port and then run an agent in the host namespace to further forward the request to the docker DNS resolver.

However neither of these methods will solve the resolution of external DNS. Today the docker DNS resolver running in the host namespace does another round of DNS resolution using the host DNS configuration for external DNS, but it does the same through the namespace. In the case of clear containers there is no network connectivity between the host and the host side container namespace.

To support this there are two options

• Add the host side DNS resolution entries to the container resolv.conf. Doing so results in the container trying the query via the host through the VM (this has been verified to work).
  - However this means we are now modifying the resolv.conf which was created by docker. And we need to do this only in the case of docker swarm.
  - Also it takes a long time to resolve, as the docker DNS resolution has to fail, which takes time
• The dns proxy agent we run inside the VM does this resolution through the host.

The challenge is that DNS works fine when not running in a swarm (and it works when we run CC in kubernetes). So the agent will need to detect this and only activate itself when running in a swarm which will end up being a very specific hack!

If there are better ideas I would welcome them before I jump into this implementation
@sameo @dlespiau @rbradford

[egernst]

@mcastelino @sameo -- we need a refresh on this: the way I see it we either:

1. move as will not fix in CC and open in Kata, or ...
2. reevaluate and implement one of the above options

(sorry if I am mistaken and this is already resolved - do we change the resolv.conf today?)