Add support for managing compute resources

[mcastelino]

Kubernetes supports the notion of CPU resources assigned to a container

spec.containers[].resources.limits.cpu
...
Although requests and limits can only be specified on individual Containers, it is convenient to talk about Pod resource requests and limits. A Pod resource request/limit for a particular resource type is the sum of the resource requests/limits of that type for each Container in the Pod.
...
Limits and requests for CPU resources are measured in cpu units. One cpu, in Kubernetes, is equivalent to:
1 AWS vCPU
1 GCP Core
1 Azure vCore
1 Hyperthread on a bare-metal Intel processor with Hyperthreading

Similarly in docker we have

--cpus |   | Number of CPUs
--cpu-period | 0 | Limit CPU CFS (Completely Fair Scheduler) period
--cpu-quota | 0 | Limit CPU CFS (Completely Fair Scheduler) quota
--cpu-rt-period | 0 | Limit CPU real-time period in microseconds
--cpu-rt-runtime | 0 | Limit CPU real-time runtime in microseconds
--cpu-shares, -c | 0 | CPU shares (relative weight)
--cpus |   | Number of CPUs
--cpuset-cpus |   | CPUs in which to allow execution (0-3, 0,1)
--cpuset-mems |   | MEMs in which to allow execution (0-3, 0,1)

Which manifests in the config file as

// LinuxCPU for Linux cgroup 'cpu' resource management
type LinuxCPU struct {
	// CPU shares (relative weight (ratio) vs. other cgroups with cpu shares).
	Shares *uint64 `json:"shares,omitempty"`
	// CPU hardcap limit (in usecs). Allowed cpu time in a given period.
	Quota *int64 `json:"quota,omitempty"`
	// CPU period to be used for hardcapping (in usecs).
	Period *uint64 `json:"period,omitempty"`
	// How much time realtime scheduling may use (in usecs).
	RealtimeRuntime *int64 `json:"realtimeRuntime,omitempty"`
	// CPU period to be used for realtime scheduling (in usecs).
	RealtimePeriod *uint64 `json:"realtimePeriod,omitempty"`
	// CPUs to use within the cpuset. Default is to use any CPU available.
	Cpus string `json:"cpus,omitempty"`
	// List of memory nodes in the cpuset. Default is to use any available memory node.
	Mems string `json:"mems,omitempty"`
}

The clear containers runtime should support at the very least the --cpus option.

However in the case of runc the CPU's visible to the application is the full set available to the host.
In the case or runc the cpu cgroups are used to actually constrain the process.

The mimic this behavior such that the application seems very similar availability of vCPU's for threading we should

• allocate the number of vCPU == number of physical CPU's on the host
• constrain QEMU to limit the CPU consumption

However on systems with very large number of CPUs, hence adding up to larger memory consumption per container, we can consider limiting the number of vCPU to equal the requests CPU shares, while still applying cgroups.

/cc @dlespiau @egernst

[mcastelino]

For the first cut we can skip cgroups and just make the CPU and Memory sizes passed to QEMU configurable. Adding cgroup support can come subsequently.

/cc @amyleeland

[egernst]

That is handled today in a different issue / PR: #297 From: Manohar Castelino [mailto:[email protected]] Sent: Tuesday, July 18, 2017 1:13 PM To: clearcontainers/runtime <[email protected]> Cc: Ernst, Eric <[email protected]>; Mention <[email protected]> Subject: Re: [clearcontainers/runtime] Add support for managing compute resources (#341) For the first cut we can skip cgroups and just make the CPU and Memory sizes passed to QEMU configurable. Adding cgroup support can come subsequently. /cc @amyleeland<https://github.com/amyleeland> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#341 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ATPXW-cu0USNsUXno_lWen2DVaO-c4CDks5sPRGvgaJpZM4ObtCw>.

[mcastelino]

@egernst #297 is not exactly this. It only sets per k8s node, not at an individual POD level. Also we need to account for the fact that in k8s POD = sum of container constraints...

[dvoytik]

@mcastelino, with #297 you can setup for each new POD (== VM) default number of vCPUs. If you set -1 then runtime will set the same number as physical cores on the host.
https://github.com/clearcontainers/runtime/blob/master/config/configuration.toml.in#L9

[mcastelino]

@dvoytik #297 provides us with the right degree of flexibility from a machine level default point of view.

IMO the default on non NUMA systems should be -1 so reflect more closely the behavior of a namespaced container (once we evaluate the memory footprint impact)

What we are missing is honoring the per container/POD values that come in via config.json in the case of docker. We are also missing setting up the cgroups setup to constrain QEMU itself.

So a valid configuration would be -1 and constrain the QEMU using cgroups. This will give us behavior similar to runc.

[jodh-intel]

Related: #19.

[wcwxyz]

Hi, I'd like to work on this one if it's not taken.

--cpu is a docker specific option. It is a equivalent to a combination of --cpu-quota and --cpu-period, eg. --cpu 1.5 is translated to --cpu-quota 150000 --cpu-period 100000. OCI spec defines cpu quota and period.

My idea is to get cpu quota and period from container OCI spec, and make the calculation of number of vcpus for qemu.

But obviously, --cpus 1.5 doesn't make sense from qemu point of view. That's a limitation. We should round down/up to integer.

Any ideas?

[grahamwhaley]

Hi @wcwxyz , sure, if you want it we can assign. I think you'd work closely with @mcastelino and @dvoytik on the details of how this interacts as a whole, with k8s and pods for instance (see above).
As @mcastelino says, adding support for --cpus is a first good step, and then later for POD support it looks like we'd add support for cgroup cpu constraining within the VM itself.

We should probably open an Issue specifically for adding non-POD --cpus support to separate the discussion a little for now from the POD/cgroup stuff. Let me do that in a minute...

I was contemplating how we map say --cpus=1.5 onto QEMU -smp cpus argument last week when we were writing https://github.com/clearcontainers/runtime/blob/master/docs/limitations.md#docker-run---cpus. Indeed, it is not a nice 1:1 mapping. As you say, we should round up/down. I vote for round-up as the 'path of least surprise'. I'd be more surprised if I got less CPU than I asked for than if I got more I think.
For reference (for everybody and the future), the docker docs I believe you referenced for the --cpus=1.5 example are here:
https://docs.docker.com/engine/admin/resource_constraints/#cpu
and they say that --cpus should be used in preference to --cpu-period and --cpu-quota, so I'd currently classify those two options as potentially deprecated.

[wcwxyz]

@grahamwhaley emm.. I'm not familiar with k8s. I mean to work on #393 ...

For reference (for everybody and the future), the docker docs I believe you referenced for the --cpus=1.5 example are here:
https://docs.docker.com/engine/admin/resource_constraints/#cpu
and they say that --cpus should be used in preference to --cpu-period and --cpu-quota, so I'd currently classify those two options as potentially deprecated.

Actually --cpus will be translated to CFS cpu period and quota in the OCI spec config.json. See https://github.com/opencontainers/runtime-spec/blob/master/specs-go/config.go#L292~L307. I'll quote it here:

type LinuxCPU struct {
	// CPU shares (relative weight (ratio) vs. other cgroups with cpu shares).
	Shares *uint64 `json:"shares,omitempty"`
	// CPU hardcap limit (in usecs). Allowed cpu time in a given period.
	Quota *int64 `json:"quota,omitempty"`
	// CPU period to be used for hardcapping (in usecs).
	Period *uint64 `json:"period,omitempty"`
	// How much time realtime scheduling may use (in usecs).
	RealtimeRuntime *int64 `json:"realtimeRuntime,omitempty"`
	// CPU period to be used for realtime scheduling (in usecs).
	RealtimePeriod *uint64 `json:"realtimePeriod,omitempty"`
	// CPUs to use within the cpuset. Default is to use any CPU available.
	Cpus string `json:"cpus,omitempty"`
	// List of memory nodes in the cpuset. Default is to use any available memory node.
	Mems string `json:"mems,omitempty"`
}

So when runtime supports --cpus, it naturally supports --cpu-period xxx --cpu-quota xxx combination.

Anyway #393 seems to be a better place to discuss how we handle --cpu.

[jodh-intel]

Related:

• docker run -m:
  Support docker run -m MEMORY memory resource constraint/definition #381
  pkg: oci: support OCI spec memory limit containers/virtcontainers#341

• docker run {--cpu-quota|--cpu-period}:
  vendor: Update virtcontainers vendoring #436
  pkg: oci: support OCI spec cpu quota/period containers/virtcontainers#344