Hot-plug (and unplug) devices into a container as they are (un)plugged.

Docker provides the --device flag to give a container access to a device. However the devices specified this way must be present when the container is created.

For dynamically created devices Docker provides the --device-cgroup-rule. However this requires knowing the device's major and minor numbers, which are dynamically allocated by the kernel. The rule accepts a glob * to mean "any minor" or "any major". However this would still give the container access to all the devices handled by a particular driver.

This program tries to solve that problem by listening to udev events to detect when a device is (un)plugged. It then interfaces directly with the container's cgroup to grant it access to that specific device.

To limit the devices the container can access, root devices are specified. The container will receive access to any device descending from one of the root devices. This is particularly useful if the root device is set to a USB hub. The hub can be specified directly, or it can be specified as "the parent of device X", e.g., we can giving a container access to all devices connected to the same hub as an Arduino board.

Another concern is providing a container with well known paths for the devices. On bare-metal systems this would usually be achieved with a SYMLINK directive in a udev rule. This program tries to provide a similar functionality for containers, allowing you to specify symlinks for certain devices.

This tool wraps runc with the additional hotplug feature, therefore it can be used as a drop in replace for many container managers/orchestrators such as Docker, Podman, and Kubernetes. You need to ensure runc is available in your PATH so container-hotplug can find it.

It supports three annotations, org.lowrisc.hotplug.devices, org.lowrisc.hotplug.symlinks and org.lowrisc.hotplug.sysfs.

For Docker, you can specify an alternative runtime by changing /etc/docker/daemon.json:

{
  "runtimes": {
    "hotplug": {
      "path": "/path/to/container-hotplug/binary"
    }
  }
}

and use it with the --runtime hotplug flag and appropriate annotation, e.g.

sudo docker run --runtime hotplug -it --annotation org.lowrisc.hotplug.devices=parent-of:usb:2b2e:c310 ubuntu:latest

For podman, you can specify the path directly, by:

sudo podman run --runtime /path/to/container-hotplug/binary -it --annotation org.lowrisc.hotplug.devices=parent-of:usb:2b2e:c310 ubuntu:latest

For containerd (e.g. when using kubernetes), you can edit /etc/containerd/config.toml to add:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.hotplug]
  runtime_type = "io.containerd.runc.v2"
  pod_annotations = ["org.lowrisc.hotplug.*"]

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.hotplug.options]
  SystemdCgroup = true
  BinaryName = "/path/to/container-hotplug/binary"

this would allow you to use hotplug as handler in k8s, e.g. add a runtime class with

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: hotplug
handler: hotplug

and use it in a pod with

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu
  annotations:
    org.lowrisc.hotplug.devices: parent-of:usb:0bda:5634
spec:
  runtimeClassName: hotplug
  containers:
  - name: ubuntu
    image: ubuntu:latest
    stdin: true
    tty: true

If you want symlinks to the tty devices created by interfaces 1 and 3 of the CW310, add

--annotation org.lowrisc.hotplug.symlinks=usb:2b3e:c310:1=/dev/ttyACM_CW310_0,usb:2b3e:c310:3=/dev/ttyACM_CW310_1

to docker/podman command line or

org.lowrisc.hotplug.symlinks: usb:2b3e:c310:1=/dev/ttyACM_CW310_0,usb:2b3e:c310:3=/dev/ttyACM_CW310_1

to k8s config.

The device nodes are enough for libusb and for the tty interfaces, but some operations are performed by writing to sysfs attributes instead, e.g. authorized, bConfigurationValue or power/control. sysfs is not namespaced, so the device directories are already visible inside the container, but /sys is mounted read-only, so such writes are rejected.

Adding --annotation org.lowrisc.hotplug.sysfs=rw (or org.lowrisc.hotplug.sysfs: rw for k8s) bind-mounts the sysfs directory of each attached device read-write inside the container, so /sys/bus/usb/devices/<bus>-<ports> and its <bus>-<ports>:<config>.<interface> subdirectories become writable. Only the devices under the specified root devices are affected; the rest of /sys stays read-only. The mounts are removed as devices are unplugged.

This is opt-in because these attributes are host state, not a per-container view of it: a container writing to them affects the device as seen by the host and by every other container using it.

Two limitations to be aware of:

• Driver binding via /sys/bus/usb/drivers/<driver>/{bind,unbind} is not covered, since that is not part of the device's own directory.
• sysfs attributes are owned by the host's root user, so DAC still applies. This is fine for a container whose user maps to the host's root user, which is the usual case. For a container using a user namespace we try to idmap the mount so that the container's root user owns the attributes, but sysfs does not currently support idmapped mounts, so this fails with EINVAL and a warning is logged. Until sysfs gains support, writable sysfs in a user namespaced container needs the attributes to be chowned on the host, e.g. from a udev rule.