Dynamic Devices
When an application needs to access a device, in many cases the device is statically attached to the machine the application is running on for the whole lifetime of the application. In such case follow the Device Discovery tutorial to learn how to select an existing device and run an application with the device mounted into the container by the container runtime.
However, in some cases the devices may be hot-plugged to the machine and the running application needs to get access and start communicating with such device without interruption. This is especially true for USB devices. This tutorial explains which permissions does an application need to access dynamic devices and gives an example of such application.
Dynamic device rules
By default a container is not allowed to access any device files. An explicit permission must be granted for container to read or write to the device file, based on the device type (character or block device), major and minor number of the device.
By default in some container runtimes a container can create device files with
mknod system call. Still, the container does not have permissions to read or
write to such device by default.
When a device is statically mounted into the container by the container runtime, the permission is automatically granted to access such device. However, when the device is not available at the time of application startup, the rules to access the required devices must be declared explicitly, for example as follows:
services:
- name: ...
containers:
- name: ...
devices:
dynamic:
rules:
- type: character
major: 250
minor: any
permissions: read,write,mknod
This rule grants the container permissions to create a character device with major number 250
and any minor number, as well as read from and write to such device.
The type of the device and its major number can be determined by inspecting an existing device:
$ ls -l /dev/rtc0
crw-rw---- 1 root root 250, 0 Sep 12 13:51 /dev/rtc0
Here the initial c symbol indicates that the file is a character special
file, and 250, 0 in the fifth and sixth column indicate the major, minor
numbers of the device.
The device type and its major and minor numbers are also possible to determine from
the output of stat command, for example stat /dev/rtc0. However, different
versions of stat command can print major and minor numbers either in decimal
or in hex format, so in order to avoid confusion the ls -l command is recommended.
The dynamic device rule in application specification expects the device numbers to be specified in decimal format.
The rule of thumb is that storage devices are often block devices, while other devices
are character devices. The major number is determined by the device driver, so it remains
the same even if the device is disconnected and reconnected again. The minor number is however
dependent on the driver behaviour, and it may change if the same device is disconnected and
reconnected again.
While it is possible to grant an application access to all devices (with any
type, any major number and any minor number), declaring such wide
permissions is not recommended. This allows the application to access such
sensitive devices as system memory device (allowing it to read or modify the
contents of the machine's RAM) or system disk (allowing it to read or modify
files on any of the machine's filesystems).
While it is often not possible to determine the device's minor number in advance, it is a good security practice to only allow an application access to a certain device type and a certain device major number.
The permissions that can be declared in the application specification can be limited by a parent tenant using resource profiles.
Creating and accessing the device node
The dynamic device rule created in the Dynamic device
rules section grants the container permission to access
the real-time clock device (/dev/rtc0 in the host OS). We have determined by
inspecting the real-time clock device in the host OS that it is a character
device with the major number 250, which we used to create the rule. However,
the rule itself does not create any device node inside the container.
Let us examplify creating and accessing a device node dynamically with a trivial busybox shell session, using the following application specification:
name: busybox
services:
- name: busybox
mode: replicated
replicas: 1
containers:
- name: busybox
image: registry-1.docker.io/library/busybox
cmd:
- sleep
- infinity
additional-capabilities:
- mknod
user-namespace:
host: true
devices:
dynamic:
rules:
- type: character
major: 250
minor: any
permissions: read,mknod
Because mknod operation is prohibited in non-initial user namespaces, in order to be
able to create device nodes the container needs to be run in host default user namespace,
as declared using the following configuration parameter.
user-namespace:
host: true
Also, some container runtimes (notably, Podman) do not grant a container MKNOD capability
by default, so it must be requested separately as follows:
additional-capabilities:
- mknod
Follow the Deploy your first application tutorial for instructions on how to deploy an application.
Open an interactive shell session as described in the Execute Commands and
Access Shell in
Containers document using
either the supctl command line tool or the WebUI.
Inside the container, we can confirm that no real-time clock device file exists.
# / ls /dev/rtc*
ls: /dev/rtc*: No such file or directory
If we try to read the real-time clock inside the container we get an error:
# / hwclock -r
hwclock: can't open '/dev/misc/rtc': No such file or directory
Note that the hwclock utility tries several well-known paths for the
real-time clock device and only reports the last attempt as unsuccessful, so it
would suffice to have a /dev/rtc0 device inside the container.
Using the parameters we noted by inspecting the /dev/rtc0 device on the host
we can create a corresponding device node inside the container. Another way to
determine the major and minor number for a device is by reading the dev file
for the corresponding device entry in the /sys filesystem inside the
container:
# / cat /sys/class/rtc/rtc0/dev
250:0
Because the minor number is often not known until the device is connected to
the system, reading the device major and minor numbers from the dev file for
the device entry in the /sys filesystem may be the preferred method for the
application to determine the required device parameters.
Let us create a device node for a character device with major number 250 and minor number 0. Inside the container we execute:
# / mknod /dev/rtc0 c 250 0
Here /dev/rtc0 is the path where the real-time clock device should be
created, c stands for character device type, 250 is the device's major
number and 0 is the device's minor number.
This time if we try to read the real-time clock inside the container again, the command succeeds.
# / hwclock -r
hwclock -r
Fri Sep 12 15:15:29 2025 0.000000 seconds
In some cases it is feasible for an application to manage the device nodes in this fashion, but in other cases the application expects the device nodes to be created for it. In such case running a device manager inside the container may be motivated.
Device manager
The task of the device manager is to listen to kernel's uevent messages, load
device drivers, create corresponding device nodes in the /dev directory and
notify other applications about the changes.
Loading the device drivers is a task that should not be performed from inside
the container, because the kernel modules must strictly match the kernel version
for which they are compiled. This would require the kernel module files inside the
container to be in sync the kernel version outside of the container, which is
not only difficult to maintain, but also is against the principles of container
applications. Instead, the necessary drivers should be identified beforehand and
pre-loaded into the kernel of the OS running on the host. On many distributions
this can be achieved by putting the relevant module name in a file under
/etc/modules-load.d directory in the host OS filesystem.
The rest of the tasks can be performed by a device manager running inside the
container. As mentioned in the Device discovery tutorial,
the udev is often used as a device manager, in
particular in distributions using systemd as the init system. However it is
often impractical and unnecessary to run the whole systemd inside the
containers, so there are more lightweight alternatives. Here we use mdev
device manager which is a part of busybox software suite as an example.
Populating the /dev tree
A trivial use-case would be to ask mdev to populate the /dev directory
once by reading all the available devices under /sys filesystem and creating
corresponding device nodes. It can be done by running mdev -s command.
Continuing the example from the previous section, in order to demonstrate
mdev first we need to either restart the service or manually remove the
/dev/rtc0 device with rm /dev/rtc0. Make sure that no device is present:
# / ls /dev/rtc*
ls: /dev/rtc*: No such file or directory
# / hwclock -r
hwclock: can't open '/dev/misc/rtc': No such file or directory
The following command populates the /dev with the devices found in /sys:
# / mdev -s
This command created many device nodes, but most of them are inaccessible
because the dynamic device rules specified in the application specification
disallow access to devices other than character device with major number 250.
For example, reading the contents of the system memory is not allowed:
# / cat /dev/mem
cat: can't open '/dev/mem': Operation not permitted
Verify that among others the /dev/rtc0 is created. It is accessible
according to the dynamic device rules, so we can read it with help of
hwclock utility:
# / ls /dev/rtc0
/dev/rtc0
# / hwclock -r
Fri Sep 12 15:35:17 2025 0.000000 seconds
The mdev -s command scans the /sys filesystem once and exits upon
completion. It may be sufficient for some use-cases, but in other cases the
devices are expected to be created immediately after they are plugged in, and
deleted when unplugged. The next section shows how to run mdev as daemon
subscribing to kernel events.
Running mdev as daemon
The real power of the device manager is subscribing to uevents broadcasted
by the Linux kernel on a netlink socket. As we noticed in the previous section,
the device-related operations are only permitted in the context of the host
default user namespace, which is also true for the device-related events.
Because a netlink socket exists in the context of a network namespace, the
user namespace in context of which the network namespace is created must
be the host default user namespace. Because a service may consist of multiple
containers sharing the same network namespace, we need to request the host
default user namespace not only for the container, but also for the service
network itself. We modify the application as follows:
name: busybox
services:
- name: busybox
mode: replicated
replicas: 1
network:
user-namespace:
host: true
containers:
- name: busybox
image: registry-1.docker.io/library/busybox
cmd:
- sleep
- infinity
additional-capabilities:
- mknod
user-namespace:
host: true
devices:
dynamic:
rules:
- type: character
major: 250
minor: any
permissions: read,mknod
Once the application has been upgraded, we can start an interactive shell session again.
We may notice that due to container restart in connection with the
application upgrade, the /dev directory contains only the essential devices
created by the container runtime.
This time we run mdev in daemon mode. In this mode mdev subscribes to
kernel uevents and stays in the background reacting to the events and managing
the devices as they are plugged and unplugged.
# / mdev -d
Note that the command returns, but the mdev process stays alive in the background.
# / ps
...
6 root 0:00 mdev -d
...
The device tree has been populated:
/ # ls /dev/rtc0
/dev/rtc0
/ # hwclock -r
Fri Sep 12 15:48:42 2025 0.000000 seconds
In order to illustrate the dynamic device creation we may create a new loop device outside
of the container.
In the host OS shell, first verify that a loop device with arbitrary high number does not exist:
$ ls /dev/loop99
ls: cannot access '/loop99': No such file or directory
It does not exist inside the container either:
/ # ls /dev/loop99
ls: cannot access '/loop99': No such file or directory
Now, in the host OS shell, create a dummy file and create a loop device backed by this file:
$ truncate -s 1024 /tmp/dummy
$ losetup /dev/loop99 /tmp/dummy
Note that the loop device has been created in the host OS:
$ ls /dev/loop99
/dev/loop99
This device has been created inside the container too. It has been created by the
device manager that reacted to the kernel's uevent. Verify by running the
following command inside the container:
/ # ls /dev/loop99
/dev/loop99
We do not however have access to the device inside the container, because the dynamic device rules do not allow reading or writing to it.
Example: hot-pluggable USB device
In the following example we are going to demonstrate the dynamic device functionality by starting an application that uses a USB smart card reader device. The application is started when the device is unplugged, and as the device is plugged it is immediately detected by the application by means of a notification from the device manager.
For the purpose of this example we are using a custom image containing the
pcsc-lite software suite for smart card access.
The image is built using the Dockerfile available in Avassa's
application-examples repository
:
FROM alpine:3.22
RUN apk add --no-cache pcsc-lite pcsc-tools ccid libudev-zero libudev-zero-helper && \
echo '-.* root:root 660 */usr/libexec/libudev-zero-helper' > /etc/mdev.conf
ENTRYPOINT ["/bin/sh", "-c", "mdev -d && pcscd -if"]
Create an application using the following specification:
name: smart-card-reader
services:
- name: scr
mode: replicated
replicas: 1
containers:
- name: reader
image: registry.gitlab.com/avassa-public/application-examples/smart-card-reader:1.0
additional-capabilities:
- mknod
- net-admin
user-namespace:
host: true
devices:
dynamic:
rules:
- type: character
major: 189
minor: any
permissions: read,write,mknod
network:
user-namespace:
host: true
There is an extra net-admin capability added to the list of
additional-capabilities. This capability is required for the device manager
(mdev) to publish notifications about the device changes for other
applications on a netlink socket. The net-admin capability is therefore not
required for mdev to receive kernel notifications, but it is required for it
to publish its own notifications for other applications.
The mknod capability, host user-namespace requirements and dynamic device
rules have been explained earlier in this tutorial.
We start by deploying this application while the smart card reader device is unplugged from the USB port.
By looking at the ENTRYPOINT instruction in the Dockerfile we can see that
the mdev device manager and the pcscd application managing smart card
readers are started at application startup.
Let us start an interactive shell
session and run a
client program that connects to the pcscd server and awaits a device to be
connected.
/ # pcsc_scan
PC/SC device scanner
V 1.7.3 (c) 2001-2024, Ludovic Rousseau <ludovic.rousseau@free.fr>
Using reader plug'n play mechanism
Scanning present readers...
Waiting for the first reader... /
As we plug in the smart card reader into the USB port the application stops spinning and immediately identifies the device.
Waiting for the first reader... found one
Scanning present readers...
0: VASCO DIGIPASS 920 [CCID] 00 00
Fri Sep 12 16:07:04 2025
Reader 0: VASCO DIGIPASS 920 [CCID] 00 00
Event number: 0
Card state: Card removed,
What happened behind the scenes is:
- upon the USB device connection the kernel sent an
ueventnotification - the device manager
mdevreceived the notification mdevdevice manager created a device file under/devmdevdevice manager sent a notification about a new device for applications according to the rule in themdev.conffile (see Dockerfile) using thelibudev-zero-helper- the
libudev-compatible applicationpcscduses a drop-in replacement librarylibudev-zeroto listen to notifications frommdev - upon receiving the notification
pcscdidentified the device that the client application displayed
In order to confirm that the device is operational inside the container we can insert a smart card into the reader:
Fri Sep 12 16:08:52 2025
Reader 0: VASCO DIGIPASS 920 [CCID] 00 00
Event number: 1
Card state: Card inserted,
...
Let us stop the client application and take a look at what is happening from
the OS perspective. Run the lsusb utility which lists the USB devices found
in the /sys filesystem along with some information:
/ # lsusb | grep -i digipass
Bus 003 Device 043: ID 1a44:0920 Vasco DIGIPASS 920
From this output we know that this device is the device number 043 connected to USB bus 003, so we can inspect a corresponding device special file for it:
/ # ls -l /dev/bus/usb/003/043
crw-rw---- 1 root root 189, 298 Sep 12 14:12 /dev/bus/usb/003/043
Now, if we unplug the device, it both disappears from lsusb output and the
corresponding device node is removed by the device manager.
/ # lsusb | grep -i digipass
/ # ls -l /dev/bus/usb/003/043
ls: /dev/bus/usb/003/043: No such file or directory
If we reconnect the same device again, we can see that it has been assigned a different number on the USB bus and a different minor number.
/ # lsusb | grep -i digipass
Bus 003 Device 044: ID 1a44:0920 Vasco DIGIPASS 920
/ # ls -l /dev/bus/usb/003/044
crw-rw---- 1 root root 189, 299 Sep 12 14:14 /dev/bus/usb/003/044