Device Discovery

In many cases an application needs access to (hardware) devices connected to the hosts, e.g. cameras, sensors, etc.

The Edge Enforcer can use udev to discover what devices are connected and allow applications access to devices.

udev

udev is a device manager for the Linux kernel. udev comes with a collection of rules that describes to the device manager how devices are mapped and named. For more information on udev, please see e.g. Wikipedia.

To list devices on a Linux host, you can use udevadm, e.g.

udevadm info -e

...

P: /devices/platform/serial8250
E: DEVPATH=/devices/platform/serial8250
E: DRIVER=serial8250
E: MODALIAS=platform:serial8250
E: SUBSYSTEM=platform

P: /devices/platform/serial8250/tty/ttyS1
N: ttyS1
E: DEVNAME=/dev/ttyS1
E: DEVPATH=/devices/platform/serial8250/tty/ttyS1
E: MAJOR=4
E: MINOR=65
E: SUBSYSTEM=tty

P: /devices/platform/serial8250/tty/ttyS2
N: ttyS2
E: DEVNAME=/dev/ttyS2
E: DEVPATH=/devices/platform/serial8250/tty/ttyS2
E: MAJOR=4
E: MINOR=66
E: SUBSYSTEM=tty

P: /devices/platform/serial8250/tty/ttyS3
N: ttyS3
E: DEVNAME=/dev/ttyS3
E: DEVPATH=/devices/platform/serial8250/tty/ttyS3
E: MAJOR=4
E: MINOR=67
E: SUBSYSTEM=tty

...

For the rest of this documentation, we will use these serial tty devices as example.

Declare what devices to search for

A system usually have a lot of devices in the device tree, most of those devices are probably not of interest to applications. The Edge Enforcer is configured by the site provider to scan for certain devices by using udev rules.

Taking a closer look at ttyS1:

udevadm info -ap /devices/platform/serial8250/tty/ttyS1

Udevadm info starts with the device specified by the devpath and then
walks up the chain of parent devices. It prints for every device
found, all possible attributes in the udev rules key format.
A rule to match, can be composed by the attributes of the device
and the attributes from one single parent device.

  looking at device '/devices/platform/serial8250/tty/ttyS1':
    KERNEL=="ttyS1"
    SUBSYSTEM=="tty"
    DRIVER==""
    ATTR{iomem_reg_shift}=="0"
    ATTR{console}=="N"
    ATTR{line}=="1"
    ATTR{xmit_fifo_size}=="0"
    ATTR{port}=="0x2F8"
    ATTR{uartclk}=="1843200"
    ATTR{type}=="0"
    ATTR{iomem_base}=="0x0"
    ATTR{custom_divisor}=="0"
    ATTR{io_type}=="0"
    ATTR{irq}=="3"
    ATTR{close_delay}=="50"
    ATTR{closing_wait}=="3000"
    ATTR{flags}=="0x10000040"

  looking at parent device '/devices/platform/serial8250':
    KERNELS=="serial8250"
    SUBSYSTEMS=="platform"
    DRIVERS=="serial8250"
    ATTRS{driver_override}=="(null)"

  looking at parent device '/devices/platform':
    KERNELS=="platform"
    SUBSYSTEMS==""
    DRIVERS==""

Above we can see what information udev has on the the ttyS1 device and its parent device (serial8250).

These key/values are used when writing udev rules, i.e. rules that describes to the Edge Enforcer what devices should be discovered.

In the Avassa system, devices are mapped to device labels by pairing a device label name with udev rules that will match devices of interest.

This command shows a label rule, configured by the site provider, that will create a device label called tty.

supctl show -c system sites udc1

...
device-labels:
  - label: tty
    udev-patterns:
      - SUBSYSTEM=="tty", DRIVERS=="serial8250", ATTR{console}=="N"
...

The rule tells the system to create a label called tty, all devices matching the rule will be added to the tty device label.

The rules will match individual devices or parent devices:

• The DRIVERS clause will match the serial8250 which is the parent of ttyS1.

• SUBSYSTEM=="tty" and ATTR{console}=="N" will match the ttyS1 device itself.

I.e. there will be a match for rules that match a device or one of its parents.

As a site provider, when looking at the udc1 site, we can see what devices were actually discovered and at what hosts.

note

The discovered devices are only available at the site.

supctl show --site udc1 system cluster hosts

- cluster-hostname: udc1-003
  version: dev-001d7b93
  oper-status: up
  hostname: h05
  host-id: 459c6eef-090d-4f54-bc9e-0b44bcd85688
  controller: true
  labels:
    system/controller: "true"
  device-labels:
    tty:
      - /dev/ttyS1
      - /dev/ttyS2
      - /dev/ttyS3
    rtc: /dev/rtc0
  devices:
    - name: /dev/ttyS3
      labels:
        tty: /dev/ttyS3
    - name: /dev/ttyS2
      labels:
        tty: /dev/ttyS2
    - name: /dev/ttyS1
      labels:
        tty: /dev/ttyS1
    - name: /dev/rtc0
      labels:
        rtc: /dev/rtc0
  local-volumes: []
  host-network:
    ip-addresses:
      - 192.168.100.105
      - fe80::5054:ff:fe56:fe3
  site-overlay-network:
    status: up
    ip-addresses:
      - 172.27.0.3
  network-interfaces:
    - name: default
      os-interface-name: eth0

There you see that the system discovered the following devices for the tty label.

• /dev/ttyS1
• /dev/ttyS2
• /dev/ttyS3

Controlling access to to devices

note

The site provider's applications have access to all discovered devices.

With device discovery in place, we now need to control what applications can access these devices, this is done by defining resource profiles.

note

We use a application owner tenant, acme, as example here.

supctl show resource-profiles t-acme-udc1-medium

- name: t-acme-udc1-medium
  device-labels:
    - name: rtc
    - name: tty

t-acme-udc1-medium will allow access to device-labels rtc and tty.

supctl show resource-profiles t-acme-udc2-medium

- name: t-acme-udc2-medium
  device-labels: []

Since device-labels is an empty list, t-acme-udc2-medium, when attached to a site, will block all devices labels from being available.

These resource profiles are attached to sites as follows:

supctl show tenants acme assigned-sites

- name: udc1
  type: edge
  labels:
    system/type: edge
    system/name: udc1
  host-labels: {}
  volume-labels: {}
  network-interface-labels: {}
  resource-profile: t-acme-udc1-medium
- name: udc2
  type: edge
  labels:
    system/type: edge
    system/name: udc2
  host-labels: {}
  volume-labels: {}
  network-interface-labels: {}
  resource-profile: t-acme-udc2-medium

If we, as acme (i.e. as an application owner) look at what device labels are available in each site. Please note that the system does not report on what hosts the devices were discovered.

supctl show --site udc1 assigned-sites udc1

name: udc1
type: edge
labels:
  system/type: edge
  system/name: udc1
host-labels: {}
volume-labels: {}
device-labels:
  tty:
    - /dev/ttyS1
    - /dev/ttyS2
    - /dev/ttyS3
  rtc: /dev/rtc0
network-interface-labels: {}
resource-profile: t-acme-udc1-medium

In udc1 we see devices according to the tty and rtc device discovery rules.

supctl show --site udc2 assigned-sites udc2

sites udc2
name: udc2
type: edge
labels:
  system/type: edge
  system/name: udc2
host-labels: {}
volume-labels: {}
device-labels: {}
network-interface-labels: {}
resource-profile: t-acme-udc2-medium

In udc2, acme doesn't have access to any devices.

If we modify t-acme-udc2-medium to allow e.g. rtc devices:

supctl show resource-profiles t-acme-udc2-medium

name: t-acme-udc2-medium
device-labels:
  - name: rtc

Again, look at the site, as acme:

supctl show --site udc2 assigned-sites udc2

name: udc2
type: edge
labels:
  system/type: edge
  system/name: udc2
host-labels: {}
volume-labels: {}
device-labels:
  rtc: /dev/rtc0
network-interface-labels: {}
resource-profile: t-acme-udc2-medium

Using devices in applications

To mount particular devices in a application you declare what device labels are of interest in the application specification.

The site scheduler will try to find hosts with matching devices and start the container there.

A simple example:

name: alpine
version: "1.0"
services:
  - name: my-srv
    containers:
      - name: alpine
        image: alpine
        cmd: ["sleep", "infinity"]
        devices:
          device-labels:
            - tty
    mode: replicated
    replicas: 1

After deploying this to udc1 and udc2

supctl show --site udc1 applications alpine service-instances

- name: my-srv-1
  application-version: "1.0"
  oper-status: running
  ready: true
  host: udc1-002
  application-network:
    ips:
      - 172.26.0.1/16
  gateway-network:
    ips:
      - 172.25.255.2/24
  ingress:
    ips: []
  containers:
    - name: alpine
      id: a4c15b840f4f
      oper-status: running
      ready: true
      start-time: 2022-09-20T08:35:32.626Z
      current-restarts: 0
      total-restarts: 0
      probes:
        startup:
          status: success
        readiness:
          status: success
        liveness:
          status: success
      devices:
        - /dev/ttyS1
        - /dev/ttyS2
        - /dev/ttyS3

note, udc2 does not have access to any tty devices, hence the application can not be started.

supctl show --site udc2 applications alpine service-instances

- name: my-srv-1
  oper-status: not-scheduled
  not-scheduled-reason: no-device-label-match
  error-message: all required devices not present on node