Convert Docker Compose

This how-to will guide your through the process of converting a Docker compose file to the corresponding Avassa Application Specification.

For a reference on Docker/Docker Compose to Avassa, see Docker to Avassa reference.

General steps

Identify container secrets, e.g. credentials and create corresponding Avassa vault and secrets.
Decide if to inline or store application configuration in a vault
Identify Docker Compose services and create an Avassa application specification with a similar service structure.

Docker Compose

We will use a Mosquitto (MQTT server), Telegraf (InfluxDB data ingester), InfluxDB (Time series DB) and Grafana (Visualization) example.

docker-compose.yaml
services:
  mosquitto:
    image: eclipse-mosquitto:latest
    container_name: mosquitto
    restart: always
    ports:
      - "1883:1883"
    networks:
      - iot
    volumes:
      - ./mosquitto.conf:/mosquitto/config/mosquitto.conf
  
  influxdb:
    image: influxdb
    container_name: influxdb
    restart: always
    ports:
      - "8086:8086"
    networks:
      - iot
    volumes:
      - influxdb-data:/var/lib/influxdb2
      - influxdb-config:/etc/influxdb2
    environment:
      - DOCKER_INFLUXDB_INIT_MODE=setup
      - DOCKER_INFLUXDB_INIT_USERNAME=avassa
      - DOCKER_INFLUXDB_INIT_PASSWORD=!Avassa2024!
      - DOCKER_INFLUXDB_INIT_ORG=some_org
      - DOCKER_INFLUXDB_INIT_BUCKET=some_data
      - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=cud0jxv_wkz7WNP1hqf
  
  telegraf:
    image: telegraf
    container_name: telegraf
    restart: always
    volumes:
      - ./telegraf.conf:/etc/telegraf/telegraf.conf:ro
    depends_on:
      - mosquitto
      - influxdb
    networks:
      - iot
  
  grafana:
    image: grafana/grafana
    container_name: grafana
    restart: always
    ports:
      - "3000:3000"
    networks:
      - iot
    volumes:
      - ./grafana-provisioning:/etc/grafana/provisioning
      - grafana-data:/var/lib/grafana
    depends_on:
      - influxdb

networks:
  iot:

volumes:
  grafana-data:
  influxdb-data:
  influxdb-config:

Docker Services

We note that this compose contains four services:

mosquitto
influxdb
telegraf
grafana

Note the depends_on between services, this is not supported in the Avassa platform. The services are required to be able to gracefully wait for other services to come up or be restarted. A restart of a single service will happen during upgrades.

Docker Networks

In compose you can, optionally, specify networks (iot). In Avassa these types of networks are automatically created.

Note well that in the Avassa system, volumes must be prepared on the site.

Identifying and Storing Secrets

First thing we will do is scan the docker compose for values that should be considered secrets. In compose, these are usually stored inline, in Avassa we strongly recommend you store these in the secure vault.

DOCKER_INFLUXDB_INIT_ADMIN_TOKEN
DOCKER_INFLUXDB_INIT_USERNAME
DOCKER_INFLUXDB_INIT_PASSWORD

Create a vault named tig.

Create vault
supctl create strongbox vaults <<EOF
name: tig
# Distribute together with the tig deployment
distribute:
  deployments:
    - tig
EOF

Next create a secret called credentials

Create credentials secret
supctl create strongbox vaults secrets <<EOF
name: credentials
data:
  ADMIN_TOKEN: cud0jxv_wkz7WNP1hqf
  INIT_USERNAME: avassa
  INIT_PASSWORD: "!Avassa2024!"
allow-image-access:
  - "*"
EOF

Application Configuration

For configuration we have two options:

Declare configuration inline in the application
Store configuration in a vault

For Telegraf, since it's fairly large, we will store the configuration in a vault.

For Mosquitto and Grafana we will store the configuration inline in the application specification.

The easiest way is to base64 encode the configuration, e.g. base64 -w0 telegraf.conf. In this way you don't have to worry about formatting.

Store mosquitto config in a secret
supctl create strongbox vaults tig secrets config <<EOF
name: config
allow-image-access: ["*"]
base64-data:
  telegraf.conf: $(base64 -w0 telegraf.conf)
EOF

Create Application Specification

Next create the Avassa application specification. See comments in the application specification below.

The general outline will be one Avassa service for each Docker service, i.e.:

services:
  - name: mosquitto
    ...
  - name: influxdb
    ...
  - name: telegraf
    ...
  - name: grafana
    ...

Application Spec
name: tig
# When deploying to production, always set the version
# version: "1.0"
services:
  # Each Docker service becomes an Avassa service
  - name: mosquitto
    # One instance per site
    mode: replicated
    replicas: 1

    # Declare data and configuration volumes
    volumes:
      - name: data
        ephemeral-volume:
          # Arbitrary size, make sure you size this for your data
          size: 100MB
      - name: log
        ephemeral-volume:
          # Arbitrary size, make sure you size this for your data
          size: 10MB
      - name: config
        config-map:
          items:
            # Inline the mosquitto configuration
            - name: mosquitto.conf
              data: |
                # This is a Mosquitto configuration file that creates a listener on port 1883
                # that allows unauthenticated access.

                listener 1883
                allow_anonymous true
    network:
      ingress-ip-per-instance:
        # Allow access (from the outside) to TCP port 1883
        protocols:
          - name: tcp
            port-ranges: "1883"
      outbound-access:
        # Allow the container to connect to the outside
        allow-all: true
    containers:
      - name: mosquitto
        image: registry-1.docker.io/eclipse-mosquitto:latest

        # Mount the data and configuration volumes into the container
        mounts:
          - volume-name: data
            mount-path: /mosquitto/data
          - volume-name: log
            mount-path: /mosquitto/log
          - volume-name: config
            files:
              - name: mosquitto.conf
                mount-path: /mosquitto/config/mosquitto.conf

  - name: influxdb
    mode: replicated
    replicas: 1
    volumes:
      - name: data
        ephemeral-volume:
          # Arbitrary size, make sure you size this for your data
          size: 100MB
      - name: etc
        ephemeral-volume:
          # Arbitrary size, make sure you size this for your data
          size: 10MB
    network:
      ingress-ip-per-instance:
        protocols:
          - name: tcp
            port-ranges: "8086"
      outbound-access:
        allow-all: true
    # Declare variables from the vault/secret values
    variables:
      - name: INIT_USER
        value-from-vault-secret:
          vault: tig
          secret: credentials
          key: INIT_USERNAME
      - name: INIT_PASS
        value-from-vault-secret:
          vault: tig
          secret: credentials
          key: INIT_PASSWORD
      - name: ADMIN_TOKEN
        value-from-vault-secret:
          vault: tig
          secret: credentials
          key: ADMIN_TOKEN
    containers:
      - name: influxdb
        image: registry-1.docker.io/influxdb:alpine
        # Variabes can be substituted into the the environment
        env:
          DOCKER_INFLUXDB_INIT_MODE: setup
          DOCKER_INFLUXDB_INIT_USERNAME: ${INIT_USER}
          DOCKER_INFLUXDB_INIT_PASSWORD: ${INIT_PASS}
          DOCKER_INFLUXDB_INIT_ORG: some_org
          DOCKER_INFLUXDB_INIT_BUCKET: some_data
          DOCKER_INFLUXDB_INIT_ADMIN_TOKEN: ${ADMIN_TOKEN}
        mounts:
          - volume-name: data
            mount-path: /var/lib/influxdb2
          - volume-name: etc
            mount-path: /etc/influxdb2

  - name: telegraf
    mode: replicated
    replicas: 1
    volumes:
      - name: config
        config-map:
          items:
            - name: telegraf.conf
              # Telegraf runs as user:gid 100:101
              file-ownership: 100:101
              # Note that the INFLUX_TOKEN variable is substituted into the config below
              data: |
                [[outputs.influxdb_v2]]
                  # Since the service is named influxdb, the Avassa DNS will resolve that name
                  urls = ["http://influxdb:8086"]
                  token = "${INFLUX_TOKEN}"
                  organization = "some_org"
                  bucket = "some_data"

                [[outputs.file]]
                    files = ["stdout"]

                [[inputs.mqtt_consumer]]
                  servers = ["tcp://mosquitto:1883"]
                  topics = [
                    "sensors/#"
                  ]
                  data_format = "json"
    variables:
      - name: INFLUX_TOKEN
        value-from-vault-secret:
          vault: tig
          secret: credentials
          key: ADMIN_TOKEN
    containers:
      - name: telegraf
        image: registry-1.docker.io/telegraf
        # Telegraf needs capa NET_RAW
        additional-capabilities:
          - net-raw
        # Run as user:gid 100:101
        user: 100:101
        mounts:
          - volume-name: config
            files:
              - name: telegraf.conf
                mount-path: /etc/telegraf/telegraf.conf

  - name: grafana
    mode: replicated
    replicas: 1
    volumes:
      - name: data
        ephemeral-volume:
          # Arbitrary size, make sure you size this for your data
          size: 100MB
          # Grafana is running as user 472
          file-ownership: 472:0
      - name: provisioning
        config-map:
          items:
            - name: datasource.yml
              # Note that the INFLUX_TOKEN variable is substituted into the config below
              data: |
                apiVersion: 1

                datasources:
                  - name: InfluxDB_v2_Flux
                    type: influxdb
                    access: proxy
                    # Since the service is named influxdb, the Avassa DNS will resolve that name
                    url: http://influxdb:8086
                    jsonData:
                      version: Flux
                      organization: some_org
                      defaultBucket: some_data
                      tlsSkipVerify: true
                    secureJsonData:
                      token: ${INFLUX_TOKEN}

    network:
      ingress-ip-per-instance:
        protocols:
          - name: tcp
            port-ranges: "3000"
      outbound-access:
        allow-all: true
    variables:
      - name: INFLUX_TOKEN
        value-from-vault-secret:
          vault: tig
          secret: credentials
          key: ADMIN_TOKEN
    containers:
      - name: grafana
        image: registry-1.docker.io/grafana/grafana
        mounts:
          - volume-name: data
            mount-path: /var/lib/grafana
            mode: read-write
          - volume-name: provisioning
            files:
              - name: datasource.yml
                mount-path: /etc/grafana/provisioning/datasources/datasource.yml
# Declare a network called iot
# NOTE, this is generally not necessary and only needed if you have
# other applications you want to share this application network
# network:
#   shared-application-network: iot

Original Application Configurations

Mosquitto

mosquitto.conf
# This is a Mosquitto configuration file that creates a listener on port 1883
# that allows unauthenticated access.

listener 1883
allow_anonymous true

Telegraf

telegraf.conf
[[outputs.influxdb_v2]]
  urls = ["http://influxdb:8086"]
  token = cud0jxv_wkz7WNP1hqf
  organization = "some_org"
  bucket = "some_data"

[[outputs.file]]
    files = ["stdout"]

[[inputs.mqtt_consumer]]
  servers = ["tcp://mosquitto:1883"]
  topics = [
    "paper_wifi/test/#"
  ]
  data_format = "json"

Grafana Datasource

Grafana datasource
apiVersion: 1

datasources:
  - name: InfluxDB_v2_Flux
    type: influxdb
    access: proxy
    url: http://influxdb:8086
    jsonData:
      version: Flux
      organization: some_org
      defaultBucket: some_data
      tlsSkipVerify: true
    secureJsonData:
      token: cud0jxv_wkz7WNP1hqf

General steps​

Docker Compose​

Docker Services​

Docker Networks​

Identifying and Storing Secrets​

Application Configuration​

Create Application Specification​

Original Application Configurations​

Mosquitto​

Telegraf​

Grafana Datasource​