Skip to main content

Volume storage configuration

This page gives examples on how to set up a site to allow for different storage options. To see examples of configuring storage for an application see Applications.

For an overview on how host storage relates to application storage, see this fundamentals chapter.

Filesystems without quota support​

It is perfectly fine to use filesystems without quota support for storing volumes. There are two drawbacks though:

  1. The Edge Enforcer cannot enforce maximum volume sizes
  2. Disk metrics will not be collected for volumes

Support container writable layer size limit​

Limiting container layer size is implemented using filesystem project quota feature. Container layers are stored by the docker daemon, typically under /var/lib/docker/overlay2. Hence, in order for the size limit to be enforced, this path must be located on a supported filesystem, such as:

  • ext4 filesystem with prjquota flag enabled.
  • XFS filesystem with d_type support and prjquota flag enabled.
findmnt -T /var/lib/docker/overlay2
SOURCE           TARGET     FSTYPE  OPTIONS
/var/lib/docker  /dev/sda1  ext4    rw,relatime,prjquota
findmnt -T /var/lib/docker/overlay2
SOURCE           TARGET     FSTYPE  OPTIONS
/var/lib/docker  /dev/sda1  xfs     rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,prjquota

Allocating storage for application ephemeral and persistent volumes​

Ephemeral and persistent application volumes are placed on host volumes called local-volumes in the site configuration. If there are no host volumes configured, then an application requesting an ephemeral or persistent volume will fail to be scheduled.

- name: stockholm-sergel
  type: edge
  hosts:
  - host-id: 94d4f410-1ef7-4f01-a296-c4ce4f23bf0e
    local-volumes:
      - name: local-disk-volume-storage
        path: /local-volumes-storage
        size: 1TB
        labels:
          type: ssd
  - host-id: af909c45-47ce-4198-a38e-6ced4ec121f6
  - host-id: 3cac9e07-8560-48ad-876f-db62d2258d6c
    local-volumes:
      - name: slow-storage
        path: /slow
        size: 100TB
        labels:
          type: hdd

All host volumes are located under a single path indicated by $HOST_VOLUMES_DIR environment variable at system start time, the default location is /var/lib/supd/volumes. Any filesystems already mounted outside this location should be bind-mounted to a path under $HOST_VOLUMES_DIR. The path indicated in line 7 and line 15 in the example above is relative to $HOST_VOLUMES_DIR regardless of the leading path separator.

Different hosts in the site may provide different storage options. Applications may explicitly request certain storage or class of storage using a label expression with the labels configured by the site provider and communicated to the application creator. This is considered in scheduling decisions.

Development option: system-defined host volume​

Allocating storage for application volumes requires coordination between someone with administrative privileges in Avassa platform to create the local-volumes configuration and someone who has privileges to access the filesystem and make sure that the paths in the local-volumes configuration point to the desired storage. In some cases, especially during the development phase, it may be practical to provide the applications with some kind of storage without the hassle of the coordination between different parties. In this case it is motivated to use the system-defined auto volume.

note

In production environment it is recommended to keep the local volumes on a separate partition to prevent the applications managed by the Avassa solution from competing for disk storage with the OS or the Avassa platform. Therefore, the system-defined host volume is only suitable for development environments.

An auto volume is an implicit local-volumes definition which is inactive by default. In order for it to be activated a filesystem path corresponding to this definition must be created.

The Avassa platform defines the following action to create the filesystem path and activate the auto volume:

supctl do -s stockholm-sergel system cluster hosts stockholm-sergel-001 create-auto-volume

Or in the UI:

create-auto-vol

This action creates a directory _auto under /var/lib/supd/volumes and activates the auto volume. An alternative to running the action is to create a directory or a mountpoint _auto under /var/lib/supd/volumes on the host manually before starting supd, which is another way to activate the auto volume definition.

Verify that the auto volume is active on the host:

supctl show -s stockholm-sergel system cluster hosts stockholm-sergel-001 --fields local-volumes
local-volumes:
  - name: auto
    status: up
    filesystem:
      type: ext4
      size-limit-support: false
    is-mountpoint: false
    space:
      total: 232.83 GiB
      allocated: 0 B
      free: 232.83 GiB

Support for ephemeral and persistent volume size limit​

Similarly to the container writable layer, the ephemeral or persistent volume size limit is implemented using filesystem quota. Hence the host volumes that should support enforceable application ephemeral or persistent volume size limit must be located on a supported filesystem with project quota flag enabled:

  • ext4 filesystem with prjquota flag enabled.
  • XFS filesystem with d_type support and prjquota flag enabled.
findmnt -T /var/lib/supd/volumes/local-volumes-storage
SOURCE                                       TARGET     FSTYPE  OPTIONS
/var/lib/supd/volumes/local-volumes-storage  /dev/sda2  ext4    rw,relatime,prjquota
findmnt -T /var/lib/supd/volumes/slow-storage
SOURCE                              TARGET     FSTYPE  OPTIONS
/var/lib/supd/volumes/slow-storage  /dev/sda2  xfs     rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,prjquota

The support for volume size limit can be confirmed by looking at the show system cluster hosts output:

supctl show --site stockholm-sergel system cluster hosts
- cluster-hostname: stockholm-sergel-001
  oper-status: up
  host-id: 94d4f410-1ef7-4f01-a296-c4ce4f23bf0e
  local-volumes:
    - name: local-disk-volume-storage
      status: up
      filesystem:
        type: ext4
        size-limit-support: true
      is-mountpoint: true
      space:
        total: 1.92 TiB
        allocated: 952 GiB
        free: 1.01 TiB

Add image with quota support​

In case your filesystem doesn't have quota support, you can create an image file and mount that file with quota support.

This example will create a 20MB image file and mount that as a volume called xfs.

Create and format the /xfs.img image with xfs.

fallocate -l 20M /xfs.img
xfsdev=$(losetup -f --show /xfs.img)
mkfs.xfs $xfsdev
losetup -d $xfsdev

Add it to /etc/fstab to be mounted after host reboot

echo '/xfs.img /var/lib/supd/volumes/xfs xfs loop,prjquota 0 0' >> /etc/fstab

Mount it

mkdir -p /var/lib/supd/volumes/xfs
mount /var/lib/supd/volumes/xfs

This volume can now be configured as

- name: stockholm-sergel
  type: edge
  hosts:
  - host-id: 94d4f410-1ef7-4f01-a296-c4ce4f23bf0e
    local-volumes:
      - name: xfs-with-quota
        path: /xfs
        size: 20MB