Skip to main content

Host requirements and sizing

Requirements​

This section explains the requirements to install and run the Edge Enforcer.

Host requirements​

Make sure the following requirements are fulfilled before proceeding with the installation preparations:

  • A host running a Linux distribution with the x86_64 kernel version 5.4 or later.
  • WireGuard is included in Linux kernel versions 5.6 and later, please see here for how to set it up on older kernels.
  • Systemd running on the host.
  • Minimum 2GB RAM: the edge enforcer consumes between 100 and 500Mb, that leaves 1.5Gb for operating system and applications
  • Minimum 1 vCPU not dedicated
  • Minimum 2 GB disk space

The installation script supports the following distributions

  • Alpine
  • CentOS 9
  • Debian 11
  • Debian 12
  • RHEL 9
  • Red Hat Device Edge 9
  • Ubuntu 20.04
  • Ubuntu 22.04

If you have a custom kernel, please see Kernel Configuration Options.

Network requirements​

Host/site to Control Tower​

All hosts on a site must be able to connect to the Control Tower on the following ports.

  • TCP 443 (mandatory)
  • UDP 443 (recommended, specifically allow the QUIC protocol in the firewall)

Between hosts in a site​

If you have a firewall between hosts in the same site, make sure the following ports are open:

  • UDP/TCP 53 (optional, if you are using the Edge Enforcer DNS Server)
  • TCP 4668 (host detection and cluster communication)
  • UDP 51820 (WireGuard, application networking)

Host ports​

On the host, the Edge Enforcer will bind to the following ports. This is just informational to avoid port conflicts with other services.

PortListen AddressDescription
UDP/TCP 53localhostDNS Server (forwarded to port 4653)
UDP/TCP 46530.0.0.0DNS Server
TCP 46460.0.0.0HTTPS REST API
TCP 46680.0.0.0Host detection and cluster communication within a site, TLS encrypted
TCP 48480.0.0.0HTTPS Container registry
TCP 4849localhostHTTP Container registry
TCP 4949localhostDocker plugin, only on Docker
TCP 4950localhostDocker plugin, only on Docker
UDP 518200.0.0.0WireGuard, application networking

Sizing guide​

This section will explain how to dimension a host (CPU, memory, and disk) from a system perspective, i.e., what resources the Edge Enforcer consumes, not including any applications.

CPU​

The Edge Enforcer does very little when idle and will not consume any significant CPU. However, CPU consumption will increase during the scheduling of applications. In addition, if any applications use the encryption/decryption functions in Strongbox, this will also increase CPU consumption by the Edge Enforcer.

Memory​

An Edge Enforcer with no applications running (and no configuration, e.g., secrets) consume ~120 Mb of memory. As applications and configurations are distributed to the site, the consumption will increase.

Disk size​

All images are stored in the Avassa system's internal image registry, and when applications are launched, images are copied into the Docker cache. The directories are

  • /var/lib/supd/state/mona - Avassa image registry
  • /var/lib/docker/overlay2 - Dockers image cache (and Copy on Write layers)

Care needs to be taken to have enough space to store each image (both the two Edge Enforcer images and application images) in both locations.

Edge Enforcer​

The Edge Enforcer container itself it about 200Mb (168Mb as of mid-2022). Since each site will automatically upgrade itself to new versions, the host must have space for a new version as well as one version back. The worst case is therefore 800 Mb (two versions in Avassa's registry and two versions in the Docker cache).

Volga topics​

All Volga topics are stored in the /var/lib/supd/state/volga directory.

The system has 7 volga topics per tenant:

Topic nameMax Sixe per HostMax replicas
system:application-metrics100 Mb3
system:audit-trail-log10 Mb3
system:deployment-events10 Mb3
system:logs10 Mb3
system:scheduler-events10 Mb3
Sum~140 Mb

NOTE If fewer than three hosts in the site, replicas will be 1.

The system tenant (not visible from the outside) has 10 topics, plus one log topic per host in the site.

Topic nameMax Sixe per HostMax replicas
infra:result:disk1 Gb3
infra:up:disk1 Gb3
strongbox-cert-monitor10 Mb3
supd-logs:<host-name>10 Mbone per host in the site
system:all-scheduler-events10 Mb3
system:audit-trail-log10 Mb3
system:deployment-events10 Mb3
system:host-metrics100 Mb3
system:scheduler-events10 Mb3
system:unauthenticated-audit-trail-log10 Mb3
telemetry:site-collected100 Mb3
Sum~2.3 Gb

NOTE If fewer than three hosts in the site, replicas will be 1.

Application logs​

Application logs are captured by the Edge Enforcer and stored in Volga topics. Each container instance gets a log topic. By default a log topic will consume 100Mb when fully utilized. The size limit and archiving options for application logs can be configured in the application specification. Configured values will effect the application log size.

System Sizing summary​

  • 1 vCPU (may need more if encryption/decryptions functions are heavily used)
  • 2.5 Gb of disk for system Volga topics
    • /var/lib/supd/state/volga
  • 800Mb of disk for Edge Enforcer upgrades (not including any volumes needed by the application)
    • 400 Mb - /var/lib/supd/state/mona
    • 400 Mb - /var/lib/docker/overlay2

Sizing guide for applications​

This part will not detail CPU and memory requirements for the application itself. This is highly dependent on the application.

CPU​

As mentioned above, if an application utilizes Strongbox encryption/decryption features, this will cause the Edge Enforce to consume more CPU resources.

Memory​

The Edge Enforcer will consume some memory for each application that is running. The amount of memory is dependent on the amount of configuration stored in the application specification and in Strongbox. To give a hint, we deployed 100 instances of an example application and measured the memory consumption of the Edge Enforcer before and after deploying the application instances.

Before deploying the applications, the Edge Enforcer consumed (docker stats supd) 112 Mb. After deploying the 100 application instances, the consumption was 126 Mb, with a peak consumption of 550 Mb.

Therefore we can conclude that each application, at steady state, causes the Edge Enforcer to consume around 0.14 Mb of memory per application.

Disk size​

Application images​

On a site, each application image is stored in the Edge Enforcer's registry. For sites with multiple controllers, each image is replicated to three controller nodes. When an application is scheduled to run on a particular host, the container images are loaded into that host's Docker cache. So for specific hosts, worst case, each image will consume twice its size (internal registry and Docker cache).

Volga topics​

All Volga topics are stored in the /var/lib/supd/state/volga directory. Each container's log topic will consume a maximum of 100Mb.

Kernel Configuration Options​

When using a custom kernel, the following is a non-exhausting list of configuration options known to be required or recommended for operation of the product or separate features.

Note that the Avassa platform has its own implementation of the networking functionality, including firewall configuration, and does not use the container engine built-in functionality. This means that when it comes to networking features these kernel requirements supersede the container engine's kernel requirements. In particular, no iptables support is required because the Avassa platform uses nftables.

  • kernel syscalls used by Docker, required

    • CONFIG_FUTEX
    • CONFIG_EPOLL
    • CONFIG_FILE_LOCKING
    • CONFIG_POSIX_MQUEUE

  • kernel syscalls used by Erlang runtime powering the Avassa platform, required

    • CONFIG_FUTEX
    • CONFIG_EPOLL
    • CONFIG_TIMERFD
    • CONFIG_ADVISE_SYSCALLS

  • linux namespaces for basic containerization, required

    • CONFIG_NAMESPACES
    • CONFIG_UTS_NS
    • CONFIG_IPC_NS
    • CONFIG_PID_NS
    • CONFIG_NET_NS

  • linux namespaces for increased isolation, recommended

    • CONFIG_USER_NS

  • cgroups for resource isolation and accounting, required

    • CONFIG_CGROUPS
    • CONFIG_CGROUP_CPUACCT
    • CONFIG_CGROUP_DEVICE
    • CONFIG_CGROUP_SCHED
    • CONFIG_CGROUP_BPF
    • CONFIG_MEMCG

  • other recommended cgroups

    • CONFIG_CPUSETS

  • CPU bandwidth provisioning, required

    • CONFIG_CFS_BANDWIDTH

  • kernel syscall isolation, required

    • CONFIG_SECCOMP

  • kernel feature used by the Avassa platform for host identification, recommended

    • CONFIG_DMIID

  • communication between the Avassa platform and the container engine, required

    • CONFIG_UNIX

  • dynamic changing of kernel parameters, among other things to enable IP forwarding, required

    • CONFIG_PROC_SYSCTL

  • nftables firewall features, required

    • CONFIG_NF_TABLES
    • CONFIG_NF_TABLES_INET
    • CONFIG_NFT_CT
    • CONFIG_NFT_COUNTER (for Linux 5.16 and older)
    • CONFIG_NFT_MASQ
    • CONFIG_NFT_REDIR
    • CONFIG_NFT_NAT
    • CONFIG_NFT_REJECT

  • optional for DCCP or SCTP transport layer protocols support

    • CONFIG_IP_DCCP
    • CONFIG_NF_CT_PROTO_DCCP
    • CONFIG_IP_SCTP
    • CONFIG_NF_CT_PROTO_SCTP

  • application networking, required

    • CONFIG_VETH
    • CONFIG_BRIDGE
    • CONFIG_PACKET

  • encrypted site overlay network, required

    • CONFIG_WIREGUARD
    • CONFIG_VXLAN

  • QoS parameters to ensure fair access to network and enable limiting network bandwidth, recommended

    • CONFIG_NET_SCH_HFSC
    • CONFIG_NET_SCH_FQ_CODEL

  • filesystem requirements, required

    • CONFIG_EXT4_FS or CONFIG_XFS_FS
    • CONFIG_OVERLAY_FS
    • CONFIG_TMPFS
    • CONFIG_PROC_FS

  • filesystem quota for disk space isolation, recommended

    • CONFIG_QUOTA
    • CONFIG_QFMT_V2
    • CONFIG_XFS_QUOTA (if CONFIG_XFS_FS)