Host requirements and sizing

Requirements

This section explains the requirements to install and run the Edge Enforcer.

Host requirements

Make sure the following requirements are fulfilled before proceeding with the installation preparations:

• A host running a Linux distribution with the x86_64 or AArch64/ARM64 kernel version 5.4 or later.
• WireGuard is included in Linux kernel versions 5.6 and later, please see here for how to set it up on older kernels.
• Systemd running on the host.
• Minimum 2GB RAM: the edge enforcer consumes between 100 and 500Mb, that leaves 1.5Gb for operating system and applications
• Minimum 1 vCPU not dedicated
• Minimum 2 GB disk space

The installation script supports the following distributions

• Alpine
• CentOS 9
• Debian 11
• Debian 12
• Debian 13
• RHEL 9
• RHEL 10
• Rocky 9
• Rocky 10
• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 24.04

If you have a custom kernel, please see Kernel Configuration Options.

Network requirements

Host/site to Control Tower

All hosts on a site must be able to connect to the Control Tower on the following ports.

• TCP 443 (mandatory)
• UDP 443 (recommended, specifically allow the QUIC protocol in the firewall)

Firewall between hosts/sites and Control Tower

If you have a firewall between the sites and Control Tower and want to lock down the traffic you must whitelist the following DNS entries:

• api.<environment name>.<organization name>.avassa.net
• volga.<environment name>.<organization name>.avassa.net
• registry.<environment name>.<organization name>.avassa.net

For example, if your UI is at test.acme.avassa.net, you should whitelist:

• api.test.acme.avassa.net
• registry.test.acme.avassa.net
• volga.test.acme.avassa.net

warning

Do not whitelist IP addresses, the IP addresses can change over time.

Host firewall

Host firewall configuration is managed by the Edge Enforcer and in most cases there should be no additional firewall rules. In cases where it is necessary to have custom firewall configuration alongside the rules installed by the Edge Enforcer, the custom rules must follow the rules outlined in Host firewall.

Between hosts in a site

If you have a firewall between hosts in the same site, make sure the following ports are open:

• UDP/TCP 53 (optional, if you are using the Edge Enforcer DNS Server)
• TCP 4668 (host detection and cluster communication)
• UDP 51820 (WireGuard, application networking)

IP addresses within a site

note

This requirement does not apply to single-host sites.

The IP addresses of the controller hosts, used for communication with other hosts within the site, must remain unchanged for the lifetime of the site. This usually means that the IP addresses should either be statically configured, or reserved in the DHCP server configuration. The reason for this is that the IP address acts as an identifier that controllers use to reach each other and the worker hosts use to reach controllers.

The worker hosts may have variable IP addresses as the connection is initiated by the workers toward the controllers and never the other way around.

This requirement does not apply to sites consisting of a single host (one controller with no workers).

Host ports

On the host, the Edge Enforcer will bind to the following ports. This is just informational to avoid port conflicts with other services.

Port	Listen Address	Description
UDP/TCP 53	localhost	DNS Server (forwarded to port 4653)
UDP/TCP 4653	0.0.0.0	DNS Server
TCP 4646	0.0.0.0	HTTPS REST API
TCP 4664	0.0.0.0	HTTP CRL (Certificate Revocation List) API
TCP 4668	0.0.0.0	Host detection and cluster communication within a site, TLS encrypted
TCP 4848	0.0.0.0	HTTPS Container registry
TCP 4849	localhost	HTTP Container registry
TCP 4949	localhost	Docker plugin, only on Docker
TCP 4950	localhost	Docker plugin, only on Docker
UDP 51820	0.0.0.0	WireGuard, application networking

Sizing guide

This section will explain how to dimension a host (CPU, memory, and disk) from a system perspective, i.e., what resources the Edge Enforcer consumes, not including any applications.

CPU

The Edge Enforcer does very little when idle and will not consume any significant CPU. However, CPU consumption will increase during the scheduling of applications. In addition, if any applications use the encryption/decryption functions in Strongbox, this will also increase CPU consumption by the Edge Enforcer.

Memory

An Edge Enforcer with no applications running (and no configuration, e.g., secrets) consume ~120 Mb of memory. As applications and configurations are distributed to the site, the consumption will increase.

Disk size

All images are stored in the Avassa system's internal image registry, and when applications are launched, images are copied into the Docker cache. The directories are

• /var/lib/supd/state/mona - Avassa image registry
• /var/lib/docker/overlay2 - Dockers image cache (and Copy on Write layers)

Care needs to be taken to have enough space to store each image (both the two Edge Enforcer images and application images) in both locations.

Edge Enforcer

The Edge Enforcer container itself it about 200Mb (168Mb as of mid-2022). Since each site will automatically upgrade itself to new versions, the host must have space for a new version as well as one version back. The worst case is therefore 800 Mb (two versions in Avassa's registry and two versions in the Docker cache).

Volga topics

All Volga topics are stored in the /var/lib/supd/state/volga directory.

The system has 7 volga topics per tenant:

Topic name	Max Sixe per Host	Max replicas
system:application-metrics	100 Mb	3
system:audit-trail-log	10 Mb	3
system:deployment-events	10 Mb	3
system:logs	10 Mb	3
system:scheduler-events	10 Mb	3
Sum	~140 Mb

NOTE If fewer than three hosts in the site, replicas will be 1.

The system tenant (not visible from the outside) has 10 topics, plus one log topic per host in the site.

Topic name	Max Sixe per Host	Max replicas
infra:result:disk	1 Gb	3
infra:up:disk	1 Gb	3
strongbox-cert-monitor	10 Mb	3
supd-logs:<host-name>	10 Mb	one per host in the site
system:all-scheduler-events	10 Mb	3
system:audit-trail-log	10 Mb	3
system:deployment-events	10 Mb	3
system:host-metrics	100 Mb	3
system:scheduler-events	10 Mb	3
system:unauthenticated-audit-trail-log	10 Mb	3
telemetry:site-collected	100 Mb	3
Sum	~2.3 Gb

NOTE If fewer than three hosts in the site, replicas will be 1.

Application logs

Application logs are captured by the Edge Enforcer and stored in Volga topics. Each container instance gets a log topic. By default a log topic will consume 100Mb when fully utilized. The size limit and archiving options for application logs can be configured in the application specification. Configured values will effect the application log size.

System Sizing summary

• 1 vCPU (may need more if encryption/decryptions functions are heavily used)
• 2.5 Gb of disk for system Volga topics
  • /var/lib/supd/state/volga
• 800Mb of disk for Edge Enforcer upgrades (not including any volumes needed by the application)
  • 400 Mb - /var/lib/supd/state/mona
  • 400 Mb - /var/lib/docker/overlay2

Sizing guide for applications

This part will not detail CPU and memory requirements for the application itself. This is highly dependent on the application.

CPU

As mentioned above, if an application utilizes Strongbox encryption/decryption features, this will cause the Edge Enforce to consume more CPU resources.

Memory

The Edge Enforcer will consume some memory for each application that is running. The amount of memory is dependent on the amount of configuration stored in the application specification and in Strongbox. To give a hint, we deployed 100 instances of an example application and measured the memory consumption of the Edge Enforcer before and after deploying the application instances.

Before deploying the applications, the Edge Enforcer consumed (docker stats supd) 112 Mb. After deploying the 100 application instances, the consumption was 126 Mb, with a peak consumption of 550 Mb.

Therefore we can conclude that each application, at steady state, causes the Edge Enforcer to consume around 0.14 Mb of memory per application.

Disk size

Application images

On a site, each application image is stored in the Edge Enforcer's registry. For sites with multiple controllers, each image is replicated to three controller nodes. When an application is scheduled to run on a particular host, the container images are loaded into that host's Docker cache. So for specific hosts, worst case, each image will consume twice its size (internal registry and Docker cache).

Volga topics

All Volga topics are stored in the /var/lib/supd/state/volga directory. Each container's log topic will consume a maximum of 100Mb.

Disk Layout

To protect the Operating System from applications accidentally taking too much disk space, we recommend having the OS on a separate partition/disk.

Relevant paths

The following paths are recommended to mount from a non OS disk/partition.

Path	Quota	Description
/var/lib/supd/state	x	Avassa state directory including distributed registry
/var/lib/supd/volumes/_auto	x	Automatic volume. If exists, the Avassa system will automatically generate a volume here without any extra site configuration.
/var/lib/supd/volumes/<volume name>	x	User defined volumes
/var/lib/docker		Docker storage
/var/lib/containers/storage		Podman storage

Paths with Quota annotation should be mounted with quota support.

The choice of filesystem can be freely made, but Debian derivatives tend to favor ext4 and RHEL/Rocky favor xfs.

Examples

Developer	Minimal Production	Separate Disks/Partition
Everything in one partition	OS partition	OS disk
Create the auto volume as a directory	One data disk/partition with quota support, bind mounts to correct locations.	Separate partitions/disks
	Cloud init example

See storage configuration how to on how to practically set up the filesystems.

Kernel Configuration Options

When using a custom kernel, the following is a non-exhausting list of configuration options known to be required or recommended for operation of the product or separate features.

Note that the Avassa platform has its own implementation of the networking functionality, including firewall configuration, and does not use the container engine built-in functionality. This means that when it comes to networking features these kernel requirements supersede the container engine's kernel requirements. In particular, no iptables support is required because the Avassa platform uses nftables.

• kernel syscalls used by Docker, required

  • CONFIG_FUTEX
  • CONFIG_EPOLL
  • CONFIG_FILE_LOCKING
  • CONFIG_POSIX_MQUEUE

• kernel syscalls used by Erlang runtime powering the Avassa platform, required

  • CONFIG_FUTEX
  • CONFIG_EPOLL
  • CONFIG_TIMERFD
  • CONFIG_ADVISE_SYSCALLS

• linux namespaces for basic containerization, required

  • CONFIG_NAMESPACES
  • CONFIG_UTS_NS
  • CONFIG_IPC_NS
  • CONFIG_PID_NS
  • CONFIG_NET_NS

• linux namespaces for increased isolation, recommended

  • CONFIG_USER_NS

• cgroups for resource isolation and accounting, required

  • CONFIG_CGROUPS
  • CONFIG_CGROUP_CPUACCT
  • CONFIG_CGROUP_DEVICE
  • CONFIG_CGROUP_SCHED
  • CONFIG_CGROUP_BPF
  • CONFIG_MEMCG

• other recommended cgroups

  • CONFIG_CPUSETS

• CPU bandwidth provisioning, required

  • CONFIG_CFS_BANDWIDTH

• kernel syscall isolation, required

  • CONFIG_SECCOMP

• kernel feature used by the Avassa platform for host identification, recommended

  • CONFIG_DMIID

• communication between the Avassa platform and the container engine, required

  • CONFIG_UNIX

• dynamic changing of kernel parameters, among other things to enable IP forwarding, required

  • CONFIG_PROC_SYSCTL

• nftables firewall features, required

  • CONFIG_NF_TABLES
  • CONFIG_NF_TABLES_INET
  • CONFIG_NFT_CT
  • CONFIG_NFT_COUNTER (for Linux 5.16 and older)
  • CONFIG_NFT_MASQ
  • CONFIG_NFT_REDIR
  • CONFIG_NFT_NAT
  • CONFIG_NFT_REJECT

• optional for DCCP or SCTP transport layer protocols support

  • CONFIG_IP_DCCP
  • CONFIG_NF_CT_PROTO_DCCP
  • CONFIG_IP_SCTP
  • CONFIG_NF_CT_PROTO_SCTP

• application networking, required

  • CONFIG_VETH
  • CONFIG_BRIDGE
  • CONFIG_PACKET

• encrypted site overlay network, required

  • CONFIG_WIREGUARD
  • CONFIG_VXLAN

• QoS parameters to ensure fair access to network and enable limiting network bandwidth, recommended

  • CONFIG_NET_SCH_HFSC
  • CONFIG_NET_SCH_FQ_CODEL

• filesystem requirements, required

  • CONFIG_EXT4_FS or CONFIG_XFS_FS
  • CONFIG_OVERLAY_FS
  • CONFIG_TMPFS
  • CONFIG_PROC_FS

• filesystem quota for disk space isolation, recommended

  • CONFIG_QUOTA
  • CONFIG_QFMT_V2
  • CONFIG_XFS_QUOTA (if CONFIG_XFS_FS)