Skip to main content

Add a site

Introduction​

A site is a collection of one or more hosts, virtual or physical, that typically reside in the same physical location. Each host has an edge enforcer installed.

The Avassa Control Tower and each host share a common secret, the host-id. This is also how you configure which hosts belongs to the same site. You can either configure (pre-provision) the site with its hosts and host-ids before installing the hosts. Or you can run the installation script on the hosts and later add them to the Control Tower.

In this tutorial you will go through the following steps:

  1. Install Edge Enforcer

Use the installation script to install the Edge Enforcer and its dependencies.

  1. Create site configuration

Collect host IDs from each host and set these in the site configuration

For troubleshooting this procedure, see Site maintenance.

For sizing and requirements, see Host requirements and sizing.

Host changes made by the install script​

The install script will install the following components:

  • docker (or podman if invoked with --podman flag)
  • jq
  • curl
  • python3 (if supctl is installed)
  • python3-websockets (if supctl is installed)
note

The Edge Enforcer can not be installed on a Ubuntu system with docker running in a snap. Please remove docker with sudo snap remove docker, the install script will automatically install docker.

All iptables rules are flushed, the Edge Enforcer configures the firewall using nftables.

Install the Edge Enforcer​

The Edge Enforcer is a software agent that runs on each host that is to be managed by the Control Tower.

The Edge Enforcer is delivered as a container image (called supd) and is expected to run on a Docker or Podman container runtime with a set of privileges allowing it to run and monitor end-user containers.

note

If the host is registered in the Control Tower before the installation script is run, the Edge Enforcer can be downloaded immediately.

If the host is registered in the Control Tower after the installation script is run, all dependencies are installed and the systemd unit will periodically try to download the Edge Enforcer in the background.

note

Assuming your UI is at test.acme.avassa.net, the address below would be api.test.acme.avassa.net.

The exact command can be fetched in the UI, click the user icon in the top right corner, expand Resources and clock Install Edge Enforcer.

Install EE

A dialog like this will show up. Install EE cmd

The actual installation

curl -s https://api.test.acme.avassa.net/install | sudo sh

This will install the Edge Enforcer and all its prerequisites.

note

There are more options to the install script, please see

curl -s https://api.test.acme.avassa.net/install | sudo sh -s -- --help
Install Script for Avassa Edge Enforcer. Usage:

  sudo ./install-supd [OPTIONS] ENVIRONMENT-ADDRESS

ENVIRONMENT-ADDRESS is the address of a Control Tower

Accepted OPTIONS are:
  -i, --host-id HOST-ID  When calling home use HOST-ID instead of the default
  -y, --yes              Do not prompt, assume 'yes' to all questions
  -n, --dry-run          Dry run mode, do not actually install anything
  -s, --no-start         Do not start supd after installation
  -h, --help             Show this help text and then exit
  -c, --install-supctl   Also install the supctl tool
  --podman               Install Podman (default: install Docker)
  --no-userns-remap      Do not enable user namespaces in daemon settings
  --cert CERT            When calling home use the supplied certificate
  --install-prefix       Install executables under this path instead of /usr
  --preflight            Run installation preflight checks

One of the following options emits the contents of the corresponding file
that would be installed, without running the installation:
  --emit-start-supd           Emit start-supd script
  --emit-systemd-service      Emit /etc/systemd/supd.service
  --emit-openrc-service       Emit /etc/init.d/supd
  --emit-openrc-start-script  Emit /usr/sbin/start-supd-openrc
  --emit-uninstall-supd       Emit uninstall-supd script

The following flag runs a standalone action without running the installation:
  --get-host-id               Emit the host-id as seen by supd
  --uninstall                 Remove supd

Preflight installation checks​

To run the preflight checks: curl -s https://api.test.acme.avassa.net/install | sudo sh -s -- --preflight.

This will output something like this:

Required Kernel Modules
-----------------------

All required kernel modules available

Recommended Kernel Modules
--------------------------

All recommended kernel modules available

Port Status
-----------
4646    Ok
4668    Ok
4848    Ok
51820   Ok

Control Tower Connectivity
--------------------------
Checking https://api.test.acme.avassa.net/healthz... Ok
Checking api.test.acme.avassa.net TCP port 443... Ok
Checking registry.test.acme.avassa.net TCP port 443... Ok
Checking test.acme.avassa.net TCP port 443... Ok
Checking volga.test.acme.avassa.net UDP port 443... Ok

Minimal Disk Requirements
-------------------------
Filesystem     Avail
/dev/nvme0n1p2  369G

Make sure you have at least 500Mb of free disk where you install the Edge Enforcer

Minimal Memory Requirements
---------------------------
Total 131722756 kB of memory... Ok

Podman-based installation​

By default the install script uses Docker as a container engine (the container engine is installed if not present). If a Podman-based installation is desired, then the install script should be invoked with --podman flag, e.g. as follows:

curl -s https://api.test.acme.avassa.net/install | sudo sh -s -- --podman

Note that the Avassa solution is not compatible with Podman older than Podman 4.0.0. This means some older distributions are not supported. Some of the minimum OS versions required for Podman-based installation are:

  • Debian 12
  • Ubuntu 22.04
  • CentOS 9
  • RHEL 9

Special case: requirement to use host networking​

By default the installation script enables the user namespaces feature in docker for increased isolation. However, when there is a requirement to run services attached to the "host network" (host's default network namespace) the user namespaces feature needs to be disabled due to a Docker limitation. The best way to ensure that the user namespaces feature is disabled in Docker configuration is to run the installation script with --no-userns-remap flag as follows:

curl -s https://api.test.acme.avassa.net/install | sudo sh -s -- --no-userns-remap
note

This is only relevant for Docker-based installation.

Special case: manually assign host id​

If you need to manually assign a host id during the installation you can pass that as a parameter:

curl -s https://api.test.acme.avassa.net/install | sudo sh -s -- --host-id <name>

Whenever you need to retrieve the host id you can do that with:

curl -s https://api.test.acme.avassa.net/scripts/get-host-id | sudo sh

This will output something like 7491b30f-655a-46cb-a947-0dfb3adc5c66.

supd.conf​

The installation generates a supd.conf file in /etc/supd It contains the host name and call home server. You can edit this file to for example control IP overlay allocation or if you need to assign some hosts as Control Tower proxies.

A default supd.conf example is shown below

host-id: "skehamn-host"
initial-site-config:
  call-home-servers:
    - api.test.acme.avassa.net

Whenever you edit the supd.conf file you need to restart supd to apply the changes:

systemctl restart supd

Special case: configure a Control Tower proxy​

If only some hosts (or a single host) within the edge site has IP connectivity to the Control Tower you can assign them to become proxies for the hosts without IP-connectivity. This is done by manually configure the supd.conf file.

note

You can use the same supd.conf for all hosts within the edge site independent if the host will act as a proxy or is in need of a proxy.

A supd.conf example with a Control Tower proxy configured is shown below

host-id: "skehamn-host"
initial-site-config:
  call-home-servers:
    - api.test.acme.avassa.net
  parent-proxy-ips:
    - 192.168.100.102
    - 192.168.100.103
  parent-proxy-call-home-port: 5657
  parent-proxy-api-port: 5656
  parent-proxy-registry-port: 5858
  parent-proxy-volga-port: 5959

In this example, the hosts with internal IP addresses 192.168.100.102 and 192.168.100.103 will act as proxies for all other hosts in the site. Ports are optional and may be omitted and system default proxy ports will be used.

Configure the site in Control Tower​

Select Sites in the left-hand menu and click +Add.

Sites

Give the site a name and optionally labels, location etc.

New Site

For each host, click the New host button and enter the host-id that was prompted by the corresponding install script on that host.

Add host

After adding all hosts, click to save the site.

The site will now be in a state where the site is not established and no host has called home: No call home

Configure ingress IP allocation method for the new site​

Ingress IP addresses is an important feature for the edge applications to be able to receive incoming traffic. In order for the ingress addresses to be reachable from outside the external network must be able to route the traffic to the relevant host in the edge site by the ingress IP address. The Avassa platform is not aware of the network configuration in the edge site, hence it is the responsibility of the site provider to indicate the relevant method the ingress IP addresses are allocated on an edge site.

By default the IP address allocation is disabled, and any application requesting an ingress address will fail to deploy. Should the edge applications be able to request ingress addresses one of the following methods needs to be configured:

  • DHCP. With this method supd requests an ingress IP address from a DHCP server on behalf of the application. The DHCP server is not managed by the Avassa platform. It must support DHCP client identifier option and be reachable from any host in the edge site.
  • configurable IP address pool (sitewide or per network interface per host). With this method the Avassa platform allocates the IP address from the pool, assuming that any address in the pool is routable from the outside network to the corresponding host interface.

Indicate that the ingress IP should be allocated by making a DHCP request. It is assumed that the DHCP server is reachable from any host within the site. No additional configuration is required.

supctl merge system sites my-site <<EOF
ingress-allocation-method: dhcp
EOF

Special case: specify intra site communication network​

By default, the edge enforcers discovers the networks they are able to communicate on. In certain cases you may need to restrict this. You can whitelist networks by adding the following configuration to your site.

supctl merge system sites my-site <<EOF
cluster:
  site-networks:
    - 192.168.2.0/24
EOF

In the example above, all control traffic between the hosts on a site will go over the 192.168.2.0/24 network.

note

Make sure you update insecure-registries in /etc/docker/daemon.json to match the above network. This is only relevant for Docker-based installation.

Special case: allow local unseal​

By enabling local unseal in the site, the site can unseal (decrypt) itself without network connection to the Control Tower. This will decrease security so make sure you understand the implications.

Site call home status​

After a short while, the site should have called home, you can verify this by checking the state of the site and the host

Called home

Inspect supd logs​

By inspecting the Docker logs on the local host you can figure out the current state of the call-home process, and if the call-home sequence terminated successfully.

docker logs supd

You can also find valuable information by inspecting the systemd log files

journalctl -xu supd -f

Logs during call home with no response​

The following example log entries below indicate that the call-home process is running, but is currently rejected by the Control Tower due to lack of matching configuration on the server side. This is the expected behaviour until the Control Tower has been configured to recognise the host-id.

docker logs supd
...
<NOTICE> 2021-04-06 12:08:00.311348+00:00 boot_callhome:82 UNKNOWN_CHOSTNAME <0.9.0>
reply from mothership 403 <<>> (retrying 264)

<NOTICE> 2021-04-06 12:08:10.344054+00:00 boot_callhome:82 UNKNOWN_CHOSTNAME <0.9.0>
reply from mothership 403 <<>> (retrying 263)

<NOTICE> 2021-04-06 12:08:20.382955+00:00 boot_callhome:82 UNKNOWN_CHOSTNAME <0.9.0>
reply from mothership 403 <<>> (retrying 262)

Logs after successful call home​

After successfully calling home, a lot of logs on e.g. IP connectivity is printed. At the end you should see a System started log entry, this indicates the Edge Enforcer has successfully connected to the Control Tower, received its configuration and is ready to accept services.

<NOTICE> 2021-04-07 13:55:48.989500+00:00 dapid:47 sthlm-002 <0.1072.0>
dapid loaded and compiled routes
<NOTICE> 2021-04-07 13:55:48.990100+00:00 dapid_sup:27 sthlm-002 <0.1072.0>
starting http server on port 4646
<NOTICE> 2021-04-07 13:55:48.990287+00:00 dapid_sup:50 sthlm-002 <0.1072.0>
starting registry http server on port 4848
<NOTICE> 2021-04-07 13:55:49.008620+00:00 dapid_sup:77 sthlm-002 <0.1072.0>
starting docker_dc_net plugin http server on port 4949
<NOTICE> 2021-04-07 13:55:49.008719+00:00 dapid_sup:77 sthlm-002 <0.1072.0>
starting docker_gw_net plugin http server on port 4950
<NOTICE> 2021-04-07 13:55:49.013052+00:00 syst_server:110 sthlm-002 <0.940.0>
System started.

Host ID configuration​

The host-id string that uniquely identifies the host to the call-home server (the Control Tower) for identification purposes. There are currently two ways to configure this value, either as a fixed string, or to configure supd to read a value from DMI.

  • The fixed string approach requires you to manually edit the name of each host in its configuration file. This may become an issue if you plan on deploying many hosts.
  • The DMI-approach uses the DMI/SMBIOS information in the /sys/class/dmi/id virtual filesystem to extract the product UUID of the local host computer and uses that as the host-id. Notice that the string after the DMI: prefix represents the content of a file with that name in the filesystem.

For example, by specifying DMI: board_serial, the Edge Enforcer will read /sys/class/dmi/id/board_serial and report that value as the host-id.

By using the DMI approach you do not have to create unique configuration files for each host since the values are generated and unique.

Please Note: some hypervisors assign the same UUID to VMs which may result in host-id collisions. If using the DMI-approach, make sure the hosts get unique IDs and if that is not feasible, use the fixed-string approach.

Manual Installation​

To manually install the Edge Enforcer, or to understand what the installation script does, please download it and open it up in your favourite editor. The script itself is well documented.

curl -O https://api.test.acme.avassa.net/install

Uninstall the Edge Enforcer​

The first step is to delete the site in Control Tower, either in the UI or:

supctl delete system sites sthlm-5

Using the uninstall script​

At a succeful installation of the Edge Enforcer, you can run

/var/lib/supd/state/bin/uninstall-supd

This will remove everything installed by the Edge Enforcer installer.

Manual uninstall​

Stop the service

systemctl stop supd

Clean up the state and volume directories

umount -R /var/lib/supd/volumes
umount -R /var/lib/supd/state/secrets
umount -R /var/lib/supd/state/dhclient/_pid
rm -rf /var/lib/supd

Remove the service and configuration

rm /etc/systemd/system/supd.service
rm -rf /etc/supd/

Remove Avassa's docker images

docker images | awk '/avassa/ {print $1}' | xargs docker rmi -f