Managing disconnected sites

This instruction explains how a site-local administrator can use the Avassa command line supctl to inspect site and application state as well as perform local modifications.

Note that you can easily build a site local Web UI and deploy to your sites. A template exists in the Avassa public repos.

In these examples we will assume that the site-local operator is a user in the site provider tenant. We will also assume that the applications are deployed using the same tenant.

When running disconnected it is likely that you do not have access to an external Oauth2 user database. Therefore make sure that you have created Avassa local users, which are distributed to all sites. This user has to be created in the Control Tower before the site goes disconnected. Also make sure that the users are distributed to the appropriate sites, for example in the .

Read more on user distribution in the reference guide.

It is a good practice to assign policies to users in order to control what the local users are allowed to do.

Another good practice is to have a site local repository of the application YAML definitions that you might need to modify locally. (Other configuration is replicated).

All interactions with a site is typically done through the Control Tower. However, when a site is disconnected from the Control Tower, certain operations can still be performed. E.g. consuming volga topics, checking application status and restarting applications. These are all examples of operations that does not modify the configuration and thereby not creating any conflicts.

But there are occasions where a site-local operator, while being disconnected, need to add, update and/or reconfigure applications locally on the site.

To be able to fully maintain applications while being disconnected from Control Tower, site-local operators may also change site configuration such as ingress IP pools or secret values locally on the site. Other site provider operations such as drain-host and reschedule are also supported while being disconnected.

The following entities can be modified locally:

• application specifications
• deployment specifications
• site configuration
• secrets

Users, policies, tenants etc are read-only at the site. Make sure that you have created the configuration of these while your site is guaranteed online.

Hosts can be added to the site configuration while disconnected, but they will not be functional within the site while disconnected. You will have to wait until there is a connection to the Control Tower until they can be used at the site.

After a scenario for a disconnected site, where local changes has been made, and it gets connected again, the operator might need to take certain actions to make sure the system is in sync.

The resync strategy when a site gets connected after being disconnected for a while is as follows:

• if no configuration changes involving the site has been made at Control Tower while the site was disconnected,
  any local changes on the site will remain.
• if configuration changes involving the site has been made at Control Tower while the site was disconnected,
  any local changes on site X will be overridden with changes from Control Tower.

To not confuse any observer, any locally created application and application deployment configuration will be deleted whenever the same application is scheduled in (and hence propagated from) the Control Tower.

Another fundamental piece to know before showing some examples is a subtle difference in semantics regarding applications and deployments vs site configuration and secrets.

The sites have a copy of the configuration for site configuration and secrets. This means that they can be modified directly on the site.

Applications deployed from the Contol Tower do not have a configuration representation on the site. They result from the scheduler and therefore have a read-only state representation locally at the site. This means that to modify a centrally deployed application while you are disconnected at the site, you must create a site-local “copy” before you can edit, either by having the original YAML or to use the supctl query parameter

--query-params 'content=config' to show the configuration of the application at the site.

Running supctl locally on the site

Download supctl

Download supctl from the local Edge Enforcer:

curl -k -OL https://localhost:4646/supctl
chmod +x supctl

Log in

Assuming you have supctl on one of the site hosts.

First you need to allow supctl to accept the site-local certificate

./supctl -k --host localhost -H api --port 4646 do get-api-ca-cert

{
  "cert": "-----BEGIN CERTIFICATE-----\nMIICGTCCAcCgAwIBAgITAOlwouO+gpFsbw96b4bHriArtzAKBggqhkjOPQQDAjBj\nMRgwFgYDVQQDEw9BdmFzc2EgQVBJIHJvb3QxEjAQBgNVBAcTCVN0b2NraG9sbTEL\nMAkGA1UEBhMCU0UxDzANBgNVBAoTBkF2YXNzYTEVMBMGA1UECxMMZGlzdHJpYnV0\naW9uMCIYDzIwMjMwNTA1MTEzNTAxWhgPMjAyNjA5MTMxNTExMDFaMGMxGDAWBgNV\nBAMTD0F2YXNzYSBBUEkgcm9vdDESMBAGA1UEBxMJU3RvY2tob2xtMQswCQYDVQQG\nEwJTRTEPMA0GA1UEChMGQXZhc3NhMRUwEwYDVQQLEwxkaXN0cmlidXRpb24wWTAT\nBgcqhkjOPQIBBggqhkjOPQMBBwNCAATAjpAziL+dqP8+cClU9gyUMxdI2vtfj3JJ\nz0mOah5OK2+vNJgEdQqbU3ErFKsp1/PN/sFnP/D+Fo25vFYlvxFHo08wTTAOBgNV\nHQ8BAf8EBAMCAYYwEgYDVR0TAQH/BAgwBgEB/wIBATAnBgNVHR8EIDAeMBygGqAY\nhhZodHRwOi8vY3JsLmF2YXNzYS5uZXQvMAoGCCqGSM49BAMCA0cAMEQCIEWMVYEc\nakMoeZgm6CsCpxUw0PiTb1riP4o+w3NZEENLAiBX5OhKkKv6IcXlJf/kQCkRxqAO\nXoTHBoNB7OyKyRTEjA==\n-----END CERTIFICATE-----"
}

The parameters are explained below:

• -k allow connection without verifying the cert, just for the first call where we get the CA
• --host localhost connect to Edge Enforcer locally
• -H api specify the server name in the certificate
• --port 4646, locally the Edge Enforcer listens on port 4646

Now login using an Avassa user that you previously have created. Note that whenever you create a user in the Avassa Control Tower that is replicated to all sites. That enables a site local login even when the site is disconnected.

./supctl do login my-user@example.com

You should get an output similar to this

Created profile localhost.
password:
{
  "token": "ec01cb75-6746-4f73-9993-1ac2051fd004",
  "expires-in": 1209600,
  "expires": "2023-05-22T14:50:40.645159Z",
  "accessor": "819bdcd8-86e6-4012-8f76-32ca227faf48",
  "creation-time": "2023-05-08T14:50:40.645159Z",
  "renewal-time": "2023-05-08T14:50:40.645159Z",
  "totp-required": false,
  "totp-enabled": false
}

Accessing the site from another host on the network

As a security measure, by default, the API is only accessible from localhost. If you need to access the APIs (which supctl does) from the local network, you need to configure this in the site configuration

name: my-site
...
management-ipv4-access-list:
  - 192.128.0.0/24
hosts:
  ...

Configure all the local networks there.

warning

This has to be done before the site goes disconnected!

management-ipv4-access-list documentation

Site-local modify operations

New application while being disconnected

When an operator has been locally authenticated at the disconnected site a new application may be added using supctl (or api) in the same way it would had been done at the Control Tower, but first the application image need to be transferred and loaded into docker (or podman) on one of the edge hosts in the site.

The best way of achieving this will vary depending on the environment, but below is an example of how this could be done, assuming you already have the image on your local laptop.

Transfer the image a to an edge host (HOST):

$LAPTOP: scp a HOST:

Login to the edge host:

$LAPTOP: ssh HOST:

NOTE: ssh access is required.

Load image a into docker:

$HOST: docker load --input a
Loaded image: a:latest

Tag the image:

$HOST: docker tag a:latest 127.0.0.1:4848/a:latest

Login and push image a to local repository:

$HOST: docker login 127.0.0.1:4848

$HOST: docker push 127.0.0.1:4848/a:latest

info

All the following supctl commands are performed locally at the site.

Verify the local image:

site$ ./supctl show image-registry repositories test/a digests
- digest: sha256: ...
  tags:
    - latest
  type: image-manifest
  platform: linux/amd64
  image-id: sha256: ...
  compressed-size: ...
  labels:
    version: "1"

Next step, deploy the application.

Create a local application:

site$ ./supctl create applications <<EOF
name: a
services:
  - name: my-srv
    containers:
      - name: a
        image: test/a:latest
        cmd: ["sleep", "infinity"]
    mode: replicated
    replicas: 1
network:
  shared-application-network: shared-network
EOF

Create a local application deployment

site$ ./supctl create application-deployments <<EOF
name: a
application: a
placement:
  match-site-labels: system/name = my-site
EOF

Verifying the local deployment with

site$ ./supctl show application-deployments a
name: a
application: a
application-version: "*"
placement:
  match-site-labels: |
    system/name = my-site
config-modified-time: 2023-10-24T13:38:31Z
status:
  oper-status: deployed
  images:
    - name: test/a:latest
      status: present
      digest: sha256:b1b4af6d3959255c686a
  selected-sites:
    - my-site
  application-versions:
    - version: undefined
      deployed-to:
        - site: my-site
          time: 2023-10-24T13:38:33Z
history:
  - time: 2023-10-24T13:38:31.499Z
    reason: application-deployment-created
    version: undefined
    result: deployed
    added-sites:
      - my-site

Modifying a centrally deployed application locally

Assume we have another application b that was deployed from the Control Tower when the site was connected. How can you modify that application?

As we mentioned in the introduction, applications that are scheduled from Control Tower only have a read-only state representation at the site. This is because the application is an effect of the local scheduler, they are not a replicated configuration as is the case for site configuration.

Therefore you would need to re-create the application locally. The steps to follow are exactly as above for creating a new application.

site$ ./supctl create applications <<EOF
name b:
...application YAML here, same as central YAML in Control Tower...
EOF

Also create a corresponding deployment

site$ ./supctl create application-deployments <<EOF
name: a
application: a
placement:
  match-site-labels: system/name = my-site
EOF

Now we have application configuration for the application available at the site and can modify it, for example by using the supctl edit command.

site$ ./supctl edit applications a

Modifying site configuration locally

Site configuration is replicated from Control Tower to the sites. This means that you have the configuraiton directly available for modifications. Reasonable edits are for example IP ingress configuration.

site$ ./supctl edit system sites my-site

Control Tower representation of locally modified application

As mentioned above, information about scheduled and deployed applications due to local modifications will be propagated to Control Tower when the site is connected again. This information will show up in the aggregated application summary states in application summary and application sites summary.

The below supctl show command illustrates a locally modified application:

./supctl show application-status summary
applications:
  - name: a
    application-deployment: a
    oper-status: running
    selected-sites: 1
    deployed-to-sites: 1
    deploy-failed-to-sites: 0
    locally-modified-sites: 1
    deploying: false
    all-sites-connected: true

./supctl show application-status applications a
name: a
sites:
  - name: my-site
    oper-status: running
    connected: true
    locally-modified:
      time: 2023-10-24T13:51:56.434Z
      application-version: undefined
      application-deployment: a

If you try to show the configuration of the site-locally created application at Control Tower a you will get

./supctl show -c applications a
not found

If you want to inspect all your sites to see any local modifications you can perform the following command:

supctl --table show system site-status sites --fields=name,locally-modified

name              locally-modified
---------------   ----------------
electric-cinema   false
metrograph        false
my-site           true

Site-local read-only operations

The above illustrated examples of modifying applications and application deployments. We will now move over to show some relevant read-only operations that might be useful when working locally at the site.

Check Host State

Check the status of the local site

site$ ./supctl show system cluster hosts

- cluster-hostname: sthlm-001
  version: dev-be812f57
  oper-status: up
  started-at: 2023-05-08T14:35:01Z
  maintenance-mode: off
  hostname: sthlm-1
...
- cluster-hostname: sthlm-002
  version: dev-be812f57
  oper-status: up
  started-at: 2023-05-08T14:35:01Z
  maintenance-mode: off
  hostname: sthlm-2
...
...

Check Application Status

List the running applications

site$ ./supctl list applications

- mongodb
- mq-bridge
- nginx
- perf-test
- popcorn-controller
- postgres
- redis
- visitor-counter

Show the state of an application

site$ ./supctl show applications visitor-counter

name: visitor-counter
version: "2.0"
services:
  - name: visitors
    mode: replicated
    replicas: 1
    volumes:
      - name: fast-storage
        ephemeral-volume:
          size: 5MB
          file-mode: "755"
          file-ownership: 0:0
...

And specifically show the service instances

site$ ./supctl show applications visitor-counter service-instances

- name: visitors-1
  application-version: "2.0"
  oper-status: running
  ready: true
  host: sthlm-3
  application-network:
    ips:
      - 172.19.0.1/16
    dns-records:
      - visitors-1.visitor-counter.internal. 15 IN A 172.19.0.1
      - visitors.visitor-counter.internal. 15 IN A 172.19.0.1
  gateway-network:
    ips:
      - 172.18.253.2/24
  ingress:
    interface: enp2s0
    ips:
      - 192.168.3.74
    dns-records:
      - visitors-1.visitor-counter.example-corp.sthlm.avassa.net. 15 IN A 192.168.3.74
      - visitors.visitor-counter.example-corp.sthlm.avassa.net. 15 IN A 192.168.3.74
  ephemeral-volumes:
    - name: fast-storage
      size: 5MB
      host-volume: volume1
  persistent-volumes: []
  containers:
    - name: visitors
      id: 8bf04f8b5f90
      oper-status: running
      ready: true
      start-time: 2023-05-08T11:40:13.386Z
      current-restarts: 0
      total-restarts: 0
      probes:
        startup:
          status: success
        readiness:
          status: success
        liveness:
          status: success

Restart an Application Service Instance

site$ ./supctl do applications visitor-counter service-instances visitors-1 restart

Consume Volga Topics

To list the available volga topics

site$ ./supctl list volga topics

- mqtt-bridge
- proxy-up
- system:alerts
- system:all-scheduler-events
- system:application-metrics
- system:audit-trail-log
- system:container-logs:mongodb.db-1.db
- system:container-logs:mongodb.express-1.express
- system:container-logs:mq-bridge.mq-1.mq-bridge
...

And to consume a topic

site$ ./supctl do volga topics system:container-logs:visitor-counter.visitors-1.visitors consume --payload-only