Skip to main content

Control Tower proxy

Background​

In some scenarios there are hosts that need to be managed, which are connected to restricted networks that do not have direct connectivity to the Control Tower. The solution in such case is to connect an Edge Enforcer running on a host connected to a restricted network to the Control Tower via another Edge Enforcer running on a host on the network perimeter, namely one that is reachable from the restricted host and has connectivity to the Control Tower. The proxy host may be a part of the same edge site or reside in a different edge site.

Such proxy functionality is disabled by default and needs to be enabled in the configuration. The proxy is not a general-purpose proxy, but only able to forward incoming connections to the same Control Tower it is connected to on selected protocols and ports.

Refer to the the Infrastructure networking document for the description of what kind of connections are proxied.

Proxy configuration​

There are two cases that should be considered when configuring a proxy:

  • when an Edge Enforcer should proxy connections only for Edge Enforcers in other edge sites, or worker Edge Enforcers in the same site as the proxy. This is a simpler case as only the general proxy configuration can be applied.
  • when an Edge Enforcer should proxy connections for some of the controllers in the same edge site as the proxy. In this case the proxy needs to be enabled at installation time in addition to the configuration in the site specification.

The difference between the two cases stems from the fact that all controllers must connect to the Control Tower for the site initialization to complete. This means that the proxy needs to be enabled at install time.

Proxy configuration: general case​

Unless there are other controller hosts within the same edge site that need to use an Edge Enforcer as proxy for connecting to the Control Tower, the proxy host only needs to be declared in the site configuration:

supctl merge system sites my-proxy-site <<EOF
hosts:
  - host-id: 6e4d8db8-0833-4b5f-960d-a358d96aa1f2
    proxy:
      act-proxy: true
      network-access:
        rules:
          10.0.0.0/8: allow
EOF

See the reference documentation for the proxy-related parameters.

Verify that the proxy is running by inspecting:

supctl show --site my-proxy-site system cluster hosts my-proxy-site-001
cluster-hostname: my-proxy-site-001
...
parent-site-connection:
  act-proxy:
    api-port: 5656
    registry-port: 5858
    volga-port: 5959

Special case: proxy configuration at install time​

When an Edge Enforcer running on a host should proxy connections for other controller hosts within the same edge site, the proxy needs to be enabled at install time. Because for site initialization to complete all controller hosts must be online, the proxy must be active even during the initialization phase before it has a chance to fetch the site configuration from the Control Tower. Once the site is initialized this initial configuration will be overridden by the site configuration.

To enable the proxy at install time, give --act-parent-proxy flag to the install script (in addition to any other desireg flags) when installing the host that should proxy connections for other sites:

curl -s https://api.test.acme.avassa.net/install | sudo sh -s -- --act-parent-proxy

Even though enabled at install time, the proxy must also be configured in the site configuration as described in the Proxy configuration: general case section.

Configuring hosts that require proxy connection​

In contrast with the configuration of the proxy host, the configuration for the host requiring the proxy connection is not stored in the configuration database, but only locally on the host. This is due to the asynchronious nature of the changes applied to the configuration database and difficulty to ensure that the configuration has been applied in time before switching the proxy hosts, as well as complexity of recovering a disconnected host in case of misconfiguration.

A proxy is specified using either a single domain name (that may resolve to a single or multiple IP addresses), or single or multiple IP addresses. The required parameters are set at install time and can be updated using the action update-use-proxy-settings at site or host-level at a later time, as explained later.

Installing Edge Enforcer via proxy​

In order to install an edge enforcer via proxy, a specially crafted curl command can be used to fetch the install script from the Control Tower via the proxy. The --connect-to flag to curl command indicates that curl should substitute a certain pair of host and port indicated in the URL with another host and port when initiating a connection. In the example below, --connect-to api.test.acme.avassa.net::proxy.acme.net:5656 means that curl will initiate a connection to proxy.acme.net:5656 whenever it needs to fetch a URL containing api.test.acme.avassa.net with any port, yet still use api.test.acme.avassa.net for the purposes of TLS and HTTP protocols (for example when sending an SNI as a part of TLS Client Hello and to verify the server certificate). For example, the following commands can be used in order to install an Edge Enforcer via proxy:

  • to configure proxy as a domain name

    curl -s --connect-to api.test.acme.avassa.net::proxy.acme.net:5656 \
        https://api.test.acme.avassa.net/install | sudo sh -s -- --proxy proxy.acme.net

  • to configure proxy as multiple IP addresses:

    curl -s --connect-to api.test.acme.avassa.net::192.0.2.4:5656 \
        https://api.test.acme.avassa.net/install | sudo sh -s -- --proxy 192.0.2.4 --proxy 192.0.2.5

An alternative to the specially crafted curl command is to distribute the installation script via a side channel, for example:

On the proxy host 192.0.2.4, save the install script to a file:

curl -o /tmp/install https://api.test.acme.avassa.net/install

On the host to be installed, first copy the install script from the proxy host, and then run it with the required flags:

scp 192.0.2.4:/tmp/install .
sudo sh install --proxy 192.0.2.4 --proxy 192.0.2.5

Note on the Edge Enforcer container image​

Currently the install script is unable to fetch the Edge Enforcer container image via proxy due to inability of the Docker client to validate the server identity via TLS when the connection is proxied. The workaround for this problem is to distribute the Edge Enforcer image out of band. To do so:

  • first, export the avassa/supd:current image from the proxy host (or any other host reachable from the target host running the current version of Edge Enforcer: 
    docker save -o supd.tar avassa/supd:current
  • copy the file to the target host using any method available, for example: 
    scp 192.0.2.4:supd.tar .
  • on the target host load the image from file: 
    docker load -i supd.tar

Updating proxy connection settings​

note

Updating proxy connection settings is inherently risky in the same way as updating firewall configuration is risky. A misconfiguration may result in a loss of communication link so that the connection must be restored by accessing the target host locally or via some other channel. Double-check and whenever possible test the settings that are to be applied to the host.

If a proxy needs to be migrated to another host, or if the domain name of the proxy needs to be changed, or the proxy is specified in terms of static IP addresses than need to be updated, then the proxy connection settings need to be updated on each host using this proxy connection.

note

The proxy connection settings are only stored locally on each host, so the system does not have an overview of the hosts that are connected via proxy. It is currently the responsibility of the administrator to keep track of the hosts requiring proxy connection to the Control Tower.

To update proxy connection settings on a single host:

supctl --verbose do --site my-isolated-site system cluster hosts my-isolated-site-001 \
    update-use-proxy-settings --parent-proxy-name proxy2.acme.net

Or, to specify a list of IP addresses

supctl --verbose do --site my-isolated-site system cluster hosts my-isolated-site-001 \
    update-use-proxy-settings --input - <<EOF
parent-proxy-ips:
  - 192.0.2.8
  - 192.0.2.9
EOF

Verify that the command does not print out any error. Also, note that the supctl command is executed with the -v flag in this case - this is to be able to verify the returned HTTP code. The expected HTTP code is 204 No Content, which means the settings have been successfully applied. If there is an error or a different HTTP code, then the settings may have not been applied.

To update proxy connections for all hosts within a single site:

supctl do --site my-isolated-site system cluster \
    update-use-proxy-settings --parent-proxy-name proxy2.acme.net

This action returns a list of hosts within the site along with the result for each host. The ok result means that the settings have been applied successfully, any other result means the settings may have not been applied.

note

This action only stores the proxy connection preferences for future connections, but any currently active connections are not affected. This means that even if the new connection preferences are successfully stored, it does not mean they are correct or that the host will be able to connect with the new connection settings. If the current proxy goes away, then the connection will be interrupted and the host will attempt using the new settings. The easiest way to force new connection attempt is to restart the Edge Enforcer.

Verifying the connection settings​

To verify currently stored connection settings, run the following command:

supctl show --site my-isolated-site system cluster hosts \
    --fields cluster-hostname,hostname,parent-site-connection
cluster-hostname: my-isolated-site-001
hostname: my-host
parent-site-connection:
  connect-via-proxy:
    parent-proxy-name: proxy.acme.net
    api-port: 5656
    registry-port: 5858
    volga-port: 5959

Inspecting the currently active connections​

To see currently active connections, run the following command:

supctl do --site my-isolated-site diagnostics get-connections
up-connections:
  hosts:
    - host: my-isolated-site-001
      status: connected
      protocol: quic
      connect-time: 2025-01-01T00:00:00Z
      peer: 192.0.2.4:5959

Misconfigured site recovery​

If a single host within a site has its proxy connection settings misconfigured, then it may be possible to update the settings normally as explained in Updating proxy connection settings . This is because other hosts within the site may still be connected to the Control Tower, while the misconfigured host maintains a local connection to the other hosts.

A misconfigured site, however, often means that all hosts within the site have lost connection to the Control Tower. Such site must be accessed and recovered locally, with the help of supctl (or corresponding API calls executed with curl, if supctl is not available).

When the site is disconnected it may be sealed and require a manual remote unseal procedure. To perform it, follow the How to Unseal an Isolated Site guide.

Once the site is unsealed, one may authenticate with supctl using a locally available user (stored in Strongbox, as OIDC typically requires a connection outside of the isolated network), from one of the hosts on the isolated site:

supctl --host 127.0.0.1 --port 4646 do login my-user@acme.com

Once authenticated, the update-use-proxy-settings command may be run as explained using the locally authenticated supctl in the Updating proxy connection settings section. The --site my-isolated-site flag does not have to be specified (although it may), for example:

supctl do system cluster update-use-proxy-settings --parent-proxy-name proxy2.acme.net

If supctl is not available locally, the Accessing the Avassa REST API using Postman will help understanding how to access the API using any available HTTP client, for example curl.

Layered proxy​

An Edge Enforcer may simultaneously act as a proxy for other Edge Enforcers and in turn use a proxy connection to connect to the Control Tower. This is to support multiple layers of isolation in the networks. The layered proxy is configured by combining the configuration in the Proxy configuration and Configuring hosts that require proxy connection sections.