Skip to main content

Verify and recover site synchronization

Avassa distributes configuration and secrets from the Control Tower to your sites automatically, and automatically resynchronizes a site that has fallen behind after a long disconnection. In normal operation there is nothing to do. This guide shows how to confirm a site is in sync, what the automatic recovery does, and how to troubleshoot a site that looks out of date.

For the concepts behind distribution and recovery, see Site synchronization and recovery. For the certificate side of long outages, see Configure certificates for extended offline periods.

Check that a site is connected​

Start from the Control Tower. A site can only receive updates while it is connected:

supctl show system site-status sites

To look at one site and just its connection state:

supctl show system site-status sites udc1 \
--fields connection-state/connected
connection-state:
connected: true

You can also confirm the site's hosts have all called home and formed a cluster:

supctl show system sites udc1 --fields call-home-state
call-home-state:
all-hosts:
called-home: 3
total: 3
controller-hosts:
called-home: 3
total: 3
cluster-established: true

Confirm configuration has converged​

The Control Tower can show you a site's actual state, as reported by the site, using -s <site>. Comparing that against the definition in the Control Tower tells you whether the site has caught up.

Compare the Control Tower's view of an application version with the version the site actually runs:

# What the Control Tower has defined
supctl show applications myapp --fields version

# What the site udc1 actually reports
supctl show -s udc1 applications myapp --fields version

When the two agree, the site has applied that configuration. The same -s <site> pattern works for other resources, so you can spot-check whatever matters for your deployment.

To see whether a site has been changed locally while disconnected (which is expected and reconciled on reconnect, see Managing disconnected sites):

supctl --table show system site-status sites --fields=name,locally-modified

What automatic recovery does​

If a site is disconnected long enough that the Control Tower's update buffer wraps past the point the site needed to resume from, the updates that site still needed are discarded and cannot be replayed. Avassa handles this without operator action:

  • On reconnect, the site notices that its resume point is gone, discards its now-untrusted local copy of the distributed state, and pulls a complete fresh snapshot from its parent. The snapshot replaces the stale state atomically, and the site resumes normal incremental updates.
  • Independently, if the Control Tower could not deliver an update to a site, it flags that site for a full resynchronization.

You do not trigger this, and there is no command to force it. To watch it happen, poll the convergence check above: after a long-offline site reconnects, its reported versions and secrets move to the current values on their own.

Troubleshooting a site that looks out of date​

Work through these in order:

  1. Is the site connected? Use the connection check above. A site that is not connected cannot receive updates. If it will not connect, the cause is usually networking or expired certificates, not distribution, see Configure certificates for extended offline periods for the certificate recovery path (reauthorize).

  2. Has it finished coming up? Confirm all hosts have called home and the cluster is established (call-home-state above). A site that is still forming its cluster has not started applying state yet.

  3. Give automatic resynchronization time. A full resynchronization re-fetches and re-applies the site's whole distributed state. On a large tenant over a constrained link this takes a little time. Re-run the convergence check; the versions should be moving toward the Control Tower values.

  4. Check for a local modification. If locally-modified is true, a site-local operator changed configuration while disconnected. See Managing disconnected sites for how local changes are reconciled.

If a site remains out of date after it is connected, its cluster is established, and enough time has passed for a resynchronization to complete, capture the state with the commands above and open a support case.