Skip to main content

Site synchronization and recovery

Introduction​

Avassa manages configuration and secrets centrally in the Control Tower and distributes them out to the sites that need them. Application definitions, deployments, policies, users and Strongbox secrets all follow the same pattern: an operator changes something in the Control Tower, and the change is delivered to the affected sites.

This page explains how that synchronization works, what happens while a site is disconnected, and how the platform ensures a site never silently serves stale data after a disruption. It complements the Strongbox and Volga fundamentals, and pairs with the certificate side of long outages described in Configure certificates for extended offline periods.

One-directional distribution​

Distribution flows in one direction only: from the Control Tower downward through the site tree. A site never propagates its configuration or secrets upward. This is a deliberate security property. An intrusion or misconfiguration at one site cannot poison the management point or other sites.

Distribution is also explicit. A secret or configuration object is only delivered to a site when the tenant is assigned to that site and the object's distribution settings select it. Nothing is shared by default.

How a site stays in sync​

Synchronization has two parts: an initial full state and a stream of incremental updates.

  • Initial state. When a site first connects (or reconnects after being wiped), it pulls a complete snapshot of the configuration and secrets it is entitled to, directly from its parent. Once the snapshot is applied, the site is initialized.

  • Incremental updates. After initialization, each change made in the Control Tower is delivered as a small, encrypted diff over the internal event stream that connects the site to its parent. The site applies each diff in order and remembers its position in the stream, so it always knows where to resume from.

Sending only minimal diffs keeps bandwidth low, which matters when a single Control Tower drives hundreds or thousands of sites over constrained links.

Disconnection and catch-up​

Sites are designed to operate autonomously while disconnected. A disconnected site keeps running its applications and serving the secrets it already holds; it simply stops receiving new updates.

When the site reconnects, it resumes the update stream from the position it last recorded and applies everything it missed while it was away. For a normal outage this catch-up is seamless and the operator sees nothing but the connection state flipping back to connected.

The update stream is a bounded buffer, not an infinite log. To keep Control Tower storage bounded, the oldest updates are eventually discarded once the buffer is full. Under normal operation a site consumes updates long before they age out, so this is invisible.

Recovering a site that has fallen too far behind​

A very long outage introduces an edge case worth understanding. If a site is disconnected long enough that the Control Tower's update buffer wraps past the site's resume position, the specific updates that site still needed are discarded before it can consume them. They cannot be replayed. Left unhandled, the site would reconnect, find its exact resume point gone, and quietly continue with stale configuration and secrets. Silent divergence like this is exactly what a secrets and configuration system must never allow.

Avassa detects and repairs this automatically, from both ends:

  • The site detects the gap. When the site reconnects and finds that the earliest update still available is beyond the point it needed to resume from, it concludes that it has missed unrecoverable updates. Rather than continue from a stale base, it discards its now-untrusted local copy of the distributed state and pulls a complete fresh snapshot from its parent, over a path that does not depend on the wrapped stream. The snapshot replaces the stale state atomically, and the site returns to normal incremental updates from a known-good point.

  • The Control Tower flags the site. If the Control Tower is ever unable to deliver an update to a site, it marks that site as needing a full resynchronization, so recovery does not depend solely on the site noticing.

Either path converges on the same result: the site ends up holding the current state, or is actively resynchronizing toward it. A site is never left silently out of date.

What an operator observes​

Recovery is automatic; there is no command to run. During a full resynchronization the site briefly re-fetches and re-applies its state, then resumes normal operation. From the Control Tower the site's connection state returns to connected and its configuration and secrets converge to the current values. The how-to on verifying site synchronization shows how to confirm this.

Relationship to certificate recovery​

Distribution recovery and certificate recovery are independent concerns that can both come into play after a long outage:

  • Certificates govern whether a host may reconnect and prove its identity at all. If a host's certificates expire while it is offline, it must be reauthorized before it can rejoin. See Configure certificates for extended offline periods.

  • Distribution governs whether a reconnected site holds current configuration and secrets. This is handled by the automatic resynchronization described above.

A site returning from a very long outage may need both: reauthorization to reconnect, and a resynchronization to catch up on state. Both are built to happen with minimal operator intervention.