Skip to main content

How to Set Up Time Synchronization for Avassa Sites

This guide explains how to configure automatic time synchronization for edge sites in Avassa. It covers both centralized time distribution from Control Tower and local Chrony configuration for reliable time synchronization.

Prerequisites​

  • Avassa Control Tower and edge sites deployed
  • Root access to edge hosts for Chrony configuration
  • Basic knowledge of NTP/Chrony time synchronization

Overview​

Avassa's time synchronization system works by:

  1. Monitor local time synchronization: Edge Enforcer checks every 30 minutes that the Chrony daemon is synchronized
  2. Fallback to Control Tower: If local time synchronization fails, correct time is fetched from Control Tower
  3. Inject time via socket: Control Tower time is injected as an external reference clock into the Chrony daemon

Key Features:

  • Automatic monitoring: Continuous monitoring of local time synchronization every 30 minutes
  • Intelligent fallback: Uses Control Tower as backup when local timekeepers are unavailable
  • Transparent operation: Works through existing Chrony mechanisms without disrupting normal operation
  • Centralized control: Enabled/disabled per site from Control Tower

Step 1: Enable Time Synchronization for Site​

Enable the time synchronization feature for your edge site:

# Enable sync-time for edge site
supctl merge system sites mysite <<EOF
sync-time: true
EOF

This instructs the Edge Enforcer to start monitoring and synchronizing time for all hosts in the site when needed.

Step 2: Install and Configure Chrony on Edge Hosts​

Install Chrony​

On most modern Linux distributions, Chrony is already installed. If not:

# For Debian/Ubuntu
sudo apt-get update
sudo apt-get install chrony

# For RHEL/CentOS/Rocky
sudo yum install chrony
# or
sudo dnf install chrony

Configure Chrony for External Reference Clock​

Add the following configuration to /etc/chrony/chrony.conf:

# Add these lines to /etc/chrony/chrony.conf
sudo cat >> /etc/chrony/chrony.conf <<EOF
# External time samples from your app (TSQ/RPC, etc.)
refclock SOCK /run/chrony/tsq.sock refid TSQ
# (optional) allow stepping at boot if the offset is large
makestep 1.0 3
EOF

Enable Refclock Logging (Optional - For Verification and Debugging Only)​

Note: Logging configuration is not required for the time synchronization feature to work. It is only useful for manual verification and debugging purposes. The exact log paths and configuration are up to your preference.

To monitor time synchronization activity (optional), add logging configuration:

sudo cat >> /etc/chrony/chrony.conf <<EOF
# Where to put log files (chronyd needs write access)
logdir /var/log/chrony
# What to log – choose what you need:
log measurements statistics tracking refclocks
EOF

Restart and Configure Chrony​

# Restart the Chrony daemon
sudo systemctl restart chronyd

# Ensure the socket file is accessible (may be needed after restart)
sudo chmod a+rwx /run/chrony/tsq.sock

Important: After adding the refclock socket configuration to Chrony, the Edge Enforcer must be restarted to be able to access the newly created Chrony socket.

# Restart the Edge Enforcer service
sudo systemctl restart supd

Step 3: Verify Configuration​

Check Chrony Status​

# Show current time sources
chronyc sources -v

# Show tracking status
chronyc tracking

# Show activity for all time sources
chronyc activity

Monitor Refclock Activity (Optional)​

Note: This section only applies if you chose to enable refclock logging in the previous optional step.

If you enabled refclock logging, you can verify that TSQ updates are being received:

# View the refclock log (only if you enabled logging)
sudo tail -f /var/log/chrony/refclocks.log

You should see entries like:

==============================================================================
   Date (UTC) Time         Refid  DP L P  Raw offset   Cooked offset      Disp.
===============================================================================
2025-09-25 08:27:39.212137 TSQ     0 N 0  2.790000e-01  2.790004e-01  2.200e-08
2025-09-25 08:27:41.919357 TSQ     1 N 0  2.789542e-01  2.787948e-01  2.200e-08
2025-09-25 08:27:42.987170 TSQ     2 N 0  2.092922e-01  2.092923e-01  2.200e-08

Testing and Troubleshooting​

Force Time Synchronization (for Testing)​

You can test the functionality by simulating time synchronization problems:

# Disconnect Chrony from external time sources
sudo chronyc offline

# Verify that Avassa detects the problem
# Check Edge Enforcer logs for messages like:
# "Syncing time with topdc (diff X ns)"
# Use: docker logs supd | grep sync
# 
# IMPORTANT: This may take up to 30 minutes to detect since 
# the Edge Enforcer checks time synchronization every 30 minutes

Restore Normal Operation​

# Re-enable external time sources
sudo chronyc online

# Enable all reference clocks
sudo chronyc enable refclock

Monitor Edge Enforcer Logs​

Check Edge Enforcer logs for time synchronization activity:

# Check for time synchronization messages using docker logs (run locally on the edge host)
docker logs supd | grep -i "time\|sync"

You should see messages like:

  • "Time in sync, no need to sync clocks" when everything is working normally
  • "Syncing time with topdc (diff X ns)" when synchronization is needed

Common Issues and Solutions​

Socket Permissions Issues​

Issue: Edge Enforcer cannot write to Chrony socket

Symptoms: Look for these error log entries from the Edge Enforcer:

  • "No Chrony socket present, will not sync time"
  • "Failed to open or write Chrony socket"

Check the Edge Enforcer logs:

# Docker logs for supd container (run locally on the edge host)
docker logs supd

Solution:

# Verify that socket file exists and has correct permissions
ls -la /run/chrony/tsq.sock

# Set correct permissions
sudo chmod a+rwx /run/chrony/tsq.sock

# Alternatively, add supd user to chrony group
sudo usermod -a -G chrony supd

Chrony Does Not Accept External Reference Clock​

Issue: SOCK refclock configuration is incorrect

Symptoms:

  • No /run/chrony/tsq.sock socket file created
  • No refclocks.log file created (if logging was enabled)

Solution:

# Verify that configuration is correct
sudo grep -A 5 -B 5 "refclock SOCK" /etc/chrony/chrony.conf

# Check Chrony logs for errors
sudo journalctl -u chronyd | grep -i error

Edge Enforcer Does Not Detect Time Synchronization Problems​

Issue: Chrony reports it is synchronized but actually is not

Solution:

# Check detailed Chrony status
chronyc tracking
chronyc sources -v

# Adjust maxupdateskew if necessary (allows larger time deviations)
sudo sed -i 's/^\s*maxupdateskew 100\.0/#&/' /etc/chrony/chrony.conf
sudo systemctl restart chronyd

Summary​

Avassa's time synchronization system provides robust and automatic time management for edge sites by:

  • Automatically monitoring local time synchronization every 30 minutes
  • Providing backup via Control Tower when local time sources fail
  • Integrating seamlessly with existing Chrony installations
  • Ensuring reliability through transparent fallback mechanisms

Important: The feature is enabled per site with the sync-time: true setting and requires Chrony to be correctly configured to accept external reference clock via the SOCK interface.

With proper configuration, the system ensures that all edge sites have correct time even in challenging network environments where traditional NTP servers may not always be available.