Create a highly available Redis cluster

tip

In order to be able to run through the steps of these tutorials in an operational Avassa environment, sign up for a free trial to get access to a running system.

The application

In this tutorial we will walk through an application specification for a Redis cluster, using the standard Redis docker image from Docker Hub.

We will deploy a Redis cluster with three instances, and one of them will serve as the primary and the other two as instances.

The full example including vaults can be found in the application examples repository.

The application consists of one image (redis) but we will run two containers in each instance:

redis-server is what it says, the actual Redis server.
redis-sentinel provides high availability for Redis and is responsible for monitoring the instance and, if needed, failing over to another instance.

The Redis instances require a slightly different configuration file from the Redis leader, which complicates things a tad bit. The configuration files are modified by Redis itself after start, hence the configuration files have to reside on a volume that is read-write to the application.

We also want to configure a liveness probe to monitor the liveness/health of the cluster.

The application specification looks like this, and we will walk through each part in this tutorial.

name: redis
version: "1.0"
services:
  - name: redis
    init-containers:
      - name: setup
        image: registry-1.docker.io/redis
        cmd: ["sh", "/cfg0/setup.sh"]
        mounts:
          - volume-name: cfg-storage
            mount-path: /cfg
          - volume-name: data
            mount-path: /data
          - volume-name: cfg0
            files:
              - name: setup.sh
                mount-path: /cfg0/setup.sh
              - name: redis.conf
                mount-path: /cfg0/redis.conf
              - name: sentinel.conf
                mount-path: /cfg0/sentinel.conf
    containers:
      - name: redis
        image: registry-1.docker.io/redis
        cmd: ["redis-server", "/cfg/redis.conf"]
        env:
          REDIS_PASS: ${REDIS_PASS}
        probes:
          readiness:
            exec:
              cmd:
                - sh
                - "-c"
                - "[ ${SYS_APP_NET_IPV4_ADDRESS} = $(redis-cli --tls --cacert /certs/ca-cert.pem -a ${REDIS_PASS} -p 26379 sentinel master mymaster | sed '4q;d') ] || exit 10"
            initial-delay: 0s
            timeout: 1s
            period: 10s
            success-threshold: 1
            failure-threshold: 3
          liveness:
            exec:
              cmd:
                - sh
                - "-c"
                - redis-cli --tls --cacert /certs/ca-cert.pem ping
        mounts:
          - volume-name: cfg-storage
            mount-path: /cfg
          - volume-name: data
            mount-path: /data
          - volume-name: certs
            mount-path: /certs
      - name: redis-sentinel
        image: registry-1.docker.io/redis
        cmd: ["redis-sentinel", "/cfg/sentinel.conf"]
        env:
          REDIS_PASS: ${REDIS_PASS}
        probes:
          liveness:
            exec:
              cmd:
                - sh
                - "-c"
                - redis-cli --tls --cacert /certs/ca-cert.pem -p 26379 ping
            initial-delay: 0s
            timeout: 1s
            period: 10s
            success-threshold: 1
            failure-threshold: 3
        mounts:
          - volume-name: cfg-storage
            mount-path: /cfg
          - volume-name: data
            mount-path: /data
          - volume-name: certs
            mount-path: /certs
    network:
      ingress-ip-per-instance:
        protocols:
          - name: tcp
            port-ranges: "6379"
    volumes:
      - name: cfg-storage
        ephemeral-volume:
          size: 3MiB
          file-ownership: 999:999
      - name: data
        ephemeral-volume:
          # this is the size of redis' data storage volume
          # it should be much larger in production
          size: 5MiB
          file-ownership: 999:999
      - name: certs
        vault-secret:
          vault: redis
          secret: cert
          file-ownership: 999:999
      - name: cfg0
        config-map:
          items:
            - name: setup.sh
              data: |
                #!/bin/sh
                cp /cfg0/redis.conf /cfg/redis.conf
                chown redis:redis /cfg/redis.conf
                cp /cfg0/sentinel.conf /cfg/sentinel.conf
                chown redis:redis /cfg/sentinel.conf
                if [ ${SYS_SERVICE_INSTANCE_INDEX} -gt 1 ]; then
                  echo "replicaof redis-1 6379" >> /cfg/redis.conf
                fi
            - name: redis.conf
              data: |
                protected-mode no
                port 6380
                requirepass ${REDIS_PASS}
                masterauth ${REDIS_PASS}

                # Enable TLS
                tls-port 6379
                tls-ca-cert-file /certs/ca-cert.pem
                tls-cert-file /certs/cert.pem 
                tls-key-file /certs/cert.key
                tls-auth-clients optional
                tls-replication yes
            - name: sentinel.conf
              data: |
                sentinel resolve-hostnames yes
                sentinel monitor mymaster redis-1 6379 2
                sentinel down-after-milliseconds mymaster 60000
                sentinel failover-timeout mymaster 180000
                sentinel parallel-syncs mymaster 1

                sentinel announce-ip ${SYS_APP_NET_IPV4_ADDRESS}
                port 0
                tls-port 26379
                tls-ca-cert-file /certs/ca-cert.pem
                tls-cert-file /certs/cert.pem
                tls-key-file /certs/cert.key
                tls-auth-clients optional
                tls-replication yes
                requirepass ${REDIS_PASS}

    variables:
      - name: REDIS_PASS
        value-from-vault-secret:
          vault: redis
          secret: credentials
          key: password
    mode: replicated
    replicas: 3
    placement:
      preferred-anti-affinity:
        services: [ redis ]
upgrade-from:
  - version-pattern: "*"
    method: per-service
    services:
      - name: redis
        instances-in-parallel: 1
        healthy-time: 1m

High Availability

This setup achieves high availability by running three instances on three different hosts. In case a host fails, a new Redis primary is automatically selected by the Redis Sentinel.

Clients connect to the single instance, the primary, at all times. This is achieved using DNS controlled by a readiness probe.

Replicas and affinity

Since we want three instances of the redis and sentinel containers, we specify replicas: 3.

If we have enough hosts in a site, we also want to run each instance on its own host. This is achieved by specifying affinity rules. In this case we simply say that the service (redis) should have anti affinity to itself. The system will then do its best to schedule each instance on different hosts.

    mode: replicated
    replicas: 3
    placement:
      preferred-anti-affinity:
        services: [ redis ]

note

When there are more than one instance, a host failure will not trigger a migration of service instances to a new host.

Rolling upgrades

When upgrading to a new version, either a new redis image or a new configuration, we want to take one instance at a time.

upgrade-from:
  - version-pattern: "*"
    method: per-service
    services:
      - name: redis
        instances-in-parallel: 1
        healthy-time: 1m

The snippet above instructs the Edge Enforcer to take on instance at a time and wait for one minute before continuing to the next instance.

For more information see here.

Readiness Probe - Ingress and DNS

The Redis container has a readiness probe configured. The probe connects to the sentinel container and compares the IP address of the primary to the service instance's IP address (SYS_APP_NET_IPV4_ADDRESS). If this service instance is NOT the primary, the probe returns 10 which indicates it is inactive.

    containers:
      - name: redis
        probes:
          readiness:
            exec:
              cmd:
                - sh
                - "-c"
                - "[ ${SYS_APP_NET_IPV4_ADDRESS} = $(redis-cli --tls --cacert /certs/ca-cert.pem -a ${REDIS_PASS} -p 26379 sentinel master mymaster | sed '4q;d') ] || exit 10"
            initial-delay: 0s
            timeout: 1s
            period: 10s
            success-threshold: 1
            failure-threshold: 3

An inactive service instance's IP address is removed from DNS and only the primary instance is present. This allows clients to resolve only the primary instance.

supctl show --site robot-cluster dns
zones:
  - name: default
    domain: the-company.robot-cluster.sl-test.the-company.avassa.net
    records:
      - rr: redis-1.redis 15 IN A 172.27.70.135
      - rr: redis.redis 15 IN A 172.27.70.135

If the readiness probe was not configured, all three instances would be available in DNS.

Additional information:

Volumes

The Redis application utilizes a number of volumes; configuration, data and secrets.

name: redis
services:
  - name: redis
    containers:
      - name: redis
        mounts:
          - volume-name: cfg-storage
            mount-path: /cfg
          - volume-name: data
            mount-path: /data
          - volume-name: certs
            mount-path: /certs
      - name: redis-sentinel
        mounts:
          - volume-name: cfg-storage
            mount-path: /cfg
          - volume-name: data
            mount-path: /data
          - volume-name: certs
            mount-path: /certs
    volumes:
      - name: cfg-storage
        ephemeral-volume:
          size: 3MiB
          file-ownership: 999:999
      - name: data
        ephemeral-volume:
          # this is the size of redis' data storage volume
          # it should be much larger in production
          size: 5MiB
          file-ownership: 999:999
      - name: certs
        vault-secret:
          vault: redis
          secret: cert
          file-ownership: 999:999
      - name: cfg0
        config-map:
          items:
            - name: setup.sh
              data: |
                #!/bin/sh
                cp /cfg0/redis.conf /cfg/redis.conf
                chown redis:redis /cfg/redis.conf
                cp /cfg0/sentinel.conf /cfg/sentinel.conf
                chown redis:redis /cfg/sentinel.conf
                if [ ${SYS_SERVICE_INSTANCE_INDEX} -gt 1 ]; then
                  echo "replicaof redis-1 6379" >> /cfg/redis.conf
                fi
            - name: redis.conf
              data: |
                protected-mode no
                port 6380
                requirepass ${REDIS_PASS}
                masterauth ${REDIS_PASS}

                # Enable TLS
                tls-port 6379
                tls-ca-cert-file /certs/ca-cert.pem
                tls-cert-file /certs/cert.pem 
                tls-key-file /certs/cert.key
                tls-auth-clients optional
                tls-replication yes
            - name: sentinel.conf
              data: |
                sentinel resolve-hostnames yes
                sentinel monitor mymaster redis-1 6379 2
                sentinel down-after-milliseconds mymaster 60000
                sentinel failover-timeout mymaster 180000
                sentinel parallel-syncs mymaster 1

                sentinel announce-ip ${SYS_APP_NET_IPV4_ADDRESS}
                port 0
                tls-port 26379
                tls-ca-cert-file /certs/ca-cert.pem
                tls-cert-file /certs/cert.pem
                tls-key-file /certs/cert.key
                tls-auth-clients optional
                tls-replication yes
                requirepass ${REDIS_PASS}

See Application storage for more information.

Configuration

Since Redis needs to modify the configuration, and it needs to be persistent across restarts of Redis, we give each instance its own read-write volume, each with a size of 3 MB. This volume is mounted into each instance at /cfg.

When we start the redis processes, we pass them the name of their configuration files on the command line.

Data

Redis requires a writable volume at /data. In this case the data volume is 5MiB large, for production this should probably be extended.

note

Data volumes are not automatically migrated/shared between different hosts.

Secrets

The Redis instances each use a certificate generated by the Edge Enforcer. The certificates (CA. certificate and private key) are mounted in /certs.

Also note that a password is read from a vault and placed in a variable (that is substituted into the configuration).

    variables:
      - name: REDIS_PASS
        value-from-vault-secret:
          vault: redis
          secret: credentials
          key: password

Configuration files

When the application is started the very first time on a host, the volume that is mounted to /cfg is empty. We need to create the required config files before the Redis processes are started.

We do this by specifying an init container. The init container is run to completion before the normal containers are started.

In our case, the init container uses the same image as the other container, and we run a short script (setup.sh) that creates the required configuration files.

The script itself and the configuration files for redis and redis-sentinel are specified in a config map. The setup.sh script copies the config files from the read-only config map to the read-write volume cfg-storage.

The SYS_SERVICE_INSTANCE_INDEX is an environment variable, in this case taking the values 1, 2 or 3 (since we have three instances). If the instance index is greater than one, we modify the /cfg/redis.conf file to indicate that this instance is redis-1, which is the DNS name of the first service-instance.

      - name: cfg0
        config-map:
          items:
            - name: setup.sh
              data: |
                #!/bin/sh
                cp /cfg0/redis.conf /cfg/redis.conf
                chown redis:redis /cfg/redis.conf
                cp /cfg0/sentinel.conf /cfg/sentinel.conf
                if [ ${SYS_SERVICE_INSTANCE_INDEX} -gt 1 ]; then
                  echo "replicaof redis-1 6379" >> /cfg/redis.conf
                fi
            - name: redis.conf
              data: |
                protected-mode no
                port 6380
                requirepass ${REDIS_PASS}
                masterauth ${REDIS_PASS}

                # Enable TLS
                tls-port 6379
                tls-ca-cert-file /certs/ca-cert.pem
                tls-cert-file /certs/cert.pem 
                tls-key-file /certs/cert.key
                tls-auth-clients optional
                tls-replication yes
            - name: sentinel.conf
              data: |
                sentinel resolve-hostnames yes
                sentinel monitor mymaster redis-1 6379 2
                sentinel down-after-milliseconds mymaster 60000
                sentinel failover-timeout mymaster 180000
                sentinel parallel-syncs mymaster 1

                sentinel announce-ip ${SYS_APP_NET_IPV4_ADDRESS}
                port 0
                tls-port 26379
                tls-ca-cert-file /certs/ca-cert.pem
                tls-cert-file /certs/cert.pem
                tls-key-file /certs/cert.key
                tls-auth-clients optional
                tls-replication yes
                requirepass ${REDIS_PASS}

The setup.sh is then executed as part of the init-container, once for each instance.

name: redis
services:
  - name: redis
    init-containers:
      - name: setup
        image: registry-1.docker.io/redis
        cmd: ["sh", "/cfg0/setup.sh"]
        mounts:
          - volume-name: cfg-storage
            mount-path: /cfg
          - volume-name: data
            mount-path: /data
          - volume-name: cfg0
            files:
              - name: setup.sh
                mount-path: /cfg0/setup.sh
              - name: redis.conf
                mount-path: /cfg0/redis.conf
              - name: sentinel.conf
                mount-path: /cfg0/sentinel.conf

Liveness probe

Liveness of a Redis server can for example be checked with the redis-cli ping command. So we simply specify this command as the liveness probe. The system will then periodically call this command to check the status of the Redis instance. If the probe fails, the system will restart the container.

name: redis
services:
  - name: redis
    init-containers:
    ...
    containers:
      - name: redis
        ...
        probes:
          liveness:
            exec:
              cmd:
                - sh
                - "-c"
                - redis-cli --tls --cacert /certs/ca-cert.pem ping

The application​

High Availability​

Replicas and affinity​

Rolling upgrades​

Readiness Probe - Ingress and DNS​

Volumes​

Configuration​

Data​

Secrets​

Configuration files​

Liveness probe​