Create a Redis cluster

tip

In order to be able to run through the steps of these tutorials in an operational Avassa environment, sign up for a free trial to get access to a running system.

warning

This is a more advanced tutorial and assumes you are famililar with e.g. supctl and the basic concepts of an application. Further, it assumes the sites have volumes configured.

The application

In this tutorial we will walk through an application specification for a Redis cluster, using the standard Redis docker image from dockerhub, unmodified.

We will deploy a Redis cluster with three instances, and one of them will serve as the leader and the other two as replicas.

The application consists of one image (redis) but we will run two containers in each instance:

• redis-server is what it says, the actual Redis server.
• redis-sentinel provides high availability for Redis and is responsible for monitoring the instance and, if needed, failing over to another instance.

The Redis replicas require a slightly different configuration file from the Redis leader, which complicates things a tad bit. The configuration files are modified by Redis itself after start, hence the configuration files have to reside on a volume that is read-write to the application.

We also want to configure a liveness probe to monitor the liveness/health of the cluster.

The application specification looks like this, and we will walk through each part in this tutorial.

name: redis
services:
  - name: redis
    init-containers:
      - name: setup
        # NOTE, the image is the same as for the containers. Otherwise the
        # system would have to download a different image just for the init
        # container.
        image: registry-1.docker.io/redis
        cmd: ["sh", "/cfg0/setup.sh"]
        # Allow the init container to run for 5 seconds
        execution-timeout: 5s
        mounts:
          - volume-name: cfg-storage
            mount-path: /cfg
          - volume-name: data
            mount-path: /data
          - volume-name: cfg0
            files:
              - name: setup.sh
                mount-path: /cfg0/setup.sh
              - name: redis.conf
                mount-path: /cfg0/redis.conf
              - name: sentinel.conf
                mount-path: /cfg0/sentinel.conf
    containers:
      - name: redis
        image: registry-1.docker.io/redis
        cmd: ["redis-server", "/cfg/redis.conf"]
        probes:
          liveness:
            exec:
              cmd: ["redis-cli", "ping"]
        mounts:
          - volume-name: cfg-storage
            mount-path: /cfg
          - volume-name: data
            mount-path: /data
      - name: redis-sentinel
        image: registry-1.docker.io/redis
        cmd: ["redis-sentinel", "/cfg/sentinel.conf"]
        mounts:
          - volume-name: cfg-storage
            mount-path: /cfg
          - volume-name: data
            mount-path: /data
    volumes:
      - name: cfg-storage
        ephemeral-volume:
          size: 3MiB
      - name: data
        ephemeral-volume:
          # this is the size of redis' data storage volume
          # it should be much larger in production
          size: 5MiB
      - name: cfg0
        config-map:
          items:
            - name: setup.sh
              data: |
                #!/bin/sh
                cp /cfg0/redis.conf /cfg/redis.conf
                chown redis:redis /cfg/redis.conf
                cp /cfg0/sentinel.conf /cfg/sentinel.conf
                if [ ${SYS_SERVICE_INSTANCE_INDEX} -gt 1 ]; then
                  echo "replicaof redis-1 6379" >> /cfg/redis.conf
                fi
            - name: redis.conf
              data: |
                protected-mode no
                port 6379
            - name: sentinel.conf
              data: |
                sentinel resolve-hostnames yes
                sentinel monitor mymaster redis-1 6379 2
                sentinel down-after-milliseconds mymaster 60000
                sentinel failover-timeout mymaster 180000
                sentinel parallel-syncs mymaster 1

    mode: replicated
    replicas: 3
    placement:
      preferred-anti-affinity:
        services: [ redis ]

Replicas and affinity

Since we want three instances of the redis and sentinel containers, we specify replicas: 3.

If we have enough hosts in a site, we also want to run each instance/replica on its own host. This is achieved by specifying affinity rules. In this case we simply say that the service (redis) should have anti affinity to itself. The system will then do its best to schedule each instance on different hosts.

    mode: replicated
    replicas: 3
    placement:
      preferred-anti-affinity:
        services: [ redis ]

Configuration volume

Since Redis needs to modify the configuration, and it needs to be persistent across restarts of Redis, we give each instance its own read-write volume, each with a size of 3 MB. This volume is mounted into each instance at /cfg.

When we start the redis processes, we pass them the name of their configuration files on the command line.

name: redis
services:
  - name: redis
    init-containers:
      - name: setup
        image: registry-1.docker.io/redis
        cmd: ["sh", "/cfg0/setup.sh"]
        mounts:
          - volume-name: cfg-storage
            mount-path: /cfg
          - volume-name: data
            mount-path: /data
          - volume-name: cfg0
            files:
              - name: setup.sh
                mount-path: /cfg0/setup.sh
              - name: redis.conf
                mount-path: /cfg0/redis.conf
              - name: sentinel.conf
                mount-path: /cfg0/sentinel.conf
    containers:
      - name: redis
        image: registry-1.docker.io/redis
        cmd: ["redis-server", "/cfg/redis.conf"]
        probes:
          liveness:
            exec:
              cmd: ["redis-cli", "ping"]
        mounts:
          - volume-name: cfg-storage
            mount-path: /cfg
          - volume-name: data
            mount-path: /data
      - name: redis-sentinel
        image: registry-1.docker.io/redis
        cmd: ["redis-sentinel", "/cfg/sentinel.conf"]
        mounts:
          - volume-name: cfg-storage
            mount-path: /cfg
          - volume-name: data
            mount-path: /data
    volumes:
      - name: cfg-storage
        ephemeral-volume:
          size: 3MB
      - name: data
        ephemeral-volume:
          # this is the size of redis' data storage volume
          # it should be much larger in production
          size: 5MiB
      - name: cfg0
        config-map:
          items:
            - name: setup.sh
              data: |
                #!/bin/sh
                cp /cfg0/redis.conf /cfg/redis.conf
                chown redis:redis /cfg/redis.conf
                cp /cfg0/sentinel.conf /cfg/sentinel.conf
                if [ ${SYS_SERVICE_INSTANCE_INDEX} -gt 1 ]; then
                  echo "replicaof redis-1 6379" >> /cfg/redis.conf
                fi
            - name: redis.conf
              data: |
                protected-mode no
                port 6379
            - name: sentinel.conf
              data: |
                sentinel resolve-hostnames yes
                sentinel monitor mymaster redis-1 6379 2
                sentinel down-after-milliseconds mymaster 60000
                sentinel failover-timeout mymaster 180000
                sentinel parallel-syncs mymaster 1

    mode: replicated
    replicas: 3
    placement:
      preferred-anti-affinity:
        services: [ redis ]

Configuration files

When the application is started the very first time on a host, the volume that is mounted to /cfg is empty. We need to create the required config files before the Redis processes are started.

We do this by specifying an init container. The init container is run to completion before the normal containers are started.

In our case, the init container uses the same image as the other container, and we run a short script (setup.sh) that creates the required configuration files.

The script itself and the configuration files for redis and redis-sentinel are specified in a config map. The setup.sh script copies the config files from the read-only config map to the read-write volume cfg-storage.

The SYS_SERVICE_INSTANCE_INDEX is an environment variable, in this case taking the values 1, 2 or 3 (since we have three replicas). If the instance index is greater than one, we modify the /cfg/redis.conf file to indicate that this instance is a replica of redis-1, which is the DNS name of the first service-instance.

      - name: cfg0
        config-map:
          items:
            - name: setup.sh
              data: |
                #!/bin/sh
                cp /cfg0/redis.conf /cfg/redis.conf
                chown redis:redis /cfg/redis.conf
                cp /cfg0/sentinel.conf /cfg/sentinel.conf
                if [ ${SYS_SERVICE_INSTANCE_INDEX} -gt 1 ]; then
                  echo "replicaof redis-1 6379" >> /cfg/redis.conf
                fi
            - name: redis.conf
              data: |
                protected-mode no
                port 6379
            - name: sentinel.conf
              data: |
                sentinel resolve-hostnames yes
                sentinel monitor mymaster redis-1 6379 2
                sentinel down-after-milliseconds mymaster 60000
                sentinel failover-timeout mymaster 180000
                sentinel parallel-syncs mymaster 1

The setup.sh is then executed as part of the init-container, once for each instance.

name: redis
services:
  - name: redis
    init-containers:
      - name: setup
        image: registry-1.docker.io/redis
        cmd: ["sh", "/cfg0/setup.sh"]
        mounts:
          - volume-name: cfg-storage
            mount-path: /cfg
          - volume-name: data
            mount-path: /data
          - volume-name: cfg0
            files:
              - name: setup.sh
                mount-path: /cfg0/setup.sh
              - name: redis.conf
                mount-path: /cfg0/redis.conf
              - name: sentinel.conf
                mount-path: /cfg0/sentinel.conf

Liveness probe

Liveness of a Redis server can for example be checked with the redis-cli ping command. So we simply specify this command as the liveness probe. The system will then periodically call this command to check the status of the Redis instance. If the probe fails, the system will restart the container.

name: redis
services:
  - name: redis
    init-containers:
    ...
    containers:
      - name: redis
        ...
        probes:
          liveness:
            exec:
              cmd: ["redis-cli", "ping"]