Thanks! We'll be in touch in the next 12 hours
Oops! Something went wrong while submitting the form.

Installing Redis Cluster with Persistent Storage on Mesosphere DC/OS

In the first part of this blog, we saw how to install standalone redis service on DC/OS with Persistent storage using RexRay and AWS EBS volumes.

A single server is a single point of failure in every system, so to ensure high availability of redis database, we can deploy a master-slave cluster of Redis servers. In this blog, we will see how to setup such 6 node (3 master, 3 slave) Redis cluster and persist data using RexRay and AWS EBS volumes. After that we will see how to import existing data into this cluster.

Redis Cluster

It is form of replicated Redis servers in multi-master architecture. All the data is sharded into 16384 buckets, where every master node is assigned subset of buckets out of them (generally evenly sharded) and each master replicated by its slaves.  It provides more resilience and scaling for production grade deployments where heavy workload is expected. Applications can connect to any node in cluster mode and the request will be redirected to respective master node.


Redis Cluster
 Source:  Octo

Objective: To create a Redis cluster with number of services in DCOC environment with persistent storage and import the existing Redis dump.rdb data to the cluster.

Prerequisites :  

  • Make sure rexray component is running and is in a healthy state for DCOS cluster.
DCOS Cluster


  • As per Redis doc, the minimal cluster should have at least 3 master and 3 slave nodes, so making it a total 6 Redis services.
  • All services will use similar json configuration except changes in names of service, external volume, and port mappings.
  • We will deploy one Redis service for each Redis cluster node and once all services are running, we will form cluster among them.
  • We will use host network for Redis node containers, for that we will restrict Redis nodes to run on particular node. This will help us to troubleshoot cluster (fixed IP, so we can restart Redis node any time without data loss).
  • Using host network adds a prerequisites that number of dcos nodes >= number of Redis nodes.
  1. First create Redis node services on DCOS:
  2. Click on the Add button in Services tab of DCOS UI
  • Click on JSON configuration
JSON Configuration
  • Add below json config for Redis service, change the values which are written in BLOCK letters with # as prefix and suffix.
  • #NODENAME# - Name of Redis node (Ex. redis-node-1)
  • #NODEHOSTIP# - IP of dcos node on which this Redis node will run. This ip must be unique for each Redis node. (Ex.
  • #VOLUMENAME# - Name of persistent volume, Give name to identify volume on AWS EBS (Ex. <dcos cluster name>-redis-node-<node number>)
  • #NODEVIP# - VIP For the Redis node. It must be ‘Redis’ for first Redis node, for others it can be the same as NODENAME (Ex. redis-node-2)


  • After updating the highlighted fields, copy above json to json configuration box, click on ‘Review & Run’ button in the right corner, this will start the service with above configuration.
  • Once above service is UP and Running, then repeat the step 2 to 4 for each Redis node with respective values for highlighted fields.
  • So if we go with 6 node cluster, at the end we will have 6 Redis nodes UP and Running, like:
Redis Nodes

Note: Since we are using external volume for persistent storage, we can not scale our services, i.e. each service will only one instance max. If we try to scale, we will get below error :

Scale Service Error

2. Form the Redis cluster between Redis node services:

  • To create or manage Redis-cluster, first deploy redis-cluster-util container on DCOS using below json config:


This will run service as :

Run Redis Service
  • Get the IP addresses of all Redis nodes to form the cluster, as Redis-cluster can not be created with node's hostname / dns. This is an open issue.

Since we are using host network, we need the dcos node IP on which Redis nodes are running.

Running Redis Nodes

Get all Redis nodes IP using:

NODE_BASE_NAME=redis-nodedcos task $NODE_BASE_NAME | grep -E "$NODE_BASE_NAME-\<[0-9]\>" | awk '{print $2":6379"}' | paste -s -d' '  

Here Redis-node is the prefix used for all Redis nodes.

Note the output of this command, we will use it in further steps.

  • Get the node where redis-cluster-util container is running and ssh to dcos node using:
dcos node ssh --master-proxy --private-ip $(dcos task | grep "redis-cluster-util" | awk '{print $2}')
  • Now find the docker container id of redis-cluster-util and exec it using:
docker exec -it $(docker ps -qf ancestor="parvezkazi13/redis-util") bash  
  • No we are inside the redis-cluster-util container. Run below command to form Redis cluster.
redis-trib.rb create --replicas 1 <Space separated IP address:PORT pair of all Redis nodes>
  • Here use the Redis nodes IP addresses which retrieved in step 2.
redis-trib.rb create --replicas 1
  • Parameters:
  • The option --replicas 1 means that we want a slave for every master created.
  • The other arguments are the list of addresses (host:port) of the instances we want to use to create the new cluster.
  • Output:
  • Select ‘yes’ when it prompts to set the slot configuration shown.
  • Run below command to check the status of the newly create cluster
redis-trib.rb check <Any redis node host:PORT>
redis-trib.rb check
  • Parameters:
  • host:port of any node from the cluster.
  • Output:
  • If all OK, it will show OK with status, else it will show ERR with the error message.

3. Import existing dump.rdb to Redis cluster

  • At this point, all the Redis nodes should be empty and each one should have an ID and some assigned slots:

Before reuse an existing dump data, we have to reshard all slots to one instance. We specify the number of slots to move (all, so 16384), the id we move to (here Node 1 - and where we take these slots from (all other nodes).

redis-trib.rb reshard  


host:port of any node from the cluster.


It will prompt for number of slots to move - here all. i.e 16384

Receiving node id - here id of node (redis-node-1)

Source node IDs  - here all, as we want to shard  all slots to one node.

And prompt to proceed - press ‘yes’

Output Prompt
  • Now check again node
redis-trib.rb check  

Parameters: host:port of any node from the cluster.

Output: it will show all (16384) slots moved to node

  • Next step is Importing our existing Redis dump data.  

Now copy the existing dump.rdb to our redis-cluster-util container using below steps:

- Copy existing dump.rdb to dcos node on which redis-cluster-util container is running. Can use scp from any other public server to dcos node.

- Now we have dump .rdb in our dcos node, copy this dump.rdb to redis-cluster-util container using below command:

docker cp dump.rdb $(docker ps -qf ancestor="parvezkazi13/redis-util"):/data

Now we have dump.rdb in our redis-cluster-util container, we can import it to our Redis cluster. Execute and go to the redis-cluster-util container using:

docker exec -it $(docker ps -qf ancestor="parvezkazi13/redis-util") bash

It will execute redis-cluster-util container which is already running and start its bash cmd.

Run  below command to import dump.rdb to Redis cluster:

rdb --command protocol /data/dump.rdb | redis-cli --pipe -h -p 6379  


Path to dump.rdb

host:port of any node from the cluster.


If successful, you’ll see something like:

All data transferred. Waiting for the last reply...Last reply received from server.errors: 0, replies: 4341259  

as well as this in the Redis server logs:

95086:M 01 Mar 21:53:42.071 * 10000 changes in 60 seconds. Saving...95086:M 01 Mar 21:53:42.072 * Background saving started by pid 9822398223:C 01 Mar 21:53:44.277 * DB saved on disk
Like our Oracle DB instance can have multiple databases, similarly Redis saves keys in keyspaces.
Now when Redis is in cluster mode, it does not accept the dumps which has more than one keyspaces. As per documentation :
Redis Cluster does not support multiple databases like the stand alone version of Redis. There is just database 0 and the SELECT command is not allowed. “
So while importing such multi-keyspace Redis dump, server fails while starting on below issue :
23049:M 16 Mar 17:21:17.772 * DB loaded from disk: 5.222 seconds
23049:M 16 Mar 17:21:17.772 # You can't have keys in a DB different than DB 0 when in Cluster mode. Exiting.
Solution / WA :
There is redis-cli command “MOVE” to move keys from one keyspace to another keyspace.
Also can run below command to move all the keys from keyspace 1 to keyspace 0 :
redis-cli -h "$HOST" -p "$PORT" -n 1 --raw keys "*" |  xargs -I{} redis-cli -h "$HOST" -p "$PORT" -n 1 move {} 0
  • Verify import status, using below commands : (inside redis-cluster-util container)
redis-cli -h -p 6379 info keyspace

It will run Redis info command on node and fetch keyspace information, like below:

# Keyspace

  • Now reshard all the slots to all instances evenly

The reshard command will again list the existing nodes, their IDs and the assigned slots.

redis-trib.rb reshard


host:port of any node from the cluster.


It will prompt for number of slots to move - here (16384 /3 Masters = 5461)

Receiving node id - here id of master node 2  

Source node IDs  - id of first instance which has currently all the slots. (master 1)

And prompt to proceed - press ‘yes’

Repeat above step and for receiving node id, give id of master node 3.

  • After the above step, all 3 masters will have equal slots and imported keys will be distributed among the master nodes.
  • Put keys to cluster for verification
redis-cli -h -p 6379 set foo bar
redis-cli -h -p 6379 set foo bar
(error) MOVED 4813

Above error shows that server saved this key to instance, so client redirected it. To follow redirection, use flag “-c” which says it is a cluster mode, like:

redis-cli -h -p 6379 -c set foo bar

Redis Entrypoint

Application entrypoint for Redis cluster is mostly depends how your Redis client handles cluster support. Generally connecting to one of master nodes should do the work.

Use below host:port in your applications :

Automation of Redis Cluster Creation

We have automation script in place to deploy 6 node Redis cluster and form a cluster between them.

Script location: Github

  • It deploys 6 marathon apps for 6 Redis nodes. All nodes are deployed on different nodes with CLUSTER_NAME as prefix to volume name.
  • Once all nodes are up and running, it deploys redis-cluster-util app which will be used to form Redis cluster.
  • Then it will print the Redis nodes and their IP addresses and prompt the user to proceed cluster creation.
  • If user selects to proceed, it will run redis-cluster-util app and create the cluster using IP addresses collected. Util container will prompt for some input that the user has to select.


We learned about Redis cluster deployment on DCOS with Persistent storage using RexRay. We also learned how rexray automatically manages volumes over aws ebs and how to integrate them in DCOS apps/services. We saw how to use redis-cluster-util container to manage Redis cluster for different purposes, like forming cluster, resharding, importing existing dump.rdb data etc. Finally, we looked at the automation part of whole cluster setup using dcos cli and bash.