Redis5 Cluster 구성하기

지난 포스팅에서는 Redis에 대해 알아보고 설치하는 과정에 대해 살펴보았다. 이번 포스팅에서는 Redis의 가용성을 높여주기 위한 Cluster 환경을 구성하는 방법에 대해 살펴보자.

지난 포스팅은 다음을 참고한다.

[Redis 5.x] 오픈소스 기반 비관계형 데이터베이스 관리 시스템
[Redis 5.x] 설치 및 CLI 활용 가이드

---

Master Cluster 구성

1. Redis Master Server Add (sh install_server.sh)

먼저 Cluster에 추가할 서버를 설치한다. (localhost:6379, localhost:6479, localhost:6579)

##### install_server.sh 위치 #####
[root@ip-192-168-73-84 redis-5.0.5]# cd utils/
[root@ip-192-168-73-84 utils]# ls
build-static-symbols.tcl  corrupt_rdb.c   generate-command-help.rb  hashtable    install_server.sh  redis-copy.rb      redis_init_script.tpl  releasetools          whatisdoing.sh
cluster_fail_time.tcl     create-cluster  graphs                    hyperloglog  lru                redis_init_script  redis-sha1.rb          speed-regression.tcl

##### 6379 server add  #####
[root@ip-192-168-73-84 utils]# sh install_server.sh 
Welcome to the redis service installer
This script will help you easily set up a running redis server

Please select the redis port for this instance: [6379] 
Selecting default: 6379
Please select the redis config file name [/etc/redis/6379.conf] 
Selected default - /etc/redis/6379.conf
Please select the redis log file name [/var/log/redis_6379.log] 
Selected default - /var/log/redis_6379.log
Please select the data directory for this instance [/var/lib/redis/6379] 
Selected default - /var/lib/redis/6379
Please select the redis executable path [/usr/local/bin/redis-server] 
Selected config:
Port           : 6379
Config file    : /etc/redis/6379.conf
Log file       : /var/log/redis_6379.log
Data dir       : /var/lib/redis/6379
Executable     : /usr/local/bin/redis-server
Cli Executable : /usr/local/bin/redis-cli
Is this ok? Then press ENTER to go on or Ctrl-C to abort.
Copied /tmp/6379.conf => /etc/init.d/redis_6379
Installing service...
Successfully added to chkconfig!
Successfully added to runlevels 345!
Starting Redis server...
Installation successful!
[root@ip-192-168-73-84 utils]#

##### 6479 server add  #####
Please select the redis port for this instance: [6379] 6479       
Please select the redis config file name [/etc/redis/6479.conf] 
Selected default - /etc/redis/6479.conf
Please select the redis log file name [/var/log/redis_6479.log] 
Selected default - /var/log/redis_6479.log
Please select the data directory for this instance [/var/lib/redis/6479] 
Selected default - /var/lib/redis/6479
Please select the redis executable path [/usr/local/bin/redis-server] 
Selected config:
Port           : 6479
Config file    : /etc/redis/6479.conf
Log file       : /var/log/redis_6479.log
Data dir       : /var/lib/redis/6479
Executable     : /usr/local/bin/redis-server
Cli Executable : /usr/local/bin/redis-cli
Is this ok? Then press ENTER to go on or Ctrl-C to abort.
Copied /tmp/6479.conf => /etc/init.d/redis_6479
Installing service...

Successfully added to chkconfig!
Successfully added to runlevels 345!
Starting Redis server...
Installation successful!

##### 6579 server add  #####
[root@ip-192-168-73-84 utils]# sh install_server.sh 
Welcome to the redis service installer
This script will help you easily set up a running redis server

Please select the redis port for this instance: [6379] 6579
Please select the redis config file name [/etc/redis/6579.conf] 
Selected default - /etc/redis/6579.conf
Please select the redis log file name [/var/log/redis_6579.log] 
Selected default - /var/log/redis_6579.log
Please select the data directory for this instance [/var/lib/redis/6579] 
Selected default - /var/lib/redis/6579
Please select the redis executable path [/usr/local/bin/redis-server] 
Selected config:
Port           : 6579
Config file    : /etc/redis/6579.conf
Log file       : /var/log/redis_6579.log
Data dir       : /var/lib/redis/6579
Executable     : /usr/local/bin/redis-server
Cli Executable : /usr/local/bin/redis-cli
Is this ok? Then press ENTER to go on or Ctrl-C to abort.
Copied /tmp/6579.conf => /etc/init.d/redis_6579
Installing service...
Successfully added to chkconfig!
Successfully added to runlevels 345!
Starting Redis server...
Installation successful!
[root@ip-192-168-73-84 utils]#

2. Redis Master Config Modify (vi /etc/redis/6379.conf)

[root@ip-192-168-73-84 utils]# cd /etc/redis/
[root@ip-192-168-73-84 redis]# ls
6379.conf  6479.conf  6579.conf
[root@ip-192-168-73-84 redis]# vi *
3 files to edit
...
...
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
...
appendonly yes
...
...
[root@ip-192-168-73-84 redis]# 

6379.conf, 6479.conf, 6579.conf 각각 위와 같이 설정파일을 수정하고 저장한다.

3. Redis Master Server Restart (service redis_6379 restart)

[root@ip-192-168-73-84 redis]# service redis_6379 restart
Stopping ...
Redis stopped
Starting Redis server...
[root@ip-192-168-73-84 redis]# service redis_6479 restart
Stopping ...
Redis stopped
Starting Redis server...
[root@ip-192-168-73-84 redis]# service redis_6579 restart
Stopping ...
Redis stopped
Starting Redis server...
[root@ip-192-168-73-84 redis]# ps -ef | grep redis
root      8432     1  0 22:47 ?        00:00:00 /usr/local/bin/redis-server 127.0.0.1:6379 [cluster]
root      8449     1  0 22:47 ?        00:00:00 /usr/local/bin/redis-server 127.0.0.1:6479 [cluster]
root      8466     1  0 22:47 ?        00:00:00 /usr/local/bin/redis-server 127.0.0.1:6579 [cluster]
root      8471  7923  0 22:47 pts/1    00:00:00 grep --color=auto redis
[root@ip-192-168-73-84 redis]#

Redis Cluster는 ruby로 구성되어 있어 아래와 같이 package 설치를 진행한다.

yum install ruby ruby-devel rubygems rpm-build

gem install redis 명령어로 Redis Cluster 설치시 다음과 같은 문제가 발생했을 경우 ruby 버전을 다음과 같이 업그레이드 한다.

[root@ip-192-168-106-237 utils]# gem install redis
Fetching: redis-4.2.5.gem (100%)
ERROR:  Error installing redis:
        redis requires Ruby version >= 2.3.0.
[root@ip-192-168-106-237 utils]# yum install curl gpg gcc gcc-c++ make patch autoconf automake bison libffi-devel libtool patch readline-devel sqlite-devel zlib-devel openssl-devel
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
Package curl-7.61.1-12.amzn2.0.2.x86_64 already installed and latest version
Package gnupg2-2.0.22-5.amzn2.0.4.x86_64 already installed and latest version
Package gcc-7.3.1-12.amzn2.x86_64 already installed and latest version
Package 1:make-3.82-24.amzn2.x86_64 already installed and latest version
Package patch-2.7.1-12.amzn2.0.2.x86_64 already installed and latest version
Package patch-2.7.1-12.amzn2.0.2.x86_64 already installed and latest version
Resolving Dependencies
...
...
[root@ip-192-168-106-237 utils]# gpg2 --keyserver hkp://pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
gpg: directory `/root/.gnupg' created
gpg: new configuration file `/root/.gnupg/gpg.conf' created
gpg: WARNING: options in `/root/.gnupg/gpg.conf' are not yet active during this run
gpg: keyring `/root/.gnupg/secring.gpg' created
gpg: keyring `/root/.gnupg/pubring.gpg' created
gpg: requesting key D39DC0E3 from hkp server pool.sks-keyservers.net
gpg: requesting key 39499BDB from hkp server pool.sks-keyservers.net
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key D39DC0E3: public key "Michal Papis (RVM signing) <mpapis@gmail.com>" imported
gpg: key 39499BDB: public key "Piotr Kuczynski <piotr.kuczynski@gmail.com>" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 2
gpg:               imported: 2  (RSA: 2)
[root@ip-192-168-106-237 utils]# curl -sSL https://get.rvm.io | bash -s stable
Downloading https://github.com/rvm/rvm/archive/1.29.12.tar.gz
Downloading https://github.com/rvm/rvm/releases/download/1.29.12/1.29.12.tar.gz.asc
gpg: Signature made Fri 15 Jan 2021 06:46:22 PM UTC using RSA key ID 39499BDB
gpg: Good signature from "Piotr Kuczynski <piotr.kuczynski@gmail.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 7D2B AF1C F37B 13E2 069D  6956 105B D0E7 3949 9BDB
GPG verified '/usr/local/rvm/archives/rvm-1.29.12.tgz'
Creating group 'rvm'
Installing RVM to /usr/local/rvm/
Installation of RVM in /usr/local/rvm/ is almost complete:

  * First you need to add all users that will be using rvm to 'rvm' group,
    and logout - login again, anyone using rvm will be operating with `umask u=rwx,g=rwx,o=rx`.

  * To start using RVM you need to run `source /etc/profile.d/rvm.sh`
    in all your open shell windows, in rare cases you need to reopen all shell windows.
  * Please do NOT forget to add your users to the rvm group.
     The installer no longer auto-adds root or users to the rvm group. Admins must do this.
     Also, please note that group memberships are ONLY evaluated at login time.
     This means that users must log out then back in before group membership takes effect!
Thanks for installing RVM 솋
Please consider donating to our open collective to help us maintain RVM.

몛  Donate: https://opencollective.com/rvm/donate


[root@ip-192-168-106-237 utils]# source /usr/local/rvm/scripts/rvm
[root@ip-192-168-106-237 utils]# rvm install 2.5.1
Searching for binary rubies, this might take some time.
Found remote file https://rvm_io.global.ssl.fastly.net/binaries/amazon/2/x86_64/ruby-2.5.1.tar.bz2
Checking requirements for amazon.
Installing requirements for amazon.
Installing required packages: libyaml-devel....
Requirements installation successful.
ruby-2.5.1 - #configure
ruby-2.5.1 - #download
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 17.5M  100 17.5M    0     0  3830k      0  0:00:04  0:00:04 --:--:-- 3830k
ruby-2.5.1 - #validate archive
ruby-2.5.1 - #extract
ruby-2.5.1 - #validate binary
ruby-2.5.1 - #setup
ruby-2.5.1 - #gemset created /usr/local/rvm/gems/ruby-2.5.1@global
ruby-2.5.1 - #importing gemset /usr/local/rvm/gemsets/global.gems..................................
ruby-2.5.1 - #generating global wrappers........
ruby-2.5.1 - #gemset created /usr/local/rvm/gems/ruby-2.5.1
ruby-2.5.1 - #importing gemsetfile /usr/local/rvm/gemsets/default.gems evaluated to empty gem list
ruby-2.5.1 - #generating default wrappers........
[root@ip-192-168-106-237 utils]# rvm use 2.5.1 --default
Using /usr/local/rvm/gems/ruby-2.5.1
[root@ip-192-168-106-237 utils]#

4. Redis Master Cluster Join (redis-cli --cluster create)

[root@ip-192-168-73-84 redis]# redis-cli --cluster create 127.0.0.1:6379 127.0.0.1:6479 127.0.0.1:6579
>>> Performing hash slots allocation on 3 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
M: 4d778457522d238b0613b96e656c611b0405f2a3 127.0.0.1:6379
   slots:[0-5460] (5461 slots) master
M: 10022df5d01fddeb7a12e483ae971784bcbd72ba 127.0.0.1:6479
   slots:[5461-10922] (5462 slots) master
M: cde730ed40c71395be543dffd3e893c66357d04e 127.0.0.1:6579
   slots:[10923-16383] (5461 slots) master
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
..
>>> Performing Cluster Check (using node 127.0.0.1:6379)
M: 4d778457522d238b0613b96e656c611b0405f2a3 127.0.0.1:6379
   slots:[0-5460] (5461 slots) master
M: 10022df5d01fddeb7a12e483ae971784bcbd72ba 127.0.0.1:6479
   slots:[5461-10922] (5462 slots) master
M: cde730ed40c71395be543dffd3e893c66357d04e 127.0.0.1:6579
   slots:[10923-16383] (5461 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@ip-192-168-73-84 redis]# redis-cli -p 6379
127.0.0.1:6379> flushall
OK
127.0.0.1:6379> exit
[root@ip-192-168-73-84 redis]# redis-cli -c -p 6379
127.0.0.1:6379> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:3
cluster_size:3
cluster_current_epoch:3
cluster_my_epoch:1
cluster_stats_messages_ping_sent:49
cluster_stats_messages_pong_sent:50
cluster_stats_messages_sent:99
cluster_stats_messages_ping_received:48
cluster_stats_messages_pong_received:49
cluster_stats_messages_meet_received:2
cluster_stats_messages_received:99
127.0.0.1:6379> cluster nodes
10022df5d01fddeb7a12e483ae971784bcbd72ba 127.0.0.1:6479@16479 master - 0 1620169482529 2 connected 5461-10922
4d778457522d238b0613b96e656c611b0405f2a3 127.0.0.1:6379@16379 myself,master - 0 1620169480000 1 connected 0-5460
cde730ed40c71395be543dffd3e893c66357d04e 127.0.0.1:6579@16579 master - 0 1620169481527 3 connected 10923-16383
127.0.0.1:6379> set a a
-> Redirected to slot [15495] located at 127.0.0.1:6579
OK
127.0.0.1:6579> set b b
-> Redirected to slot [3300] located at 127.0.0.1:6379
OK
127.0.0.1:6379> set c c
-> Redirected to slot [7365] located at 127.0.0.1:6479
OK
127.0.0.1:6479> set d d
-> Redirected to slot [11298] located at 127.0.0.1:6579
OK
127.0.0.1:6579> set e e
OK
127.0.0.1:6579> exit
[root@ip-192-168-73-84 redis]#

---

Slave Server 구성

Master Node만 구성하여 Redis를 설계할 경우 가용성에 문제가 발생할 수 있다. 데이터 유실 방지 등을 위해 각 Redis Master Node는 Slave Node를 갖도록 구성하는 것이 좋다.

다음은 Redis Slave Node를 구성하는 과정이다.

1. Redis Slave Server Create (sh install_server.sh)

먼저 Cluster에 추가할 서버를 설치한다. (localhost:7379, localhost:7479, localhost:7579)

 

[root@ip-192-168-73-84 utils]# sh install_server.sh 
Welcome to the redis service installer
This script will help you easily set up a running redis server

Please select the redis port for this instance: [6379] 7379
Please select the redis config file name [/etc/redis/7379.conf] 
Selected default - /etc/redis/7379.conf
Please select the redis log file name [/var/log/redis_7379.log] 
Selected default - /var/log/redis_7379.log
Please select the data directory for this instance [/var/lib/redis/7379] 
Selected default - /var/lib/redis/7379
Please select the redis executable path [/usr/local/bin/redis-server] 
Selected config:
Port           : 7379
Config file    : /etc/redis/7379.conf
Log file       : /var/log/redis_7379.log
Data dir       : /var/lib/redis/7379
Executable     : /usr/local/bin/redis-server
Cli Executable : /usr/local/bin/redis-cli
Is this ok? Then press ENTER to go on or Ctrl-C to abort.
Copied /tmp/7379.conf => /etc/init.d/redis_7379
Installing service...
Successfully added to chkconfig!
Successfully added to runlevels 345!
Starting Redis server...
Installation successful!
[root@ip-192-168-73-84 utils]# sh install_server.sh 
Welcome to the redis service installer
This script will help you easily set up a running redis server

Please select the redis port for this instance: [6379] 7479
Please select the redis config file name [/etc/redis/7479.conf] 
Selected default - /etc/redis/7479.conf
Please select the redis log file name [/var/log/redis_7479.log] 
Selected default - /var/log/redis_7479.log
Please select the data directory for this instance [/var/lib/redis/7479] 
Selected default - /var/lib/redis/7479
Please select the redis executable path [/usr/local/bin/redis-server] 
Selected config:
Port           : 7479
Config file    : /etc/redis/7479.conf
Log file       : /var/log/redis_7479.log
Data dir       : /var/lib/redis/7479
Executable     : /usr/local/bin/redis-server
Cli Executable : /usr/local/bin/redis-cli
Is this ok? Then press ENTER to go on or Ctrl-C to abort.
Copied /tmp/7479.conf => /etc/init.d/redis_7479
Installing service...
Successfully added to chkconfig!
Successfully added to runlevels 345!
Starting Redis server...
Installation successful!
[root@ip-192-168-73-84 utils]# sh install_server.sh 
Welcome to the redis service installer
This script will help you easily set up a running redis server

Please select the redis port for this instance: [6379] 7579
Please select the redis config file name [/etc/redis/7579.conf] 
Selected default - /etc/redis/7579.conf
Please select the redis log file name [/var/log/redis_7579.log] 
Selected default - /var/log/redis_7579.log
Please select the data directory for this instance [/var/lib/redis/7579] 
Selected default - /var/lib/redis/7579
Please select the redis executable path [/usr/local/bin/redis-server] 
Selected config:
Port           : 7579
Config file    : /etc/redis/7579.conf
Log file       : /var/log/redis_7579.log
Data dir       : /var/lib/redis/7579
Executable     : /usr/local/bin/redis-server
Cli Executable : /usr/local/bin/redis-cli
Is this ok? Then press ENTER to go on or Ctrl-C to abort.
Copied /tmp/7579.conf => /etc/init.d/redis_7579
Installing service...
Successfully added to chkconfig!
Successfully added to runlevels 345!
Starting Redis server...
Installation successful!
[root@ip-192-168-73-84 utils]# ps -ef | grep redis
root      5267     1  0 23:34 ?        00:00:00 /usr/local/bin/redis-server 127.0.0.1:7379
root      5340     1  0 23:34 ?        00:00:00 /usr/local/bin/redis-server 127.0.0.1:7479
root      5414     1  0 23:35 ?        00:00:00 /usr/local/bin/redis-server 127.0.0.1:7579
root      5422  7923  0 23:38 pts/1    00:00:00 grep --color=auto redis
root      8432     1  0 22:47 ?        00:00:02 /usr/local/bin/redis-server 127.0.0.1:6379 [cluster]
root      8449     1  0 22:47 ?        00:00:02 /usr/local/bin/redis-server 127.0.0.1:6479 [cluster]
root      8466     1  0 22:47 ?        00:00:02 /usr/local/bin/redis-server 127.0.0.1:6579 [cluster]
[root@ip-192-168-73-84 utils]#

2. Redis Slave Config Modify (vi /etc/redis/7379.conf)

[root@ip-192-168-73-84 utils]# cd /etc/redis/
[root@ip-192-168-73-84 redis]# ls 7*
7379.conf  7479.conf  7579.conf
[root@ip-192-168-73-84 redis]# vi 7*
3 files to edit
...
...
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
...
appendonly yes
...
...
[root@ip-192-168-73-84 redis]# 

7379.conf, 7479.conf, 7579.conf 각각 위와 같이 설정파일을 수정하고 저장한다.

3. Redis Master Server Restart (service redis_7379 restart)

[root@ip-192-168-73-84 redis]# service redis_7379 restart
Stopping ...
Redis stopped
Starting Redis server...
[root@ip-192-168-73-84 redis]# service redis_7479 restart
Stopping ...
Redis stopped
Starting Redis server...
[root@ip-192-168-73-84 redis]# service redis_7579 restart
Stopping ...
Redis stopped
Starting Redis server...
[root@ip-192-168-73-84 utils]# ps -ef | grep redis
root      5554     1  0 23:41 ?        00:00:00 /usr/local/bin/redis-server 127.0.0.1:7379 [cluster]
root      5571     1  0 23:42 ?        00:00:00 /usr/local/bin/redis-server 127.0.0.1:7479 [cluster]
root      5589     1  0 23:42 ?        00:00:00 /usr/local/bin/redis-server 127.0.0.1:7579 [cluster]
root      5598  7923  0 23:45 pts/1    00:00:00 grep --color=auto redis
root      8432     1  0 22:47 ?        00:00:03 /usr/local/bin/redis-server 127.0.0.1:6379 [cluster]
root      8449     1  0 22:47 ?        00:00:03 /usr/local/bin/redis-server 127.0.0.1:6479 [cluster]
root      8466     1  0 22:47 ?        00:00:03 /usr/local/bin/redis-server 127.0.0.1:6579 [cluster]
[root@ip-192-168-73-84 utils]#

4. Redis Slave Server Join (redis-cli --cluster add-node --cluster-slave)

[root@ip-192-168-73-84 utils]# redis-cli --cluster add-node 127.0.0.1:7379 127.0.0.1:6379 --cluster-slave
>>> Adding node 127.0.0.1:7379 to cluster 127.0.0.1:6379
>>> Performing Cluster Check (using node 127.0.0.1:6379)
M: 4d778457522d238b0613b96e656c611b0405f2a3 127.0.0.1:6379
   slots:[0-5460] (5461 slots) master
M: 10022df5d01fddeb7a12e483ae971784bcbd72ba 127.0.0.1:6479
   slots:[5461-10922] (5462 slots) master
M: cde730ed40c71395be543dffd3e893c66357d04e 127.0.0.1:6579
   slots:[10923-16383] (5461 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Automatically selected master 127.0.0.1:6379
>>> Send CLUSTER MEET to node 127.0.0.1:7379 to make it join the cluster.
Waiting for the cluster to join

>>> Configure node as replica of 127.0.0.1:6379.
[OK] New node added correctly.
[root@ip-192-168-73-84 utils]# redis-cli --cluster add-node 127.0.0.1:7479 127.0.0.1:6479 --cluster-slave
>>> Adding node 127.0.0.1:7479 to cluster 127.0.0.1:6479
>>> Performing Cluster Check (using node 127.0.0.1:6479)
M: 10022df5d01fddeb7a12e483ae971784bcbd72ba 127.0.0.1:6479
   slots:[5461-10922] (5462 slots) master
M: 4d778457522d238b0613b96e656c611b0405f2a3 127.0.0.1:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 6d559070ff6dbef46431d985397fbdd2ab0c6842 127.0.0.1:7379
   slots: (0 slots) slave
   replicates 4d778457522d238b0613b96e656c611b0405f2a3
M: cde730ed40c71395be543dffd3e893c66357d04e 127.0.0.1:6579
   slots:[10923-16383] (5461 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Automatically selected master 127.0.0.1:6479
>>> Send CLUSTER MEET to node 127.0.0.1:7479 to make it join the cluster.
Waiting for the cluster to join

>>> Configure node as replica of 127.0.0.1:6479.
[OK] New node added correctly.
[root@ip-192-168-73-84 utils]# redis-cli --cluster add-node 127.0.0.1:7579 127.0.0.1:6579 --cluster-slave
>>> Adding node 127.0.0.1:7579 to cluster 127.0.0.1:6579
>>> Performing Cluster Check (using node 127.0.0.1:6579)
M: cde730ed40c71395be543dffd3e893c66357d04e 127.0.0.1:6579
   slots:[10923-16383] (5461 slots) master
S: d47e315c1bab0bfa9a7b4dc663cea391c1d90def 127.0.0.1:7479
   slots: (0 slots) slave
   replicates 10022df5d01fddeb7a12e483ae971784bcbd72ba
S: 6d559070ff6dbef46431d985397fbdd2ab0c6842 127.0.0.1:7379
   slots: (0 slots) slave
   replicates 4d778457522d238b0613b96e656c611b0405f2a3
M: 4d778457522d238b0613b96e656c611b0405f2a3 127.0.0.1:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: 10022df5d01fddeb7a12e483ae971784bcbd72ba 127.0.0.1:6479
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Automatically selected master 127.0.0.1:6579
>>> Send CLUSTER MEET to node 127.0.0.1:7579 to make it join the cluster.
Waiting for the cluster to join

>>> Configure node as replica of 127.0.0.1:6579.
[OK] New node added correctly.
[root@ip-192-168-73-84 utils]#

5. Redis Cluster Info

[root@ip-192-168-73-84 utils]# redis-cli --cluster info 127.0.0.1:6379
127.0.0.1:6379 (4d778457...) -> 1 keys | 5461 slots | 1 slaves.
127.0.0.1:6479 (10022df5...) -> 1 keys | 5462 slots | 1 slaves.
127.0.0.1:6579 (cde730ed...) -> 3 keys | 5461 slots | 1 slaves.
[OK] 5 keys in 3 masters.
0.00 keys per slot on average.
[root@ip-192-168-73-84 utils]# redis-cli -p 6379
127.0.0.1:6379> cluster nodes
4d778457522d238b0613b96e656c611b0405f2a3 127.0.0.1:6379@16379 myself,master - 0 1620172101000 1 connected 0-5460
10022df5d01fddeb7a12e483ae971784bcbd72ba 127.0.0.1:6479@16479 master - 0 1620172100000 2 connected 5461-10922
579851e6d3a9a8db16b2fec5ef5418dfd9f7a7f3 127.0.0.1:7579@17579 slave cde730ed40c71395be543dffd3e893c66357d04e 0 1620172100000 3 connected
6d559070ff6dbef46431d985397fbdd2ab0c6842 127.0.0.1:7379@17379 slave 4d778457522d238b0613b96e656c611b0405f2a3 0 1620172099000 1 connected
cde730ed40c71395be543dffd3e893c66357d04e 127.0.0.1:6579@16579 master - 0 1620172100671 3 connected 10923-16383
d47e315c1bab0bfa9a7b4dc663cea391c1d90def 127.0.0.1:7479@17479 slave 10022df5d01fddeb7a12e483ae971784bcbd72ba 0 1620172101673 2 connected
127.0.0.1:6379> exit
[root@ip-192-168-73-84 utils]# redis-cli --cluster check 127.0.0.1:6379
127.0.0.1:6379 (4d778457...) -> 1 keys | 5461 slots | 1 slaves.
127.0.0.1:6479 (10022df5...) -> 1 keys | 5462 slots | 1 slaves.
127.0.0.1:6579 (cde730ed...) -> 3 keys | 5461 slots | 1 slaves.
[OK] 5 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 127.0.0.1:6379)
M: 4d778457522d238b0613b96e656c611b0405f2a3 127.0.0.1:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: 10022df5d01fddeb7a12e483ae971784bcbd72ba 127.0.0.1:6479
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 579851e6d3a9a8db16b2fec5ef5418dfd9f7a7f3 127.0.0.1:7579
   slots: (0 slots) slave
   replicates cde730ed40c71395be543dffd3e893c66357d04e
S: 6d559070ff6dbef46431d985397fbdd2ab0c6842 127.0.0.1:7379
   slots: (0 slots) slave
   replicates 4d778457522d238b0613b96e656c611b0405f2a3
M: cde730ed40c71395be543dffd3e893c66357d04e 127.0.0.1:6579
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: d47e315c1bab0bfa9a7b4dc663cea391c1d90def 127.0.0.1:7479
   slots: (0 slots) slave
   replicates 10022df5d01fddeb7a12e483ae971784bcbd72ba
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@ip-192-168-73-84 utils]#

Master & Slave Node 구성이 완료되면 위와 같이 확인이 가능하다.

---

Redis 구성 및 동작 방식 확인

총 6개의 Redis Server가 구성되어 있으며, 3개의 Master와 3개의 Slave로 구성된다.

[root@ip-192-168-73-84 6379]# redis-cli -c -p 6379
127.0.0.1:6379> set a a
-> Redirected to slot [15495] located at 127.0.0.1:6579
OK
127.0.0.1:6579> set b b
-> Redirected to slot [3300] located at 127.0.0.1:6379
OK
127.0.0.1:6379> set c c
-> Redirected to slot [7365] located at 127.0.0.1:6479
OK
127.0.0.1:6479> set d d
-> Redirected to slot [11298] located at 127.0.0.1:6579
OK
127.0.0.1:6579> set e e
OK
127.0.0.1:6579> set f f
-> Redirected to slot [3168] located at 127.0.0.1:6379
OK
127.0.0.1:6379> cluster nodes
4d778457522d238b0613b96e656c611b0405f2a3 127.0.0.1:6379@16379 myself,master - 0 1620206875000 1 connected 0-5460
10022df5d01fddeb7a12e483ae971784bcbd72ba 127.0.0.1:6479@16479 master - 0 1620206876000 2 connected 5461-10922
579851e6d3a9a8db16b2fec5ef5418dfd9f7a7f3 127.0.0.1:7579@17579 slave cde730ed40c71395be543dffd3e893c66357d04e 0 1620206877407 3 connected
6d559070ff6dbef46431d985397fbdd2ab0c6842 127.0.0.1:7379@17379 slave 4d778457522d238b0613b96e656c611b0405f2a3 0 1620206875401 1 connected
cde730ed40c71395be543dffd3e893c66357d04e 127.0.0.1:6579@16579 master - 0 1620206876405 3 connected 10923-16383
d47e315c1bab0bfa9a7b4dc663cea391c1d90def 127.0.0.1:7479@17479 slave 10022df5d01fddeb7a12e483ae971784bcbd72ba 0 1620206875000 2 connected
127.0.0.1:6379>

위와 같이 데이터는 Master Node 3대에만 저장되는 것을 확인할 수 있다.

[root@ip-192-168-106-237 utils]# redis-cli -c -p 6379
127.0.0.1:6379> set a a
-> Redirected to slot [15495] located at 127.0.0.1:6579
OK
127.0.0.1:6579> set b b
-> Redirected to slot [3300] located at 127.0.0.1:6379
OK
127.0.0.1:6379> set c c
-> Redirected to slot [7365] located at 127.0.0.1:6479
OK
127.0.0.1:6479> set e e 
-> Redirected to slot [15363] located at 127.0.0.1:6579
OK
127.0.0.1:6579> set f f
-> Redirected to slot [3168] located at 127.0.0.1:6379
OK
127.0.0.1:6379> set g g
-> Redirected to slot [7233] located at 127.0.0.1:6479
OK
127.0.0.1:6479> keys *
1) "c"
2) "g"
127.0.0.1:6479>

특히 Cluster에 포함되어 있는 Master Node 들에 각자 저장된 key와 data는 각 노드에 저장된 값만 확인이 가능하여 반드시 Slave를 구성하여 Master 데이터를 백업할 수 있도록 구성해야 한다. 위와 같이 6479 node에 저장된 c와 g의 key만 조회되는 것을 확인할 수 있다.

여기서 Master Node 중 한대를 다운시켜보도록 하자.

[root@ip-192-168-73-84 6379]# service redis_6379 stop
Stopping ...
Redis stopped
[root@ip-192-168-73-84 6379]# redis-cli -c -p 6479
127.0.0.1:6479> cluster nodes
d47e315c1bab0bfa9a7b4dc663cea391c1d90def 127.0.0.1:7479@17479 slave 10022df5d01fddeb7a12e483ae971784bcbd72ba 0 1620206975167 2 connected
6d559070ff6dbef46431d985397fbdd2ab0c6842 127.0.0.1:7379@17379 slave 4d778457522d238b0613b96e656c611b0405f2a3 0 1620206976171 1 connected
cde730ed40c71395be543dffd3e893c66357d04e 127.0.0.1:6579@16579 master - 0 1620206973000 3 connected 10923-16383
579851e6d3a9a8db16b2fec5ef5418dfd9f7a7f3 127.0.0.1:7579@17579 slave cde730ed40c71395be543dffd3e893c66357d04e 0 1620206974163 3 connected
4d778457522d238b0613b96e656c611b0405f2a3 127.0.0.1:6379@16379 master - 1620206963732 1620206962130 1 disconnected 0-5460
10022df5d01fddeb7a12e483ae971784bcbd72ba 127.0.0.1:6479@16479 myself,master - 0 1620206974000 2 connected 5461-10922
127.0.0.1:6479> set a a
-> Redirected to slot [15495] located at 127.0.0.1:6579
OK
127.0.0.1:6579> set b b
-> Redirected to slot [3300] located at 127.0.0.1:7379
OK
127.0.0.1:7379> set c c
-> Redirected to slot [7365] located at 127.0.0.1:6479
OK
127.0.0.1:6479>

위와 같이 Master Node의 Slave가 존재하기 때문에 기존 6379에 저장되어 있던 b라는 key는 7379에서 관리되는 것을 확인할 수 있다.

물론 7379 서버까지 다운된 경우에는 데이터를 조회할 수 없다.

[root@ip-192-168-73-84 6379]# service redis_7379 stop
Stopping ...
Redis stopped
[root@ip-192-168-73-84 6379]# redis-cli -c -p 6479
127.0.0.1:6479> get a
-> Redirected to slot [15495] located at 127.0.0.1:6579
"a"
127.0.0.1:6579> get b
-> Redirected to slot [3300] located at 127.0.0.1:7379
Could not connect to Redis at 127.0.0.1:7379: Connection refused
Could not connect to Redis at 127.0.0.1:7379: Connection refused
not connected> 

물론 서버 장애가 해결되어 기동이 완료될 경우 해당 데이터는 복구된다. (Redis 저장 방식에 따라 유실될 수도 있다. RDB vs AOF)

마지막으로 장애 복구 과정에 대해 살펴보자.

[root@ip-192-168-73-84 6379]# service redis_7379 start
Starting Redis server...
[root@ip-192-168-73-84 6379]# redis-cli -c -p 6479
127.0.0.1:6479> cluster nodes
d47e315c1bab0bfa9a7b4dc663cea391c1d90def 127.0.0.1:7479@17479 slave 10022df5d01fddeb7a12e483ae971784bcbd72ba 0 1620207331000 2 connected
6d559070ff6dbef46431d985397fbdd2ab0c6842 127.0.0.1:7379@17379 master - 0 1620207331352 4 connected 0-5460
cde730ed40c71395be543dffd3e893c66357d04e 127.0.0.1:6579@16579 master - 0 1620207332356 3 connected 10923-16383
579851e6d3a9a8db16b2fec5ef5418dfd9f7a7f3 127.0.0.1:7579@17579 slave cde730ed40c71395be543dffd3e893c66357d04e 0 1620207329347 3 connected
4d778457522d238b0613b96e656c611b0405f2a3 127.0.0.1:6379@16379 master,fail - 1620206963732 1620206962130 1 disconnected
10022df5d01fddeb7a12e483ae971784bcbd72ba 127.0.0.1:6479@16479 myself,master - 0 1620207327000 2 connected 5461-10922
127.0.0.1:6479> 

먼저 Slave Server 였던 7379 server를 기동한 후 cluster 정보를 확인해 보면 다음과 같다. 6379 서버는 master, fail 상태이며, 7379 server가 master server로 승격되어 있는 것을 확인할 수 있다.

[root@ip-192-168-73-84 6379]# redis-cli -c -p 6379
127.0.0.1:6379> cluster nodes
d47e315c1bab0bfa9a7b4dc663cea391c1d90def 127.0.0.1:7479@17479 slave 10022df5d01fddeb7a12e483ae971784bcbd72ba 0 1620207895000 2 connected
cde730ed40c71395be543dffd3e893c66357d04e 127.0.0.1:6579@16579 master - 0 1620207895588 3 connected 10923-16383
579851e6d3a9a8db16b2fec5ef5418dfd9f7a7f3 127.0.0.1:7579@17579 slave cde730ed40c71395be543dffd3e893c66357d04e 0 1620207894000 3 connected
6d559070ff6dbef46431d985397fbdd2ab0c6842 127.0.0.1:7379@17379 master - 0 1620207892000 4 connected 0-5460
4d778457522d238b0613b96e656c611b0405f2a3 127.0.0.1:6379@16379 myself,slave 6d559070ff6dbef46431d985397fbdd2ab0c6842 0 1620207892000 1 connected
10022df5d01fddeb7a12e483ae971784bcbd72ba 127.0.0.1:6479@16479 master - 0 1620207893583 2 connected 5461-10922
127.0.0.1:6379> set a a
-> Redirected to slot [15495] located at 127.0.0.1:6579
OK
127.0.0.1:6579> set b b
-> Redirected to slot [3300] located at 127.0.0.1:7379
OK
127.0.0.1:7379> set c c
-> Redirected to slot [7365] located at 127.0.0.1:6479
OK
127.0.0.1:6479> 

다음으로 Master Server 였던 6379 server를 기동한 후 cluster 정보를 확인해 보면 다음과 같다. 6379 서버는 master에서 slave 서버로 변경되었으며, 7379 server가 master 서버의 상태를 유지하는 것을 알 수 있다.

# 참고

Redis 환경 정보

a. 기동/종료 : /etc/init.d/redis_6379 활용 (service redis_6379 stop / start / restart)

b. 설정파일 : /etc/redis/6379.conf

c. nodes.conf : /var/lib/redis/6379/nodes.conf

d. 로그파일 : /var/lib/redis_6379.log