部署Redis Cluster 6.0 集群並開啟密碼認證 和 Redis-cluster-proxy負載

部署Redis Cluster集群並開啟密碼認證

如果只想簡單的搭建Redis Cluster,不需要設置密碼和公網訪問,可以參考官方文檔。

節點介紹

Cluster模式推薦最少有6個節點,本次實驗搭建了6個節點,使用的埠為7000-7005。Cluster模式是數據分開存放在不同的節點上,如果有6個節點,通常設置3個主節點,每個主節點有一個從節點作為備份。正常情況下讀寫操作的是主節點,從節點會同步主節點的更變。當某個主節點掛了之後,其對應的從節點會變成主節點。如果一段時間後之前掛掉的主節點恢復了,它將變成從節點。如果某個主節點掛了之後,其對應的從節點也掛了,集群將不可訪問。即每個主節點相互獨立,從節點作為主節點的備份。

下載和編譯

下載地址://redis.io/download
下載、解壓並編譯:

wget //download.redis.io/releases/redis-6.0.5.tar.gz
tar -zxvf redis-6.0.5.tar.gz
cd cd redis-6.0.5
make

坑:6.x版本的Redis需要新版的gcc才能編譯,centOS 7 自帶的gcc版本為4.8.5,無法完成編譯,需要升級版本。

解決方案:

yum install centos-release-scl
yum install devtoolset-9-gcc*

# 執行如下僅對當前bash生效
scl enable devtoolset-9 bash

# 以上操作僅對當bash生效,若想永久生效,按以下操作:
echo "source /opt/rh/devtoolset-9/enable" >>/etc/profile
source /etc/profile

gcc升級版本之後,需要重新make編譯,編譯完成後src目錄下會生成幾個可執行文件redis-benchmark redis-cli redis-sentinel redis-server等。

集群部署

新建6個目錄,並將redis.conf和src目錄中的redis-server複製到每個目錄,目錄名稱可隨意,這裡採用redis運行時的埠號作為目錄的名稱。

cd /data/soft/redis/

mkdir -p /data/soft/redis/conf
cp redis.conf /data/soft/redis/conf/7001_redis.conf
cp redis.conf /data/soft/redis/conf/7002_redis.conf
cp redis.conf /data/soft/redis/conf/7003_redis.conf
cp redis.conf /data/soft/redis/conf/7004_redis.conf
cp redis.conf /data/soft/redis/conf/7005_redis.conf
cp redis.conf /data/soft/redis/conf/7006_redis.conf

cp src/redis-server /data/bin/redis-server
cp src/redis-cli    /data/bin/redis-cli

修改配置

修改conf目錄中的*.conf,需要修改配置項如下:

# Redis configuration file.
# ./redis-server /path/to/redis.conf

################################## MODULES #####################################
# include /path/to/local.conf
# loadmodule /path/to/my_module.so

################################## NETWORK #####################################
# By default, if no "bind" configuration directive is specified, Redis listens
# for connections from all the network interfaces available on the server.
bind 192.168.0.239
protected-mode no
port 7005
tcp-backlog 2000
# unixsocket /tmp/redis.sock
# unixsocketperm 700
timeout 0
tcp-keepalive 300

################################# GENERAL #####################################
daemonize yes
supervised no
pidfile /log/redis/redis-7005.pid
# debug, verbose, notice, warning
loglevel notice
logfile /log/redis/redis-7005.log
# syslog-enabled no
# syslog-ident redis
# syslog-facility local0
databases 16
always-show-logo yes

################################ SNAPSHOTTING  ################################
# save <seconds> <changes>
save 600 1
save 300 10
save 250 50
save 100 1000
save 80 5000
save 60 10000

stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump-7005.rdb
dir /log/redis/dump/

################################# REPLICATION #################################
# slaveof 192.168.53.5 7005
masterauth 123456
# slave-serve-stale-data yes
# slave-read-only yes
# repl-diskless-sync no
# repl-ping-slave-period 10
# repl-timeout 60
# repl-disable-tcp-nodelay no
# repl-backlog-size 5mb
# repl-backlog-ttl 3000
# slave-priority 100
# min-slaves-to-write 1
# min-slaves-max-lag 10
# slave-announce-ip 5.5.5.5
# slave-announce-port 1234

################################## SECURITY ###################################
requirepass 123456
# rename-command CONFIG XCONFIG

################################### CLIENTS ####################################
maxclients 10000

############################## MEMORY MANAGEMENT ################################
maxmemory 2048mb
# volatile-lru -> Evict using approximated LRU among the keys with an expire set.
# allkeys-lru -> Evict any key using approximated LRU.
# volatile-lfu -> Evict using approximated LFU among the keys with an expire set.
# allkeys-lfu -> Evict any key using approximated LFU.
# volatile-random -> Remove a random key among the ones with an expire set.
# allkeys-random -> Remove a random key, any key.
# volatile-ttl -> Remove the key with the nearest expire time (minor TTL)
# noeviction -> Don't evict anything, just return an error on write operations.
maxmemory-policy allkeys-lru
maxmemory-samples 5

############################# LAZY FREEING ####################################
# release memory in a non-blocking. DEL, UNLINK and ASYNC
# option of FLUSHALL and FLUSHDB are user-controlled.
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
slave-lazy-flush no

############################## APPEND ONLY MODE ###############################
appendonly yes
appendfilename appendonly-7005.aof
# always|everysec|no
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 3mb
aof-load-truncated yes
aof-use-rdb-preamble no

################################ LUA SCRIPTING  ###############################
lua-time-limit 5000

################################ REDIS CLUSTER  ###############################
cluster-enabled yes
cluster-config-file /log/redis/nodes-7005.conf
cluster-node-timeout 15000
cluster-slave-validity-factor 10
cluster-migration-barrier 1
cluster-require-full-coverage yes

################################## SLOW LOG ###################################
slowlog-log-slower-than 5000
slowlog-max-len 516

################################ LATENCY MONITOR ##############################
latency-monitor-threshold 0

############################# EVENT NOTIFICATION ##############################
notify-keyspace-events ""

############################### ADVANCED CONFIG ###############################
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
# -5: max size: 64 Kb  <-- not recommended for normal workloads
# -4: max size: 32 Kb  <-- not recommended
# -3: max size: 16 Kb  <-- probably not recommended
# -2: max size: 8 Kb   <-- good
# -1: max size: 4 Kb   <-- good
list-max-ziplist-size -2
# 0|1|2|3
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
# normal -> normal clients including MONITOR clients
# slave  -> slave clients
# pubsub -> clients subscribed to at least one pubsub channel or pattern
# client-output-buffer-limit <class> <hard limit> <soft limit> <soft seconds>
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
# client-query-buffer-limit 1gb
# proto-max-bulk-len 512mb
hz 10
aof-rewrite-incremental-fsync yes
# lfu-log-factor 10
# lfu-decay-time 1

########################### ACTIVE DEFRAGMENTATION #######################
# Enabled active defragmentation
# activedefrag yes

# Minimum amount of fragmentation waste to start active defrag
# active-defrag-ignore-bytes 100mb

# Minimum percentage of fragmentation to start active defrag
# active-defrag-threshold-lower 10

# Maximum percentage of fragmentation at which we use maximum effort
# active-defrag-threshold-upper 100

# Minimal effort for defrag in CPU percentage
# active-defrag-cycle-min 25

# Maximal effort for defrag in CPU percentage
# active-defrag-cycle-max 75

上面的配置文件已經列出,當前為7005的埠,其他埠的配置文件都類似,改下埠既可。

創建集群

啟動每個節點:

/data/soft/redis/src/redis-check-aof --fix /log/redis/dump/appendonly-7001.aof

/data/soft/redis/src/redis-check-aof --fix /log/redis/dump/appendonly-7002.aof

/data/soft/redis/src/redis-check-aof --fix /log/redis/dump/appendonly-7003.aof

/data/soft/redis/src/redis-check-aof --fix /log/redis/dump/appendonly-7004.aof

/data/soft/redis/src/redis-check-aof --fix /log/redis/dump/appendonly-7005.aof

/data/soft/redis/src/redis-check-aof --fix /log/redis/dump/appendonly-7006.aof

/data/bin/redis-server /data/soft/redis/conf/redis-7001.conf >/dev/null 2>&1 &

/data/bin/redis-server /data/soft/redis/conf/redis-7002.conf >/dev/null 2>&1 &

/data/bin/redis-server /data/soft/redis/conf/redis-7003.conf >/dev/null 2>&1 &

/data/bin/redis-server /data/soft/redis/conf/redis-7004.conf >/dev/null 2>&1 &

/data/bin/redis-server /data/soft/redis/conf/redis-7005.conf >/dev/null 2>&1 &

/data/bin/redis-server /data/soft/redis/conf/redis-7006.conf >/dev/null 2>&1 &

啟動集群:

/data/bin/redis-cli --cluster create ip:7000 ip:7001 ip:7002 ip:7003 ip:7004 ip:7005 --cluster-replicas 1 -a 123456

Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.0.239:7006 to 192.168.0.239:7001
Adding replica 192.168.0.239:7003 to 192.168.0.239:7004
Adding replica 192.168.0.239:7005 to 192.168.0.239:7002
M: 4c8fd6640bac7463a517e10fda942568177f1175 192.168.0.239:7001
   slots:[0-5460] (5461 slots) master
M: 8a924d45c5094bca66c736661f267de3ff348134 192.168.0.239:7002
   slots:[10923-16383] (5461 slots) master
S: 9e6a1995558a142002579a1bf2659b0fff6a986b 192.168.0.239:7003
   replicates b8985dcd19735b50a9b13ba8ef2449504b1da755
M: b8985dcd19735b50a9b13ba8ef2449504b1da755 192.168.0.239:7004
   slots:[5461-10922] (5462 slots) master
S: baf6fb8fa06e58e7ca541d4f9c76ee63255e797f 192.168.0.239:7005
   replicates 8a924d45c5094bca66c736661f267de3ff348134
S: 10993fd7b1cdf5d0293373bec821bdb7763426ed 192.168.0.239:7006
   replicates 4c8fd6640bac7463a517e10fda942568177f1175
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
...
>>> Performing Cluster Check (using node 192.168.0.239:7001)
M: 4c8fd6640bac7463a517e10fda942568177f1175 192.168.0.239:7001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 9e6a1995558a142002579a1bf2659b0fff6a986b 192.168.0.239:7003
   slots: (0 slots) slave
   replicates b8985dcd19735b50a9b13ba8ef2449504b1da755
S: 10993fd7b1cdf5d0293373bec821bdb7763426ed 192.168.0.239:7006
   slots: (0 slots) slave
   replicates 4c8fd6640bac7463a517e10fda942568177f1175
M: 8a924d45c5094bca66c736661f267de3ff348134 192.168.0.239:7002
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: baf6fb8fa06e58e7ca541d4f9c76ee63255e797f 192.168.0.239:7005
   slots: (0 slots) slave
   replicates 8a924d45c5094bca66c736661f267de3ff348134
M: b8985dcd19735b50a9b13ba8ef2449504b1da755 192.168.0.239:7004
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.


這裡的IP可以是公網IP也可以是內網IP,-a後面的參數是設置的訪問密碼,–cluster-replicas後面的參數是每個主節點對應的從節點的數量。
這時會自動分配主從節點,如果確認分配無誤,輸入yes即可創建cluster。

坑:在cluster模式中,每個節點除了要使用設置的訪問埠以外,還需要使用訪問埠加10000的埠號進行數據傳輸,所以在防火牆開放埠時也需要打開對應的埠,否則會出現wait…的提示。
在本例中,需要開放的埠號為7001-7006和17001-17006。

測試集群:

可以使用redis-cli來連接集群,-c代表使用cluster模式。

/data/soft/redis-cli -c -h ip -p 7000 -a 密碼

查看集群信息:

192.168.0.239:7001> cluster info 
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:107
cluster_stats_messages_pong_sent:111
cluster_stats_messages_sent:218
cluster_stats_messages_ping_received:106
cluster_stats_messages_pong_received:107
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:218

測試節點讀寫信息:

192.168.0.239:7001> set foo bar
-> Redirected to slot [12182] located at 192.168.0.239:7002
OK
192.168.0.239:7002> get foo
"bar"

設置公網訪問

現在通過公網訪問會出現unknown cluster ip:7000的報錯信息,這是因為在nodes.conf文件中某些節點的IP是內網IP。解決方法為如下:

  • 停止每個節點的進程。
  • 編輯每個節點的nodes.conf文件,將其中的內網IP換成公網IP。
  • 重新啟動每個節點。

注意:設置公網訪問Redis集群屬於危險行為,一般生產環境進位使用公網redis集群

配置Redis-cluster-proxy

以下信息來自於官方的說明:

redis-cluster-proxy是Redis集群的代理。Redis能夠在基於自動故障轉移和分片的集群模式下運行。
這種特殊模式(指Redis集群模式)需要使用特殊的客戶端來理解集群協議:通過代理,集群被抽象了出來,可以實現像單實例一樣實現redis集群的訪問。

Redis集群代理是多執行緒的,默認情況下,它目前使用多路復用通訊模型,這樣每個執行緒都有自己的集群連接,所有屬於執行緒本身的客戶端都可以共享該連接。

無論如何,在某些特殊情況下(多事務或阻塞命令),多路復用被禁用,客戶端將擁有自己的集群連接。
通過這種方式,只發送簡單命令(比如get和set)的客戶端將不需要一組到Redis集群的私有連接。

註:依賴 gcc 4.9 以上版本,所以需要升級gcc版本,前面已經寫過了。

下載proxy工具

//github.com/RedisLabs/redis-cluster-proxy/releases

wget //github.com/RedisLabs/redis-cluster-proxy/releases/redis-cluster-proxy-1.0-beta2.tar.gz

tar -zxvf redis-cluster-proxy-1.0-beta2.tar.gz

mv redis-cluster-proxy-1.0-beta2 /data/soft/redis-cluster-proxy

cd /data/soft/redis-cluster-proxy/ &&  make install

修改配置文件

修改/data/soft/redis-cluster-proxy目錄下proxy.conf文件,該配置文件和redis配置文件相似,具體參數還未整理,後續更新文檔

重點修改以下配置:

  • 配置監聽的集群節點信息

  • 可修改代理埠,默認7777

  • 可修改執行緒數,因6.0後是多執行緒

  • 如果後端redis集群配置了密碼,需要在auth參數增加後端redis密碼

配置文件如下:

# Redis Cluster Proxy configuration file example.
#
# Note that in order to read the configuration file, the proxy must be
# started with the file path passed to the -c option:
#
# ./redis-cluster-proxy -c /path/to/proxy.conf

################################## INCLUDES ###################################

# Include one or more other config files here.  Include files can include
# other files.
#
# If instead you are interested in using includes to override configuration
# options, it is better to use include as the last line.
#
# include /path/to/local.conf
# include /path/to/other.conf

######################## CLUSTER ENTRY POINT ADDRESS ##########################

# Indicate the entry point address in the same way it can be indicated in the
# redis-cluster-proxy command line arguments.
# Note that it can be overridden by the command line argument itself.
# You can also specify multiple entry-points, by adding more lines, ie:
# cluster 127.0.0.1:7000
# cluster 127.0.0.1:7001
# You can also use the "entry-point" alias instead of cluster, ie:
# entry-point 127.0.0.1:7000
#
cluster 192.168.0.239:7001 
cluster 192.168.0.239:7002 
cluster 192.168.0.239:7003 
cluster 192.168.0.239:7004 
cluster 192.168.0.239:7005 
cluster 192.168.0.239:7006

################################### MAIN ######################################

# Set the port used by Redis Cluster Proxy to listen to incoming connections
# from clients (default 7777)
port 7000

# If you want you can bind a single interface, if the bind option is not
# specified all the interfaces will listen for incoming connections.
# You can also bind on multiple interfaces by declaring bind on multiple lines
#
# bind 127.0.0.1

# Specify the path for the Unix socket that will be used to listen for
# incoming connections. There is no default, so Redis Cluster Proxy won't
# listen on a Unix socket when not specified.
#
# unixsocket /path/to/proxy.socket

# Set the Unix socket file permissions (default 0)
#
# unixsocketperm 760

# Set the number of threads.
threads 8

# Set the TCP keep-alive value on the Redis Cluster Proxy's socket
#
# tcpkeepalive 300

# Set the TCP backlog on the Redis Cluster Proxy's socket
#
# tcp-backlog 511


# Size of the connections pool used to provide ready-to-use sockets to
# private connections. The number (size) indicates the number of starting
# connections in the pool.
# Use 0 to disable connections pool at all.
# Every thread will have its pool of ready-to-use connections.
# When the proxy starts, every thread will populate a pool containing
# connections to all the nodes of the cluster.
# Whenever a client needs a private connection, it can take a connection
# from the pool, if available. This will speed-up the client transition from
# the thread's shared connection to its own private connection, since the
# connection from the thread's pool should be already connected and
# ready-to-use. Otherwise, clients with priovate connections must re-connect
# the the nodes of the cluster (this re-connection will act in a 'lazy' way).
#
# connections-pool-size 10

# Minimum number of connections in the the pool. Below this value, the
# thread will start re-spawning connections at the defined rate until
# the pool will be full again.
#
# connections-pool-min-size 10

# Interval in milliseconds used to re-spawn connections in the pool.
# Whenever the number of connections in the pool drops below the minimum
# (see 'connections-pool-min-size' above), the thread will start
# re-spawing connections in the pool, until the pool will be full again.
# New connections will be added at this specified interval.
#
# connections-pool-spawn-every 50

# Number of connections to re-spawn in the pool at every cycle that will
# happen with an interval defined by 'connections-pool-spawn-every' (see above).
#
# connections-pool-spawn-rate 50

# Run Redis Cluster Proxy as a daemon.
daemonize no

# If a pid file is specified, the proxy writes it where specified at startup
# and removes it at exit.
#
# When the proxy runs non daemonized, no pid file is created if none is
# specified in the configuration. When the proxy is daemonized, the pid file
# is used even if not specified, defaulting to
# "/var/run/redis-cluster-proxy.pid".
#
# Creating a pid file is best effort: if the proxy is not able to create it
# nothing bad happens, the server will start and run normally.
#
pidfile /log/redis/redis-cluster-proxy.pid

# Specify the log file name. Also the empty string can be used to force
# Redis Cluster Porxy to log on the standard output. Note that if you use
# standard output for logging but daemonize, logs will be sent to /dev/null
#
logfile "/log/redis/redis-cluster-proxy.log"

# Enable cross-slot queries that can use multiple keys belonging to different
# slots or even different nodes.
# WARN: these queries will break the the atomicity deisgn of many Redis
# commands.
# NOTE: cross-slots queries are not supported by all the commands, even if
# this feature is enabled
#
# enable-cross-slot no

# Maximum number of clients allowed
#
# max-clients 10000

# Authentication password used to authenticate on the cluster in case its nodes
# are password-protected. The password will be used both for fetching cluster's
# configuration and to automatically authenticate proxy's internal connections
# to the cluster itself (both multiplexing shared connections and clients'
# private connections. So, clients connected to the proxy won't need to issue
# the Redis AUTH command in order to be authenticated.
#
auth nfJNhCDwv0hOlyFdbT9XvlkdZq 

# Authentication username (supported by Redis >= 6.0)
#
# auth-user myuser

################################# LOGGING #####################################

# Log level: can be debug, info, success, warning o error.
log-level error

# Dump queries received from clients in the log (log-level debug required)
#
# dump-queries no

# Dump buffer in the log (log-level debug required)
#
# dump-buffer no

# Dump requests' queues (requests to send to cluster, request pending, ...)
# in the log (log-level debug required)
#
# dump-queues no


啟動redis-cluster-proxy

第一次先當前控制台啟動,可以看下錯誤輸出

/data/bin/redis-cluster-proxy -c /usr/local/redis-cluster-proxy/proxy.conf

第二次用後台啟動

/data/bin/redis-cluster-proxy --daemonize -c /usr/local/redis-cluster-proxy/proxy.conf

啟動成功後,可以通過ps -ef 查看是否啟動

ps -ef|grep redis

然後通過redis客戶端連接7000埠(proxy代理埠)

/data/bin/redis-cli -h 127.0.0.1 -p 7000 -a 123456
127.0.0.1:7000> set foo bar1 
OK
127.0.0.1:7000> get foo
"bar1"

redis-cluster-proxy是完美的解決方案?

因為剛推出不久,生產環境基本上不會有太多實際的應用,裡面肯定有不少坑,但不妨害對其有更多的期待。
初次嘗試可以感受的到,redis-cluster-proxy是一個非常輕量級,清爽簡單的proxy代理層,它解決了一些redis cluster存在的一些實際問題,對應於程式來說也帶來了一些方便性。

如果沒有源碼開發能力,相比其他第三方proxy中間件,必須要承認官方可靠性和權威性。

那麼,redis-cluster-proxy是一個完美的解決方案么,留下兩個問題

  1. 如何解決redis-cluster-proxy單點故障?

  2. proxy節點的如何面對網路流量風暴?

Tags: