(二)redis docker 哨兵模式

发表于

Redis主从复制的原理及相关问题

  1. 保存主节点信息
  2. 主从建立socket连接
  3. 发送ping命令
  4. 权限验证
  5. 同步数据集
  6. 命令持续复制

全量复制

部分复制

redis2.8以后才支持

复制偏移量

1
2
3
info replication
slave0:ip=172.10.0.3,port=6379,state=online,offset=5545,lag=0
master_repl_offset

积压缓存区

dockerfile 指令

COPY
ADD
CMD
ENTRYPOINT
ENV
ARG
VOLUME
EXPOSE
WORKDIR
USER

主从复制的常见问题

repl-disable-tcp-nodelay
yes 时 TCP协议会合并小包统一发送，减少主从节点的包数量并节省带宽，会增加数据同步到slave的时间

slave-priority
指定slave的优先级。在不止1个slave的情况下，master宕机时，redis sentinel 会将priority值最小的slave提升为master。需要注意的是，若该配置项为0，则对应的slave永远不会被redis sentinel 自动提升为master

redis sentinel 哨兵的作用

  1. 监控：定期检查Redis数据节点、其余sentinel节点是否可达
  2. 通知：sentinel 节点会将故障转移的结果通知给应用方
  3. 主节点故障转移：实现从节点晋升为主节点并维护后续正确的主从关系
  4. sentinel节点本身就是独立的Redis节点，只不过她们有一些特殊，自己不存储数据，只支持部分命令

哨兵配置

新增一个Redis-slave2

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
[root@VM_0_6_centos ~]# docker run -itd --name redis-slave2 --net mynetwork -p 6382:6379 --ip 172.10.0.4 redis
d8ceb85aa3bedacad69e0765e5dcac28a4906eeb99c2f22b4e3af2bd3bb29501
[root@VM_0_6_centos ~]# docker exec -it redis-slave2 bash
[root@d8ceb85aa3be /]# vi /etc/redis.conf

...
# JUST COMMENT THE FOLLOWING LINE.
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#bind 127.0.0.1
bind 0.0.0.0
# Protected mode is a layer of security protection, in order to avoid that
# Redis instances left open on the internet are accessed and exploited.
#
# When protected mode is on and if:
#
# 1) The server is not binding explicitly to a set of addresses using the
#    "bind" directive.
# 2) No password is configured.
#
# The server only accepts connections from clients connecting from the
# IPv4 and IPv6 loopback addresses 127.0.0.1 and ::1, and from Unix domain
# sockets.
#
# By default protected mode is enabled. You should disable it only if
# you are sure you want clients from other hosts to connect to Redis
# even if no authentication is configured, nor a specific set of interfaces
# are explicitly listed using the "bind" directive.
#protected-mode yes
protected-mode no

...

# replicaof <masterip> <masterport>
replicaof 172.10.0.2 6379
# If the master is password protected (using the "requirepass" configuration
# directive below) it is possible to tell the replica to authenticate before
# starting the replication synchronization process, otherwise the master will
# refuse the replica request.
#
# masterauth <master-password>

...

[root@d8ceb85aa3be /]# redis-server /etc/redis.conf &
[1] 29
[root@d8ceb85aa3be /]# redis-cli
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:172.10.0.2
master_port:6379
master_link_status:up
master_last_io_seconds_ago:3
master_sync_in_progress:0
slave_repl_offset:101921
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:73aa73a6ca2674e214b2081989f8f5cd642b1774
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:101921
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:101880
repl_backlog_histlen:42
127.0.0.1:6379>

配置sentinel集群

核心配置

sentinel monitor mymaster 127.0.0.1 7000 2

监控的主节点的名字、IP和端口，最后一个2的意思是有几台 sentinel发现问题，就会发生故障转移，例如 配置为2，代表至少有2个sentinel节点认为主节点不可达，那么这个不可达的判断才是客观的。对于设置的越小，那么达到下线的条件越宽松，反之越严格。一般建议将其设置为sentinel节点的一半加1

sentinel down-after-minllsenconds mymaster 30000

这个是超时的时间（单位是毫秒）。当你去ping一个机器的时候，多长时间后仍ping不通，那么就热为它是有问题的

sentinel parallel-syncs mymaster 1

parallel-syncs就是用来限制在一次故障转移之后，每次向新的主节点发起复制操作的从节点个数，支出sentinel属于并发还是创新，1代表每次只能复制一个，可以减轻master的压力

sentinel auth-pass <master-name> <password>

如果sentinel监控的主节点配置了密码，sentinel auth-pass配置通过添加主节点的密码，房子sentinel节点对主节点无法监控

sentinel failover-timeout mymaster 180000

标识故障转移的时间

1
2
3
docker run -itd --name redis-sentinel1 --net mynetwork -p 22530:6379 --ip 172.10.0.30 redis
docker run -itd --name redis-sentinel2 --net mynetwork -p 22531:6379 --ip 172.10.0.31 redis
docker run -itd --name redis-sentinel3 --net mynetwork -p 22532:6379 --ip 172.10.0.32 redis

操作记录

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
[root@VM_0_6_centos ~]# docker run -itd --name redis-sentinel1 --net mynetwork -p 22530:6379 --ip 172.10.0.30 redis
706f81a08f15c74581b5f54a351ac809f3d782e7ee96d4f51b4ff745c8520039
[root@VM_0_6_centos ~]# docker exec -it redis-sentinel1 bash
[root@706f81a08f15 /]# vi /etc/redis-sentinel.conf

...
# bind 127.0.0.1 192.168.1.1
#
# protected-mode no
bind 0.0.0.0
protected-mode no
# port <sentinel-port>
# The port that this sentinel instance will run on
port 26379

...

# Note: master name should not include special characters or spaces.
# The valid charset is A-z 0-9 and the three characters ".-_".
#sentinel monitor mymaster 127.0.0.1 6379 2
sentinel monitor mymaster 172.10.0.2 6379 2
# sentinel auth-pass <master-name> <password>
#
...

[root@706f81a08f15 /]# redis-sentinel /etc/redis-sentinel.conf &

配置日志查看

1
2
3
4
5
6
7
8
9
10
11
12
[root@706f81a08f15 /]# vi /var/log/redis/sentinel.log
29:X 23 Nov 2019 04:29:30.667 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
29:X 23 Nov 2019 04:29:30.667 # Redis version=5.0.3, bits=64, commit=00000000, modified=0, pid=29, just started
29:X 23 Nov 2019 04:29:30.667 # Configuration loaded
29:X 23 Nov 2019 04:29:30.670 * Running mode=sentinel, port=26379.
29:X 23 Nov 2019 04:29:30.670 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
29:X 23 Nov 2019 04:29:30.678 # Sentinel ID is 7475415502d8812fb642905c4837a8e071f7a75a
29:X 23 Nov 2019 04:29:30.678 # +monitor master mymaster 172.10.0.2 6379 quorum 2
29:X 23 Nov 2019 04:29:30.766 * +slave slave 172.10.0.3:6379 172.10.0.3 6379 @ mymaster 172.10.0.2 6379
29:X 23 Nov 2019 04:29:30.772 * +slave slave 172.10.0.4:6379 172.10.0.4 6379 @ mymaster 172.10.0.2 6379
29:X 23 Nov 2019 04:29:37.290 * +sentinel sentinel 9e58d80dbc53015c666e1f3c5dba5146ddf56795 172.10.0.31 26379 @ mymaster 172.10.0.2 6379
29:X 23 Nov 2019 04:29:39.245 * +sentinel sentinel acf0b15e88a1734fdc1af5213817c6ac83aadb78 172.10.0.32 26379 @ mymaster 172.10.0.2 6379

备注：其它两台的配置一模一样，参数都不需要改变。哨兵集群只需要配置对主节点的监控即可。重启后0.2新的从节点会和新的0.4主节点建立连接

配置查看

登录查看

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
[root@706f81a08f15 /]# redis-cli -p 26379
127.0.0.1:26379> info
# Server
redis_version:5.0.3
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:8c0bf22bfba82c8f
redis_mode:sentinel
os:Linux 3.10.0-862.11.6.el7.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:8.2.1
process_id:29
run_id:efcfa986c11fb8a4815bba1db1caee13dcb60bae
tcp_port:26379
uptime_in_seconds:4473
uptime_in_days:0
hz:17
configured_hz:10
lru_clock:14206755
executable:/redis-sentinel
config_file:/etc/redis-sentinel.conf

# Clients
connected_clients:3
client_recent_max_input_buffer:2
client_recent_max_output_buffer:0
blocked_clients:0

# CPU
used_cpu_sys:8.485302
used_cpu_user:4.427473
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000

# Stats
total_connections_received:3
total_commands_processed:12991
instantaneous_ops_per_sec:2
total_net_input_bytes:716377
total_net_output_bytes:77880
instantaneous_input_kbps:0.16
instantaneous_output_kbps:0.01
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0

# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=172.10.0.4:6379,slaves=2,sentinels=3
127.0.0.1:26379> sentinel get-master-addr-by-name mymaster
1) "172.10.0.4"
2) "6379"

PHP通过哨兵去任意从库读取数据

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
<?php
/**
 * Created by PhpStorm.
 * User: uiste
 * Date: 2019/11/23
 * Time: 下午2:51
 */
$sentinelConf = [
	['ip' => '172.10.0.30', 'port' => 26379],
	['ip' => '172.10.0.31', 'port' => 26379],
	['ip' => '172.10.0.32', 'port' => 26379],
];
//随机访问
$sentinelInfo = $sentinelConf[array_rand($sentinelConf)];


$redis = new Redis();
$redis->connect($sentinelInfo['ip'], $sentinelInfo['port']);

$slavesInfo = $redis->rawCommand('SENTINEL', 'slaves', 'mymaster');
$slaves = [];
foreach ($slavesInfo as $val) {
	$slaves[] = ['ip' => $val[3], 'port' => $val[5]];
}

$count = 0;
while (true) {

	try {
		# 随机从任意从库读取数据
		$slave = $slaves[array_rand($slaves)];
		
		$redis = new Redis();
		$redis->connect($slave['ip'], $slave['port']);
		var_dump($slave, $redis->get('name'));

		$count++;
		echo "第{$count}次访问的是：[{$slave['ip']}] : [{$slave['port']}] \r\n";
		sleep(3);

	} catch (\RedisException $e) {
		echo $e->getMessage();
		die;
	}
}

debug info

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
[root@VM_0_6_centos test]# php test.php
array(2) {
  ["ip"]=>
  string(10) "172.10.0.2"
  ["port"]=>
  string(4) "6379"
}
string(5) "uiste"
第1次访问的是：[172.10.0.2] : [6379]
array(2) {
  ["ip"]=>
  string(10) "172.10.0.2"
  ["port"]=>
  string(4) "6379"
}
string(5) "uiste"
第2次访问的是：[172.10.0.2] : [6379]
array(2) {
  ["ip"]=>
  string(10) "172.10.0.3"
  ["port"]=>
  string(4) "6379"
}
string(5) "uiste"
第3次访问的是：[172.10.0.3] : [6379]
array(2) {
  ["ip"]=>
  string(10) "172.10.0.2"
  ["port"]=>
  string(4) "6379"
}
string(5) "uiste"
第4次访问的是：[172.10.0.2] : [6379]

debug $slavesInfo
array(2) {
  [0]=>
  array(40) {
    [0]=>
    string(4) "name"
    [1]=>
    string(15) "172.10.0.2:6379"
    [2]=>
    string(2) "ip"
    [3]=>
    string(10) "172.10.0.2"
    [4]=>
    string(4) "port"
    [5]=>
    string(4) "6379"
    [6]=>
    string(5) "runid"
    [7]=>
    string(40) "0e681802737476f1432e1bd21fa85b15d008892a"
    [8]=>
    string(5) "flags"
    [9]=>
    string(5) "slave"
    [10]=>
    string(21) "link-pending-commands"
    [11]=>
    string(1) "0"
    [12]=>
    string(13) "link-refcount"
    [13]=>
    string(1) "1"
    [14]=>
    string(14) "last-ping-sent"
    [15]=>
    string(1) "0"
    [16]=>
    string(18) "last-ok-ping-reply"
    [17]=>
    string(3) "490"
    [18]=>
    string(15) "last-ping-reply"
    [19]=>
    string(3) "490"
    [20]=>
    string(23) "down-after-milliseconds"
    [21]=>
    string(5) "30000"
    [22]=>
    string(12) "info-refresh"
    [23]=>
    string(4) "5825"
    [24]=>
    string(13) "role-reported"
    [25]=>
    string(5) "slave"
    [26]=>
    string(18) "role-reported-time"
    [27]=>
    string(7) "9292293"
    [28]=>
    string(21) "master-link-down-time"
    [29]=>
    string(1) "0"
    [30]=>
    string(18) "master-link-status"
    [31]=>
    string(2) "ok"
    [32]=>
    string(11) "master-host"
    [33]=>
    string(10) "172.10.0.4"
    [34]=>
    string(11) "master-port"
    [35]=>
    string(4) "6379"
    [36]=>
    string(14) "slave-priority"
    [37]=>
    string(3) "100"
    [38]=>
    string(17) "slave-repl-offset"
    [39]=>
    string(7) "2791574"
  }
  [1]=>
  array(40) {
    [0]=>
    string(4) "name"
    [1]=>
    string(15) "172.10.0.3:6379"
    [2]=>
    string(2) "ip"
    [3]=>
    string(10) "172.10.0.3"
    [4]=>
    string(4) "port"
    [5]=>
    string(4) "6379"
    [6]=>
    string(5) "runid"
    [7]=>
    string(40) "74c3a5ad5201ec08b04c34a6c1e8be43e3454f2f"
    [8]=>
    string(5) "flags"
    [9]=>
    string(5) "slave"
    [10]=>
    string(21) "link-pending-commands"
    [11]=>
    string(1) "0"
    [12]=>
    string(13) "link-refcount"
    [13]=>
    string(1) "1"
    [14]=>
    string(14) "last-ping-sent"
    [15]=>
    string(1) "0"
    [16]=>
    string(18) "last-ok-ping-reply"
    [17]=>
    string(3) "490"
    [18]=>
    string(15) "last-ping-reply"
    [19]=>
    string(3) "490"
    [20]=>
    string(23) "down-after-milliseconds"
    [21]=>
    string(5) "30000"
    [22]=>
    string(12) "info-refresh"
    [23]=>
    string(4) "2184"
    [24]=>
    string(13) "role-reported"
    [25]=>
    string(5) "slave"
    [26]=>
    string(18) "role-reported-time"
    [27]=>
    string(8) "13031310"
    [28]=>
    string(21) "master-link-down-time"
    [29]=>
    string(1) "0"
    [30]=>
    string(18) "master-link-status"
    [31]=>
    string(2) "ok"
    [32]=>
    string(11) "master-host"
    [33]=>
    string(10) "172.10.0.4"
    [34]=>
    string(11) "master-port"
    [35]=>
    string(4) "6379"
    [36]=>
    string(14) "slave-priority"
    [37]=>
    string(3) "100"
    [38]=>
    string(17) "slave-repl-offset"
    [39]=>
    string(7) "2792254"
  }
}