MHA高可用部署及测试
MHA高可用部署及测试
MHA高可用部署及测试,MySQL高可用解决方案
目录
一.MHA特性
二.MHA工作机制
三.MHA适用的主从架构
四.MHA高可用环境的构建
4.1 实验环境
4.2 实验大概步骤
4.3 脚本相关说明
4.4 MHA部署过程
4.5 配置VIP的方式
五.MHA常用命令
六.注意事项
七.部署过程遇到的问题
一.MHA特性
1.主服务器的自动监控和故障转移
MHA监控复制架构的主服务器，一旦检测到主服务器故障，就会自动进行故障转移。即使有些从服务器没有收到最新的relay log，MHA自动从最新的从服务器上识别差异的relay log并把这些日志应用到其他从服务器上，因此所有的从服务器保持一致性了。MHA通常在几秒内完成故障转移，9-12秒可以检测出主服务器故障，7-10秒内关闭故障的主服务器以避免脑裂，几秒中内应用差异的relay log到新的主服务器上，整个过程可以在10-30s内完成。还可以设置优先级指定其中的一台slave作为master的候选人。由于MHA在slaves之间修复一致性，因此可以将任何slave变成新的master，而不会发生一致性的问题，从而导致复制失败。
2.交互式主服务器故障转移
可以只使用MHA的故障转移，而不用于监控主服务器，当主服务器故障时，人工调用MHA来进行故障故障。
3.非交互式的主故障转移
不监控主服务器，但自动实现故障转移。这种特征适用于已经使用其他软件来监控主服务器状态，比如heartbeat来检测主服务器故障和虚拟IP地址接管，可以使用MHA来实现故障转移和slave服务器晋级为master服务器。
4.在线切换主服务器
在许多情况下，需要将现有的主服务器迁移到另外一台服务器上。比如主服务器硬件故障，RAID控制卡需要重建，将主服务器移到性能更好的服务器上等等。维护主服务器引起性能下降，导致停机时间至少无法写入数据。另外，阻塞或杀掉当前运行的会话会导致主主之间数据不一致的问题发生。MHA提供快速切换和优雅的阻塞写入，这个切换过程只需要0.5-2s的时间，这段时间内数据是无法写入的。在很多情况下，0.5-2s的阻塞写入是可以接受的。因此切换主服务器不需要计划分配维护时间窗口(呵呵，不需要你在夜黑风高时通宵达旦完成切换主服务器的任务)。
二.MHA工作机制
MHA自动Failover过程解析
http://www.mysqlsystems.com/2012/03/figure-out-process-of-autofailover-on-mha.html
https://code.google.com/p/mysql-master-ha/wiki/Sequences_of_MHA
三.MHA适用的主从架构
https://code.google.com/p/mysql-master-ha/wiki/UseCases
四.MHA高可用环境的构建
4.1 实验环境
•Node1:192.168.10.216 (主)
•Node2:192.168.10.217 (从,主故障切换的备主)
•Node3:192.168.10.218 (从,兼MHA管理节点)
•VIP : 192.168.10.219
•Mysql:Percona-Server-5.6.16-rel64.2-569
•以上节点系统均为CentOS6.5 x64
4.2 实验大概步骤
A. 三节点配置epel的yum源，安装相关依赖包
B. 建立主从复制关系
C. ssh-keygen实现三台机器之间相互免密钥登录
D. 三节点安装mha4mysql-node-0.56,node3上安装mha4mysql-manager-0.56
E. 在node3上管理MHA配置文件
F. masterha_check_ssh验证ssh信任登录是否成功,masterha_check_repl验证mysql复制是否成功
G. 启动MHA manager，并监控日志文件
H. 测试master(Node1)的mysql宕掉后，是否会自动切换正常
I . 配置VIP，切换后从自动接管主服务，并对客户端透明
4.3 脚本相关说明
MHA node有三个脚本，依赖perl模块
save_binary_logs：保存和拷贝宕掉的主服务器二进制日志
apply_diff_relay_logs:识别差异的relay log事件，并应用到所有从服务器节点
purge_relay_logs:清除relay log日志文件
4.4 MHA部署过程
A.三节点配置epel的yum源，安装相关依赖包
rpm -Uvh http://dl.Fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
yum -y install perl-DBD-MySQL ncftp
B. 建立主从复制关系
在node1上：
mysql>grant replication slave on *.* to 'rep'@'192.168.10.%' identified by 'geekwolf';
mysql>grant all on *.* to 'root'@'192.168.10.%' identified by 'geekwolf';
mysql>show master status;
拷贝node1的data目录同步到node2，node3
在node2 node3上：
mysql>change master to master_host='192.168.10.216', master_user='rep', master_password='geekwolf',master_port=3306, master_log_file='mysql-in.000006',master_log_pos=120,master_connect_retry=1;
mysql>start slave;
每个节点都做好mysql命令的软链
ln -s /usr/local/mysql/bin/* /usr/local/bin/
C. ssh-keygen实现三台机器之间相互免密钥登录
在node1(在其他两个节点一同)执行
ssh-keygen -t rsa
ssh-copy-id -i /root/.ssh/id_rsa.pub root@node1
ssh-copy-id -i /root/.ssh/id_rsa.pub root@node2
ssh-copy-id -i /root/.ssh/id_rsa.pub root@node3
D. 三节点安装mha4mysql-node-0.56,node3上安装mha4mysql-manager-0.56
在node1 node2 node3安装mha4mysql-node
wget https://googledrive.com/host/0B1lu97m8-haWeHdGWXp0YVVUSlk/mha4mysql-node-0.56.tar.gz
tar xf mha4mysql-node-0.56.tar.gz
cd mha4mysql-node
perl Makefile.PL
make && make install
在node3上安装mha4mysql-manager
wget https://googledrive.com/host/0B1lu97m8-haWeHdGWXp0YVVUSlk/mha4mysql-manager-0.56.tar.gz
tar xf mha4mysql-manager-0.56.tar.gz
cd mha4mysql-manager-0.56
yum install perl-DBD-MySQL perl-Config-Tiny perl-Log-Dispatch perl-Parallel-ForkManager perl-Config-IniFiles perl-Time-HiRes
E. 在node3上管理MHA配置文件
mkdir -p /etc/mha/{app1,scripts}
cp mha4mysql-manager-0.56/samples/conf/ /etc/mha/
cp mha4mysql-manager-0.56/samples/scripts/ /etc/mha/scripts/
mv /etc/mha/app1.cnf /etc/mha/app1/
mv /etc/mha/masterha_default.cnf /etc/masterha_default.cnf
设置全局配置：
vim /etc/mha/masterha_default.cnf
[server default]
user=root
password=geekwolf
ssh_user=root
repl_user=rep
repl_password=geekwolf
ping_interval=1
#shutdown_script=""
secondary_check_script = masterha_secondary_check -s node1 -s node2 -s node3 --user=root --master_host=node1 --master_ip=192.168.10.216 --master_port=3306
#master_ip_failover_script="/etc/mha/scripts/master_ip_failover"
#master_ip_online_change_script="/etc/mha/scripts/master_ip_online_change"
# shutdown_script= /script/masterha/power_manager
#report_script=""
vim /etc/mha/app1/app1.cnf
[server default]
manager_workdir=/var/log/mha/app1
manager_log=/var/log/mha/app1/manager.log
[server1]
hostname=node1
master_binlog_dir="/usr/local/mysql/logs"
candidate_master=1
[server2]
hostname=node2
master_binlog_dir="/usr/local/mysql/logs"
candidate_master=1
[server3]
hostname=node3
master_binlog_dir="/usr/local/mysql/logs"
no_master=1
注释：
candidate_master=1 表示该主机优先可被选为new master，当多个[serverX]等设置此参数时，优先级由[serverX]配置的顺序决定
secondary_check_script mha强烈建议有两个或多个网络线路检查MySQL主服务器的可用性。默认情况下,只有单一的路线 MHA Manager检查:从Manager to Master,但这是不可取的。MHA实际上可以有两个或两个以上的检查路线通过调用外部脚本定义二次检查脚本参数
master_ip_failover_script 在MySQL从服务器提升为新的主服务器时，调用此脚本，因此可以将vip信息写到此配置文件
master_ip_online_change_script 使用masterha_master_switch命令手动切换MySQL主服务器时后会调用此脚本，参数和master_ip_failover_script 类似，脚本可以互用
shutdown_script 此脚本(默认samples内的脚本)利用服务器的远程控制IDRAC等，使用ipmitool强制去关机，以避免fence设备重启主服务器，造成脑列现象
report_script 当新主服务器切换完成以后通过此脚本发送邮件报告，可参考使用 http://caspian.dotconf.net/menu/Software/SendEmail/sendEmail-v1.56.tar.gz
以上涉及到的脚本可以从mha4mysql-manager-0.56/samples/scripts/*拷贝进行修改使用
其他manager详细配置参数https://code.google.com/p/mysql-master-ha/wiki/Parameters
F. masterha_check_ssh验证ssh信任登录是否成功,masterha_check_repl验证mysql复制是否成功
验证ssh信任：masterha_check_ssh –conf=/etc/mha/app1/app1.cnf
[root@localhost ~]# masterha_check_ssh --conf=/etc/mha/app1/app1.cnf
Tue May 13 07:53:15 2014 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Tue May 13 07:53:15 2014 - [info] Reading application default configuration from /etc/mha/app1/app1.cnf..
Tue May 13 07:53:15 2014 - [info] Reading server configuration from /etc/mha/app1/app1.cnf..
Tue May 13 07:53:15 2014 - [info] Starting SSH connection tests..
Tue May 13 07:53:16 2014 - [debug]
Tue May 13 07:53:15 2014 - [debug] Connecting via SSH from root@node1(192.168.10.216:22) to root@node2(192.168.10.217:22)..
Tue May 13 07:53:15 2014 - [debug] ok.
Tue May 13 07:53:15 2014 - [debug] Connecting via SSH from root@node1(192.168.10.216:22) to root@node3(192.168.10.218:22)..
Tue May 13 07:53:16 2014 - [debug] ok.
Tue May 13 07:53:16 2014 - [debug]
Tue May 13 07:53:16 2014 - [debug] Connecting via SSH from root@node2(192.168.10.217:22) to root@node1(192.168.10.216:22)..
Tue May 13 07:53:16 2014 - [debug] ok.
Tue May 13 07:53:16 2014 - [debug] Connecting via SSH from root@node2(192.168.10.217:22) to root@node3(192.168.10.218:22)..
Tue May 13 07:53:16 2014 - [debug] ok.
Tue May 13 07:53:17 2014 - [debug]
Tue May 13 07:53:16 2014 - [debug] Connecting via SSH from root@node3(192.168.10.218:22) to root@node1(192.168.10.216:22)..
Tue May 13 07:53:16 2014 - [debug] ok.
Tue May 13 07:53:16 2014 - [debug] Connecting via SSH from root@node3(192.168.10.218:22) to root@node2(192.168.10.217:22)..
Tue May 13 07:53:17 2014 - [debug] ok.
Tue May 13 07:53:17 2014 - [info] All SSH connection tests passed successfully.
验证主从复制：masterha_check_repl –conf=/etc/mha/app1/app1.cnf
[root@localhost mha]# masterha_check_repl --conf=/etc/mha/app1/app1.cnf
Tue May 13 08:10:54 2014 - [info] Reading default configuration from /etc/masterha_default.cnf..
Tue May 13 08:10:54 2014 - [info] Reading application default configuration from /etc/mha/app1/app1.cnf..
Tue May 13 08:10:54 2014 - [info] Reading server configuration from /etc/mha/app1/app1.cnf..
Tue May 13 08:10:54 2014 - [info] MHA::MasterMonitor version 0.56.
Tue May 13 08:10:54 2014 - [info] GTID failover mode = 0
Tue May 13 08:10:54 2014 - [info] Dead Servers:
Tue May 13 08:10:54 2014 - [info] Alive Servers:
Tue May 13 08:10:54 2014 - [info] node1(192.168.10.216:3306)
Tue May 13 08:10:54 2014 - [info] node2(192.168.10.217:3306)
Tue May 13 08:10:54 2014 - [info] node3(192.168.10.218:3306)
Tue May 13 08:10:54 2014 - [info] Alive Slaves:
Tue May 13 08:10:54 2014 - [info] node2(192.168.10.217:3306) Version=5.6.16-64.2-rel64.2-log (oldest major version between slaves) log-bin:enabled
Tue May 13 08:10:54 2014 - [info] Replicating from 192.168.10.216(192.168.10.216:3306)
Tue May 13 08:10:54 2014 - [info] Primary candidate for the new Master (candidate_master is set)
Tue May 13 08:10:54 2014 - [info] node3(192.168.10.218:3306) Version=5.6.16-64.2-rel64.2-log (oldest major version between slaves) log-bin:enabled
Tue May 13 08:10:54 2014 - [info] Replicating from 192.168.10.216(192.168.10.216:3306)
Tue May 13 08:10:54 2014 - [info] Not candidate for the new Master (no_master is set)
Tue May 13 08:10:54 2014 - [info] Current Alive Master: node1(192.168.10.216:3306)
Tue May 13 08:10:54 2014 - [info] Checking slave configurations..
Tue May 13 08:10:54 2014 - [info] read_only=1 is not set on slave node2(192.168.10.217:3306).
Tue May 13 08:10:54 2014 - [warning] relay_log_purge=0 is not set on slave node2(192.168.10.217:3306).
Tue May 13 08:10:54 2014 - [info] read_only=1 is not set on slave node3(192.168.10.218:3306).
Tue May 13 08:10:54 2014 - [warning] relay_log_purge=0 is not set on slave node3(192.168.10.218:3306).
Tue May 13 08:10:54 2014 - [info] Checking replication filtering settings..
Tue May 13 08:10:54 2014 - [info] binlog_do_db= , binlog_ignore_db=
Tue May 13 08:10:54 2014 - [info] Replication filtering check ok.
Tue May 13 08:10:54 2014 - [info] GTID (with auto-pos) is not supported
Tue May 13 08:10:54 2014 - [info] Starting SSH connection tests..
Tue May 13 08:10:55 2014 - [info] All SSH connection tests passed successfully.
Tue May 13 08:10:55 2014 - [info] Checking MHA Node version..
Tue May 13 08:10:55 2014 - [info] Version check ok.
Tue May 13 08:10:55 2014 - [info] Checking SSH publickey authentication settings on the current master..
Tue May 13 08:10:56 2014 - [info] HealthCheck: SSH to node1 is reachable.
Tue May 13 08:10:56 2014 - [info] Master MHA Node version is 0.56.
Tue May 13 08:10:56 2014 - [info] Checking recovery script configurations on node1(192.168.10.216:3306)..
Tue May 13 08:10:56 2014 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/usr/local/mysql/logs --output_file=/var/tmp/save_binary_logs_test --manager_version=0.56 --start_file=mysql-bin.000009
Tue May 13 08:10:56 2014 - [info] Connecting to root@192.168.10.216(node1:22)..
Creating /var/tmp if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /usr/local/mysql/logs, up to mysql-bin.000009
Tue May 13 08:10:56 2014 - [info] Binlog setting check done.
Tue May 13 08:10:56 2014 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Tue May 13 08:10:56 2014 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=node2 --slave_ip=192.168.10.217 --slave_port=3306 --workdir=/var/tmp --target_version=5.6.16-64.2-rel64.2-log --manager_version=0.56 --relay_log_info=/usr/local/mysql/data/relay-log.info --relay_dir=/usr/local/mysql/data/ --slave_pass=xxx
Tue May 13 08:10:56 2014 - [info] Connecting to root@192.168.10.217(node2:22)..
Checking slave recovery environment settings..
Opening /usr/local/mysql/data/relay-log.info ... ok.
Relay log found at /usr/local/mysql/logs, up to relay-bin.000006
Temporary relay log file is /usr/local/mysql/logs/relay-bin.000006
Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Tue May 13 08:10:57 2014 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=node3 --slave_ip=192.168.10.218 --slave_port=3306 --workdir=/var/tmp --target_version=5.6.16-64.2-rel64.2-log --manager_version=0.56 --relay_log_info=/usr/local/mysql/data/relay-log.info --relay_dir=/usr/local/mysql/data/ --slave_pass=xxx
Tue May 13 08:10:57 2014 - [info] Connecting to root@192.168.10.218(node3:22)..
Checking slave recovery environment settings..
Opening /usr/local/mysql/data/relay-log.info ... ok.
Relay log found at /usr/local/mysql/logs, up to relay-bin.000006
Temporary relay log file is /usr/local/mysql/logs/relay-bin.000006
Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Tue May 13 08:10:57 2014 - [info] Slaves settings check done.
Tue May 13 08:10:57 2014 - [info]
node1(192.168.10.216:3306) (current master)
+--node2(192.168.10.217:3306)
+--node3(192.168.10.218:3306)
Tue May 13 08:10:57 2014 - [info] Checking replication health on node2..
Tue May 13 08:10:57 2014 - [info] ok.
Tue May 13 08:10:57 2014 - [info] Checking replication health on node3..
Tue May 13 08:10:57 2014 - [info] ok.
Tue May 13 08:10:57 2014 - [warning] master_ip_failover_script is not defined.
Tue May 13 08:10:57 2014 - [warning] shutdown_script is not defined.
Tue May 13 08:10:57 2014 - [info] Got exit code 0 (Not master dead).
MySQL Replication Health is OK.
G. 启动MHA manager，并监控日志文件
在node1上killall mysqld的同时在node3上启动manager服务
[root@localhost mha]# masterha_manager --conf=/etc/mha/app1/app1.cnf
Tue May 13 08:19:01 2014 - [info] Reading default configuration from /etc/masterha_default.cnf..
Tue May 13 08:19:01 2014 - [info] Reading application default configuration from /etc/mha/app1/app1.cnf..
Tue May 13 08:19:01 2014 - [info] Reading server configuration from /etc/mha/app1/app1.cnf..
Creating /var/tmp if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /usr/local/mysql/logs, up to mysql-bin.000009
Tue May 13 08:19:18 2014 - [info] Reading default configuration from /etc/masterha_default.cnf..
Tue May 13 08:19:18 2014 - [info] Reading application default configuration from /etc/mha/app1/app1.cnf..
Tue May 13 08:19:18 2014 - [info] Reading server configuration from /etc/mha/app1/app1.cnf..
之后观察node3上/var/log/mha/app1/manager.log日志会发现node1 dead状态，主自动切换到node2上，而node3上的主从配置指向了node2，并且发生一次切换后会生成/var/log/mha/app1/app1.failover.complete文件；
更多详情见请继续阅读下一页的精彩内容：
|
评论暂时关闭