MySQL主从失败 错误Got fatal error 1236


同事给我打电话说MySQL团购数据库主从不同步了，尼玛，咋就这么苦逼呢，好吧，速度开电脑拨VPN解决。
1、登录从库查看主从同步状态，确实是否不同步
mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State:
Master_Host: 10.10.0.236
Master_User: slave
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.001501
Read_Master_Log_Pos: 144001931
Relay_Log_File: mysql-relay-bin.000296
Relay_Log_Pos: 144002077
Relay_Master_Log_File: mysql-bin.001501
Slave_IO_Running: No
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB: mysql,information_schema,performance_schema,test
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 144001931
Relay_Log_Space: 144002276
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 1236
Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'Client requested master to start replication from impossible position'
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 2
1 row in set (0.00 sec)

看来尼玛确实出问题了，查看从库错误日志：
[root@tg_0_zjc tuan]# tail -n 30 logs/mysqld.log
130619 22:23:36 [Note] Slave SQL thread initialized, starting replication in log 'mysql-bin.001343' at position 107, relay log './mysqld-relay-bin.000001' position: 4
130619 22:23:36 [Note] Slave I/O thread: connected to master 'slave@10.10.0.236:3306',replication started in log 'mysql-bin.001343' at position 107
130620 14:04:21 [Note] Slave: received end packet from server, apparent master shutdown:
130620 14:04:21 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mysql-bin.001343' at position 2544563
130620 14:04:21 [ERROR] Slave I/O: error reconnecting to master 'slave@10.10.0.236:3306' - retry-time: 60  retries: 86400, Error_code: 2003
130620 14:13:45 [Note] Slave: connected to master 'slave@10.10.0.236:3306',replication resumed in log 'mysql-bin.001343' at position 2544563
130623 22:04:17 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013)
130623 22:04:17 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mysql-bin.001402' at position 121870679
130623 22:04:20 [ERROR] Slave I/O: error reconnecting to master 'slave@10.10.0.236:3306' - retry-time: 60  retries: 86400, Error_code: 2003
130623 22:08:29 [Note] Slave: connected to master 'slave@10.10.0.236:3306',replication resumed in log 'mysql-bin.001402' at position 121870679
130623 22:08:29 [ERROR] Error reading packet from server: Client requested master to start replication from impossible position ( server_errno=1236)
130623 22:08:29 [ERROR] Slave I/O: Got fatal error 1236 from master when reading data from binary log: 'Client requested master to start replication from impossible position', Error_code: 1236
130623 22:08:29 [Note] Slave I/O thread exiting, read up to log 'mysql-bin.001402', position 121870679
130623 22:26:29 [Note] Error reading relay log event: slave SQL thread was killed
130623 22:26:34 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='10.10.0.236', master_port='3306', master_log_file='mysql-bin.001402', master_log_pos='121870679'. New state master_host='10.10.0.236', master_port='3306', master_log_file='mysql-bin.001403', master_log_pos='107'.
130623 22:26:42 [Note] Slave SQL thread initialized, starting replication in log 'mysql-bin.001403' at position 107, relay log './mysqld-relay-bin.000001' position: 4
130623 22:26:42 [Note] Slave I/O thread: connected to master 'slave@10.10.0.236:3306',replication started in log 'mysql-bin.001403' at position 107
130629 12:04:21 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013)
130629 12:04:21 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mysql-bin.001501' at position 144001931
130629 12:04:24 [ERROR] Slave I/O: error reconnecting to master 'slave@10.10.0.236:3306' - retry-time: 60  retries: 86400, Error_code: 2003
130629 12:08:33 [Note] Slave: connected to master 'slave@10.10.0.236:3306',replication resumed in log 'mysql-bin.001501' at position 144001931
130629 12:08:33 [ERROR] Error reading packet from server: Client requested master to start replication from impossible position ( server_errno=1236)
130629 12:08:33 [ERROR] Slave I/O: Got fatal error 1236 from master when reading data from binary log: 'Client requested master to start replication from impossible position', Error_code: 1236
130629 12:08:33 [Note] Slave I/O thread exiting, read up to log 'mysql-bin.001501', position 144001931
130629 22:10:08 [Note] Error reading relay log event: slave SQL thread was killed
130629 22:10:12 [Note] Slave SQL thread initialized, starting replication in log 'mysql-bin.001501' at position 144001931, relay log './mysqld-relay-bin.000296' position: 144002077
130629 22:10:12 [Note] Slave I/O thread: connected to master 'slave@10.10.0.236:3306',replication started in log 'mysql-bin.001501' at position 144001931
130629 22:10:12 [ERROR] Error reading packet from server: Client requested master to start replication from impossible position ( server_errno=1236)
130629 22:10:12 [ERROR] Slave I/O: Got fatal error 1236 from master when reading data from binary log: 'Client requested master to start replication from impossible position', Error_code: 1236
130629 22:10:12 [Note] Slave I/O thread exiting, read up to log 'mysql-bin.001501', position 144001931

尝试更改position位置恢复：
mysql> slave stop;
Query OK, 0 rows affected (0.01 sec)
mysql> change master to master_host='10.10.0.236', master_Port=3306, master_user='slave', master_password='123456', master_log_file='mysql-bin.001501', master_log_pos=144001931;
Query OK, 0 rows affected (0.05 sec)
mysql> slave start;
Query OK, 0 rows affected (0.00 sec)
mysql> show slave status \G
*************************** 1. row ***************************
Slave_IO_State:
Master_Host: 10.10.0.236
Master_User: slave
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.001501
Read_Master_Log_Pos: 144001931
Relay_Log_File: mysqld-relay-bin.000001
Relay_Log_Pos: 4
Relay_Master_Log_File: mysql-bin.001501
Slave_IO_Running: No
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB: mysql,information_schema,performance_schema,test,mysql,information_schema,performance_schema,test
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 144001931
Relay_Log_Space: 107
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 1236
Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'Client requested master to start replication from impossible position'
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 2
1 row in set (0.00 sec)

坑爹啊，还是不行，查看主库的binlog日志，按照错误点的标记去主库的日志中查找：
[root@tg_0_zq ~]#  /usr/local/mysql/bin/mysqlbinlog --start-position=144001931 /data/mysql/mysql-bin.001501
/*!40010 SET @@session.max_insert_delayed_threads=0*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
# at 40019
#130629 12:00:50 server id 2  end_log_pos 107  Start: binlog v 4, server v 5.5.15-log created 130629 12:00:50
# Warning: this binlog is either in use or was not closed properly.
BINLOG '
8lvOUQ8CAAAAZwAAAGsAAAABAAQANS41LjE1LWxvZwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAEzgNAAgAEgAEBAQEEgAAVAAEGggAAAAICAgCAA==
'/*!*/;
DELIMITER ;
# End of log file
ROLLBACK /* added by mysqlbinlog */;
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;

找到和出错最近的那个点，重新change master，把position止到这个位置：
 
mysql> change master to master_host='10.10.0.236', master_Port=3306, master_user='slave', master_password='123456', master_log_file='mysql-bin.001501', master_log_pos=40019;
Query OK, 0 rows affected (0.00 sec)
mysql> slave start;
Query OK, 0 rows affected (0.00 sec)
mysql> show slave status \G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 10.10.0.236
Master_User: slave
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.001502
Read_Master_Log_Pos: 204293576
Relay_Log_File: mysqld-relay-bin.000003
Relay_Log_Pos: 28179632
Relay_Master_Log_File: mysql-bin.001502
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:

尼玛，终于好使了，关机睡觉。

推荐阅读：

Ubuntu下Nginx做负载实现高性能WEB服务器5---MySQL主主同步

生产环境MySQL主主同步主键冲突处理

相关内容