Tungsten Replicator Master/Slave MySQL replication
– switching master and slave roles
Here’s a quick run down of how to switch Tungsten Replicator master and slave roles.
Getting current status of .101 and .102
[root@centos6 ~]# mysqladmin create test2
[root@centos6 ~]# mysql -e "show databases;"
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| test |
| test2 |
| tungsten |
| tungsten_rep1 |
+--------------------+
[tungsten@centos6 tungsten]$ ./tungsten/tungsten-replicator/bin/trepctl services
Processing services command...
NAME VALUE
---- -----
appliedLastSeqno: 1
appliedLatency : 0.299
role : master
serviceName : rep1
serviceType : local
started : true
state : ONLINE
Finished services command... |
[root@centos6 ~]# mysqladmin create test2
[root@centos6 ~]# mysql -e "show databases;"
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| test |
| test2 |
| tungsten |
| tungsten_rep1 |
+--------------------+
[tungsten@centos6 tungsten]$ ./tungsten/tungsten-replicator/bin/trepctl services
Processing services command...
NAME VALUE
---- -----
appliedLastSeqno: 1
appliedLatency : 0.299
role : master
serviceName : rep1
serviceType : local
started : true
state : ONLINE
Finished services command...
[tungsten@centos6 tungsten]$ ./tungsten/tungsten-replicator/bin/trepctl -service rep1 status
Processing status command...
NAME VALUE
---- -----
appliedLastEventId : mysql-bin.000006:0000000000000501;151
appliedLastSeqno : 1
appliedLatency : 0.299
clusterName : default
currentEventId : mysql-bin.000006:0000000000000501
currentTimeMillis : 1316293854178
dataServerHost : 192.168.56.101
extensions :
host : null
latestEpochNumber : 0
masterConnectUri : thl://:/
masterListenUri : thl://192.168.56.101:2112/
maximumStoredSeqNo : 1
minimumStoredSeqNo : 0
offlineRequests : NONE
pendingError : NONE
pendingErrorCode : NONE
pendingErrorEventId : NONE
pendingErrorSeqno : -1
pendingExceptionMessage: NONE
resourcePrecedence : 99
rmiPort : -1
role : master
seqnoType : java.lang.Long
serviceName : rep1
serviceType : local
simpleServiceName : rep1
siteName : default
sourceId : 192.168.56.101
state : ONLINE
timeInStateSeconds : 2029.207
uptimeSeconds : 2034.258
Finished status command... |
[tungsten@centos6 tungsten]$ ./tungsten/tungsten-replicator/bin/trepctl -service rep1 status
Processing status command...
NAME VALUE
---- -----
appliedLastEventId : mysql-bin.000006:0000000000000501;151
appliedLastSeqno : 1
appliedLatency : 0.299
clusterName : default
currentEventId : mysql-bin.000006:0000000000000501
currentTimeMillis : 1316293854178
dataServerHost : 192.168.56.101
extensions :
host : null
latestEpochNumber : 0
masterConnectUri : thl://:/
masterListenUri : thl://192.168.56.101:2112/
maximumStoredSeqNo : 1
minimumStoredSeqNo : 0
offlineRequests : NONE
pendingError : NONE
pendingErrorCode : NONE
pendingErrorEventId : NONE
pendingErrorSeqno : -1
pendingExceptionMessage: NONE
resourcePrecedence : 99
rmiPort : -1
role : master
seqnoType : java.lang.Long
serviceName : rep1
serviceType : local
simpleServiceName : rep1
siteName : default
sourceId : 192.168.56.101
state : ONLINE
timeInStateSeconds : 2029.207
uptimeSeconds : 2034.258
Finished status command...
[tungsten@localhost tungsten]$ mysql -e "show databases;"
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| test |
| test2 |
| tungsten |
| tungsten_rep1 |
+--------------------+
[tungsten@localhost tungsten]$ ./tungsten/tungsten-replicator/bin/trepctl services
Processing services command...
NAME VALUE
---- -----
appliedLastSeqno: 1
appliedLatency : 26.72
role : slave
serviceName : rep1
serviceType : local
started : true
state : ONLINE
Finished services command... |
[tungsten@localhost tungsten]$ mysql -e "show databases;"
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| test |
| test2 |
| tungsten |
| tungsten_rep1 |
+--------------------+
[tungsten@localhost tungsten]$ ./tungsten/tungsten-replicator/bin/trepctl services
Processing services command...
NAME VALUE
---- -----
appliedLastSeqno: 1
appliedLatency : 26.72
role : slave
serviceName : rep1
serviceType : local
started : true
state : ONLINE
Finished services command...
————————-
switch master/slave
————————-
Update: I forgot one crucial step when switching master/slave i.e. in cases of Master failover. That is to properly flush the log on Tungsten Replicator master .101 and note it’s log sequence number (118). And then on Tungsten Replicator slave which is to be promoted to master (in this case 192.168.56.102), tell it to wait until catch up to that sequence number (118).
[tungsten@localhost tungsten]$ ./tungsten/tungsten-replicator/bin/trepctl -service rep1 status
Processing status command...
NAME VALUE
---- -----
appliedLastEventId : mysql-bin.000006:0000000000000501;151
appliedLastSeqno : 1
appliedLatency : 26.72
clusterName : default
currentEventId : NONE
currentTimeMillis : 1316293891656
dataServerHost : 192.168.56.102
extensions :
host : null
latestEpochNumber : 0
masterConnectUri : thl://192.168.56.101:2112/
masterListenUri : thl://192.168.56.102:2112/
maximumStoredSeqNo : 1
minimumStoredSeqNo : 0
offlineRequests : NONE
pendingError : NONE
pendingErrorCode : NONE
pendingErrorEventId : NONE
pendingErrorSeqno : -1
pendingExceptionMessage: NONE
resourcePrecedence : 99
rmiPort : -1
role : slave
seqnoType : java.lang.Long
serviceName : rep1
serviceType : local
simpleServiceName : rep1
siteName : default
sourceId : 192.168.56.102
state : ONLINE
timeInStateSeconds : 2035.203
uptimeSeconds : 2038.652
Finished status command... |
[tungsten@localhost tungsten]$ ./tungsten/tungsten-replicator/bin/trepctl -service rep1 status
Processing status command...
NAME VALUE
---- -----
appliedLastEventId : mysql-bin.000006:0000000000000501;151
appliedLastSeqno : 1
appliedLatency : 26.72
clusterName : default
currentEventId : NONE
currentTimeMillis : 1316293891656
dataServerHost : 192.168.56.102
extensions :
host : null
latestEpochNumber : 0
masterConnectUri : thl://192.168.56.101:2112/
masterListenUri : thl://192.168.56.102:2112/
maximumStoredSeqNo : 1
minimumStoredSeqNo : 0
offlineRequests : NONE
pendingError : NONE
pendingErrorCode : NONE
pendingErrorEventId : NONE
pendingErrorSeqno : -1
pendingExceptionMessage: NONE
resourcePrecedence : 99
rmiPort : -1
role : slave
seqnoType : java.lang.Long
serviceName : rep1
serviceType : local
simpleServiceName : rep1
siteName : default
sourceId : 192.168.56.102
state : ONLINE
timeInStateSeconds : 2035.203
uptimeSeconds : 2038.652
Finished status command...
[root@centos6 ~]# tail -10 /opt/tungsten/tungsten/tungsten-replicator/log/trepsvc.log
INFO | jvm 1 | 2011/09/18 06:37:04 | 2011-09-18 06:37:04,791 [rep1 - Event dispatcher thread] INFO pipeline.Pipeline Starting pipeline: master
INFO | jvm 1 | 2011/09/18 06:37:04 | 2011-09-18 06:37:04,792 [rep1 - binlog-to-q-0] INFO pipeline.SingleThreadStageTask Starting stage task thread
INFO | jvm 1 | 2011/09/18 06:37:04 | 2011-09-18 06:37:04,792 [rep1 - Event dispatcher thread] INFO management.tungsten.TungstenPlugin Adding heartbeat for master
INFO | jvm 1 | 2011/09/18 06:37:04 | 2011-09-18 06:37:04,796 [rep1 - q-to-thl-0] INFO pipeline.SingleThreadStageTask Starting stage task thread
INFO | jvm 1 | 2011/09/18 06:37:04 | 2011-09-18 06:37:04,969 [rep1 - Event dispatcher thread] INFO management.OpenReplicatorManager Sent State Change Notification OFFLINE:NORMAL -> GOING-ONLINE:SYNCHRONIZING
INFO | jvm 1 | 2011/09/18 06:37:04 | 2011-09-18 06:37:04,970 [ - WrapperSimpleAppMain] INFO management.ReplicationServiceManager internal/local replication service 'rep1' started successfully
INFO | jvm 1 | 2011/09/18 06:37:04 | 2011-09-18 06:37:04,971 [rep1 - Event dispatcher thread] INFO management.OpenReplicatorManager Sent State Change Notification GOING-ONLINE:SYNCHRONIZING -> ONLINE
INFO | jvm 1 | 2011/09/18 06:37:13 | 2011-09-18 06:37:13,026 [rep1 - ConnectorHandler: initializing] INFO thl.Protocol THL protocol buffering enabled: size=10
INFO | jvm 1 | 2011/09/18 06:37:13 | 2011-09-18 06:37:13,034 [rep1 - ConnectorHandler: initializing] INFO thl.ConnectorHandler New THL client connection: sourceID=192.168.56.102 heartbeatMillis=3000
INFO | jvm 1 | 2011/09/18 06:37:13 | 2011-09-18 06:37:13,034 [rep1 - ConnectorHandler: initializing] INFO thl.ConnectorHandler Client log checking disabled; not checking for diverging histories |
[root@centos6 ~]# tail -10 /opt/tungsten/tungsten/tungsten-replicator/log/trepsvc.log
INFO | jvm 1 | 2011/09/18 06:37:04 | 2011-09-18 06:37:04,791 [rep1 - Event dispatcher thread] INFO pipeline.Pipeline Starting pipeline: master
INFO | jvm 1 | 2011/09/18 06:37:04 | 2011-09-18 06:37:04,792 [rep1 - binlog-to-q-0] INFO pipeline.SingleThreadStageTask Starting stage task thread
INFO | jvm 1 | 2011/09/18 06:37:04 | 2011-09-18 06:37:04,792 [rep1 - Event dispatcher thread] INFO management.tungsten.TungstenPlugin Adding heartbeat for master
INFO | jvm 1 | 2011/09/18 06:37:04 | 2011-09-18 06:37:04,796 [rep1 - q-to-thl-0] INFO pipeline.SingleThreadStageTask Starting stage task thread
INFO | jvm 1 | 2011/09/18 06:37:04 | 2011-09-18 06:37:04,969 [rep1 - Event dispatcher thread] INFO management.OpenReplicatorManager Sent State Change Notification OFFLINE:NORMAL -> GOING-ONLINE:SYNCHRONIZING
INFO | jvm 1 | 2011/09/18 06:37:04 | 2011-09-18 06:37:04,970 [ - WrapperSimpleAppMain] INFO management.ReplicationServiceManager internal/local replication service 'rep1' started successfully
INFO | jvm 1 | 2011/09/18 06:37:04 | 2011-09-18 06:37:04,971 [rep1 - Event dispatcher thread] INFO management.OpenReplicatorManager Sent State Change Notification GOING-ONLINE:SYNCHRONIZING -> ONLINE
INFO | jvm 1 | 2011/09/18 06:37:13 | 2011-09-18 06:37:13,026 [rep1 - ConnectorHandler: initializing] INFO thl.Protocol THL protocol buffering enabled: size=10
INFO | jvm 1 | 2011/09/18 06:37:13 | 2011-09-18 06:37:13,034 [rep1 - ConnectorHandler: initializing] INFO thl.ConnectorHandler New THL client connection: sourceID=192.168.56.102 heartbeatMillis=3000
INFO | jvm 1 | 2011/09/18 06:37:13 | 2011-09-18 06:37:13,034 [rep1 - ConnectorHandler: initializing] INFO thl.ConnectorHandler Client log checking disabled; not checking for diverging histories
* stop slave before master
trepctl -service rep1 -host 192.168.56.102 offline
trepctl -service rep1 -host 192.168.56.101 offline |
trepctl -service rep1 -host 192.168.56.102 offline
trepctl -service rep1 -host 192.168.56.101 offline
# Assign each server its new role
# 192.168.56.102 becomes the new master
trepctl -service rep1 -host 192.168.56.102 setrole -role master |
trepctl -service rep1 -host 192.168.56.102 setrole -role master
# Assign 192.168.56.101 as new slave
# listening to 192.168.56.102 for data
trepctl -service rep1 -host 192.168.56.101 setrole -role slave -uri thl://192.168.56.102 |
trepctl -service rep1 -host 192.168.56.101 setrole -role slave -uri thl://192.168.56.102
# start both servers
trepctl -service rep1 -host 192.168.56.102 online
trepctl -service rep1 -host 192.168.56.101 online |
trepctl -service rep1 -host 192.168.56.102 online
trepctl -service rep1 -host 192.168.56.101 online
# check service for new roles
trepctl -host 192.168.56.101 services
trepctl -host 192.168.56.102 services |
trepctl -host 192.168.56.101 services
trepctl -host 192.168.56.102 services
[tungsten@centos6 tungsten]$ trepctl -host 192.168.56.101 services
Processing services command...
NAME VALUE
---- -----
appliedLastSeqno: 13
appliedLatency : 11.897
role : slave
serviceName : rep1
serviceType : local
started : true
state : ONLINE
Finished services command... |
[tungsten@centos6 tungsten]$ trepctl -host 192.168.56.101 services
Processing services command...
NAME VALUE
---- -----
appliedLastSeqno: 13
appliedLatency : 11.897
role : slave
serviceName : rep1
serviceType : local
started : true
state : ONLINE
Finished services command...
[tungsten@localhost root]$ trepctl -host 192.168.56.102 services
Processing services command...
NAME VALUE
---- -----
appliedLastSeqno: 13
appliedLatency : 0.768
role : master
serviceName : rep1
serviceType : local
started : true
state : ONLINE
Finished services command... |
[tungsten@localhost root]$ trepctl -host 192.168.56.102 services
Processing services command...
NAME VALUE
---- -----
appliedLastSeqno: 13
appliedLatency : 0.768
role : master
serviceName : rep1
serviceType : local
started : true
state : ONLINE
Finished services command...