Master-replica: automated failover
Example on GitHub: auto_leader
This tutorial shows how to configure and work with a replica set with automated failover.
Before starting this tutorial:
Install the tt utility.
Create a tt environment in the current directory by executing the tt init command.
Inside the
instances.enabled
directory of the created tt environment, create theauto_leader
directory.Inside
instances.enabled/auto_leader
, create theinstances.yml
andconfig.yaml
files:instances.yml
specifies instances to run in the current environment and should look like this:The
config.yaml
file is intended to store a replica set configuration.
This section describes how to configure a replica set in config.yaml
.
Define a replica set topology inside the groups section. The iproto.listen option specifies an address used to listen for incoming requests and allows replicas to communicate with each other.
After configuring a replica set, execute the tt start command from the tt environment directory:
Check that instances are in the
RUNNING
status using the tt status command:
Connect to
instance001
using tt connect:Check the instance state in regard to leader election using box.info.election. The output below shows that
instance001
is a follower whileinstance002
is a replica set leader.Check that
instance001
is in read-only mode usingbox.info.ro
:Execute
box.info.replication
to check a replica set status. Make sure thatupstream.status
anddownstream.status
arefollow
forinstance002
andinstance003
.To see the diagrams that illustrate how the
upstream
anddownstream
connections look, refer to Monitoring a replica set.
To check that replicas (instance001
and instance003
) get all updates from the master (instance002
), follow the steps below:
Connect to
instance002
usingtt connect
:Create a space and add data as described in CRUD operation examples.
Use the
select
operation oninstance001
andinstance003
to make sure data is replicated.Check that the
1
component of box.info.vclock values are the same on all instances:instance001
:instance002
:instance003
:
Note
Note that a
vclock
value might include the0
component that is related to local space operations and might differ for different instances in a replica set.
To test how automated failover works if the current master is stopped, follow the steps below:
Stop the current master instance (
instance002
) using thett stop
command:On
instance001
, checkbox.info.election
. In this example, a new replica set leader isinstance001
.Check replication status using
box.info.replication
forinstance002
:upstream.status
isdisconnected
.downstream.status
isstopped
.
The diagram below illustrates how the
upstream
anddownstream
connections look like:Start
instance002
back usingtt start
:
Make sure that box.info.vclock values (except the
0
components) are the same on all instances:instance001
:instance002
:instance003
:
On
instance002
, run box.ctl.promote() to choose it as a new replica set leader:Check
box.info.election
to make sureinstance002
is a leader now:
The process of adding instances to a replica set and removing them is similar for all failover modes. Learn how to do this from the Master-replica: manual failover tutorial:
Before removing an instance from a replica set with replication.failover set to election
, make sure this instance is in read-only mode.
If the instance is a master, choose a new leader manually.