Summary

Setup

We will use the same setup as last time: one master and two slaves, all using GTID replication. We can see the topology usingmysqlfailoverwith thehealthcommand:

$ mysqlfailover --master=root@localhost:13001 --discover-slaves-login=root health[...]MySQL Replication Failover UtilityFailover Mode = auto Next Interval = Tue Jul1 10:01:22 2014Master Information------------------Binary Log File PositionBinlog_Do_DBBinlog_Ignore_DBmysql-bin.000003700GTID Executed Seta9a396c6-00f3-11e4-8e66-9cebe8067a3f:1-3Replication Health Status+------------+--------+---------+--------+------------+---------+| host | port | role| state| gtid_mode| health|+------------+--------+---------+--------+------------+---------+| localhost| 13001| MASTER| UP | ON | OK|| localhost| 13002| SLAVE | UP | ON | OK|| localhost| 13003| SLAVE | UP | ON | OK|+------------+--------+---------+--------+------------+---------+

Note that--master-info-repository=TABLEneeds to be configured on all slaves or the tool will exit with an error message:

2014-07-01 10:18:55 AM CRITICAL Failover requires --master-info-repository=TABLE for all slaves.ERROR: Failover requires --master-info-repository=TABLE for all slaves.

Failover

You can use 2 commands to trigger automatic failover:

Let’s start the tool withauto:

$ mysqlfailover --master=root@localhost:13001 --discover-slaves-login=root auto

The monitoring console is visible and is refreshed every--intervalseconds (default: 15). Its output is similar to what you get when using thehealthcommand.

Then let’s kill -9 the master to see what happens once the master is detected as down:

Failed to reconnect to the master after 3 attemps.Failover starting in 'auto' mode...# Candidate slave localhost:13002 will become the new master.# Checking slaves status (before failover).# Preparing candidate for failover.# Creating replication user if it does not exist.# Stopping slaves.# Performing STOP on all slaves.# Switching slaves to new master.# Disconnecting new master as slave.# Starting slaves.# Performing START on all slaves.# Checking slaves for errors.# Failover complete.# Discovering slaves for master at localhost:13002Failover console will restart in 5 seconds.MySQL Replication Failover UtilityFailover Mode = auto Next Interval = Tue Jul1 10:59:47 2014Master Information------------------Binary Log File PositionBinlog_Do_DBBinlog_Ignore_DBmysql-bin.000005191GTID Executed Seta9a396c6-00f3-11e4-8e66-9cebe8067a3f:1-3Replication Health Status+------------+--------+---------+--------+------------+---------+| host | port | role| state| gtid_mode| health|+------------+--------+---------+--------+------------+---------+| localhost| 13002| MASTER| UP | ON | OK|| localhost| 13003| SLAVE | UP | ON | OK|+------------+--------+---------+--------+------------+---------+

Looks good! The tool is then ready to fail over to another slave if the new master becomes unavailable.

You can also run custom scripts at several points of execution with the--exec-before,--exec-after,--exec-fail-check,--exec-post-failoveroptions.

However it would be great to have a--failover-and-exitoption to avoid flapping: the tool would detect master failure, promote one of the slaves, reconfigure replication and then exit (this is what MHA does for instance).

Tool registration

When the tool is started, it registers itself on the master by writing a few things in the specific table:

mysql> SELECT * FROM mysql.failover_console;+-----------+-------+| host| port|+-----------+-------+| localhost | 13001 |+-----------+-------+

This is nice as it avoids that you start several instances ofmysqlfailoverto monitor the same master. If we try, this is what we get:

$ mysqlfailover --master=root@localhost:13001 --discover-slaves-login=root auto[...]Multiple instances of failover console found for master localhost:13001.If this is an error, restart the console with --force.Failover mode changed to 'FAIL' for this instance.Console will start in 10 seconds..........starting Console.

With thefailcommand,mysqlfailoverwill monitor replication health and exit in the case of a master failure, without actually performing failover.

Running in the background

In all previous examples,mysqlfailoverwas running in the foreground. This is very good for demo, but in a production environment you are likely to prefer running it in the background. This can be done with the--daemonoption:

$ mysqlfailover --master=root@localhost:13001 --discover-slaves-login=root auto --daemon=start --log=/var/log/mysqlfailover.log

and it can be stopped with:

$ mysqlfailover --daemon=stop

Errant transactions

If we create an errant transaction on one of the slaves, it will be detected:

MySQL Replication Failover UtilityFailover Mode = auto Next Interval = Tue Jul1 16:29:44 2014[...]WARNING: Errant transaction(s) found on slave(s).Replication Health Status[...]

However this does not prevent failover from occurring! You have to use--pedantic:

$ mysqlfailover --master=root@localhost:13001 --discover-slaves-login=root --pedantic auto[...]# WARNING: Errant transaction(s) found on slave(s).#- For slave 'localhost@13003': db906eee-012d-11e4-8fe1-9cebe8067a3f:12014-07-01 16:44:49 PM CRITICAL Errant transaction(s) found on slave(s). Note: If you want to ignore this issue, please do not use the --pedantic option.ERROR: Errant transaction(s) found on slave(s). Note: If you want to ignore this issue, please do not use the --pedantic option.

Limitations

Conclusion

mysqlfailoveris a good tool to automate failover in clusters using GTID replication. It is flexible and looks reliable. Its main drawback is that there is no easy way to make it highly available itself: ifmysqlfailovercrashes, you will have to manually restart it.