有关pt-heartbeat工具的安装可以参考：percona-toolkit的安装及简介
有关pt-heartbeat工具的介绍可以参考：使用pt-heartbeat监控主从复制延迟

1、脚本概述
a、脚本定期使用--check方式单次检查当前的延迟性(定期的方式可以使用cron job比如每1分钟或5分钟)
b、通过设定指定的延迟阀值来判断当时的延迟性是否在可控范围
c、一旦当前的延迟大于指定阀值，则马上使用--monitor方式不停的监控其延迟性并写入到日志文件
d、对于--monitor方式，其进程运行超过30分钟，自kill其进程，以避免无限期运行导致日志过大，空间不够用

2、脚本内容

[mysql@SZDB run]$ more ck_slave_lag.sh 
#!/bin/bash
#set -x
if [ $# -ne 3 ];then
 echo "usage:"
 echo "ck_slave_lag.sh <Servier-id> <MaxLag> <LogDir>"
 exit 0;
fi

# Author : Leshami
# Blog : http://www.gxlcms.com/

ServerID=$1
MaxLag=$2
LogDir=$3
Timestamp=`date +%Y%m%d_%H%M%S`
Rentition=7
LogFile=$LogDir/slave_lag_$Timestamp.log
LagDetail=$LogDir/slave_lag_Detail_$Timestamp.log
mailadd=leshami@12306.cn

echo $ServerID
echo $MaxLag
echo $LogDir
echo $LogFile
echo $LagDetail
echo $mailadd

if [ ! -d $LogDir ];then
 mkdir -p $LogDir
fi

Lag=`/usr/bin/pt-heartbeat --user=monitor --password=xxx -S /tmp/mysql.sock -D test --master-server-id=$ServerID --check`
Lag=`echo ${Lag%.*}`
#Lag=3
echo $Lag
ptStatus=`ps -ef|grep pt-heart|grep daemonize`
echo $ptStatus

if [ $Lag -gt $MaxLag ]; then
 echo "The current date is `date` at `hostname`." >>$LogFile 
 echo "The current lag log file is $LogFile." >>$LogFile
 echo "The current replication lag is $Lag." >>$LogFile
 echo "The replication lag is larger than max lag $MaxLag." >>$LogFile
 
 if [ -z "$ptStatus" ] ; then
 echo "Start a monitor daemon with below command: " >>$LogFile 
 echo "pt-heartbeat --user=monitor --password=xxx -S /tmp/mysql.sock -D test " >>$LogFile
 echo " --master-server-id=11 --monitor --print-master-server-id --daemonize --log=$LagDetail" >>$LogFile
 /usr/bin/pt-heartbeat --user=monitor --password=xxx -S /tmp/mysql.sock -D test \
 --master-server-id=$ServerID --monitor --print-master-server-id --daemonize --log=$LagDetail
 echo "More detail please check lag log from $LagDetail." >>$LogFile
 cat $LogFile | mutt -s "Found slave lag on `hostname`." $mailadd
 fi
fi

if [ -n "$ptStatus" ] ; then
 STime=`ps -ef|grep pt-heart|grep daemonize |gawk '{print $5}'`
 Pid=`ps -ef|grep pt-heart|grep daemonize |gawk '{print $2}'`
 STime=`date '+%Y%m%d'`" "$STime
 s_STime=`date -d "$STime" '+%s'`
 s_ETime=`date +%s`
 DiffSec=`expr $s_ETime - $s_STime`

 echo $STime
 echo $s_STime
 echo $s_ETime
 echo $DiffSec

 if [ "$DiffSec" -gt 1800 ]; then
 echo "kill -9 $Pid"
 kill -9 $Pid
 fi
fi

# Remove history slave lag log.
find $LogDir -name "*slave_lag*" -ctime +$Rentition -delete 
exit

3、部署参考

[mysql@SZDB run]$ crontab -l

#check slave lag
*/1 * * * * /run/ck_slave_lag.sh 11 3 /log/SlaveLag