HAproxy reload config file with uninterrupt session

HAProxy is a high performance load balancer. It is very light-weight, and free, making it a great option if you are in the market for a load balancer and need to keep your costs down.

Lately we’ve been making a lot of load balancer changes at work to accommodate new systems and services. Even though we have two load balancers running with keepalived taking care of any failover situations, I was thinking about how we go about reloading our configuration files. In the event of a change, the “common” way to get the changes to take effect is to run /etc/init.d/haproxy restart. This is bad for a couple major reasons:

You are temporarily shutting your load balancer down
You are severing any current connections going through the load balancer
You might say, “if you have two load balancers with keepalived, restarting the service should be fine since keepalived will handle the failover.” This, however, isn’t always true. Keepalived uses advertisements to determine when to fail over. The default advertisement interval is 1 second (configurable in keepalived.conf). The skew time helps to keep everyone from trying to transition at once. It is a number between 0 and 1, based on the formula (256 – priority) / 256. As defined in the RFC, the backup must receive an advertisement from the master every (3 * advert_int) + skew_time seconds. If it doesn’t hear anything from the master, it takes over.

Let’s assume you are using the default interval of 1 second. On my test machine, this is the duration of time it takes to restart haproxy:

time /etc/init.d/haproxy restart
Restarting haproxy haproxy
   ...done.real    0m0.022s
user    0m0.000s
sys     0m0.016s

In this situation, haproxy would restart much faster than your 1 second interval. You could get lucky and happen to restart it just before the check, but luck is not consistent enough to be useful. Also, in very high-traffic situations, you’ll be causing a lot of connection issues. So we cannot rely on keepalived to solve the first problem, and it definitely doesn’t solve the second problem.

After sifting through haproxy documentation (the text-based documentation, not the man page) (/usr/share/doc/haproxy/haproxy-en.txt.gz on Ubuntu), I came across this:

    314     global    315         daemon    316         quiet    317         nbproc  2
    318         pidfile /var/run/haproxy-private.pid    319
    320     # to stop only those processes among others :    321     # kill $(</var/run/haproxy-private.pid)    
    323     # to reload a new configuration with minimal service impact and without    
    324     # breaking existing sessions :    
    325     # haproxy -f haproxy.cfg -p $(</var/run/haproxy-private.pid) -st $(</var/run/haproxy-private.pid)

That last command is the one of interest. The -p asks the process to write down each of its children’s pids to the specified pid file, and the -st specifies a list of pids to send a SIGTERM to after startup. But it does this in an interesting way:

    609 The '-st' and '-sf' command line options are used to inform previously running
    610 processes that a configuration is being reloaded. They will receive the SIGTTOU    
    611 signal to ask them to temporarily stop listening to the ports so that the new
    612 process can grab them. If anything wrong happens, the new process will send
    613 them a SIGTTIN to tell them to re-listen to the ports and continue their normal
    614 work. Otherwise, it will either ask them to finish (-sf) their work then softly    
    615 exit, or immediately terminate (-st), breaking existing sessions. A typical use    
    616 of this allows a configuration reload without service interruption :    
    618  # haproxy -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)

The end-result is a reload of the configuration file which is not visible by the customer. It also solves the second problem! Let’s look at an example of the command and look at the time compared to our above example:

# time haproxy -f /etc/haproxy.cfg -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)

real    0m0.018s
user    0m0.000s
sys     0m0.004s

I’ve specified the config file I want to use and the pid file haproxy is currently using. The $(cat /var/run/haproxy.pid) takes the output of cat /var/run/haproxy.pid and passes it in to the -sf parameter as a list, which is what it is expecting. You will notice that the time is actually faster too (.012s sys, and .004s real). It may not seem like much, but if you are dealing with very high volumes of traffic, this can be pretty important. Luckily for us it doesn’t matter because we’ve been able to reload the haproxy configuration without dropping any connections and without causing any customer-facing issues.

UPDATE: There is a reload in some of the init.d scripts (I haven’t checked every OS, so this can vary), but it uses the -st option which will break existing sessions, as opposed to using -sf to do a graceful hand-off. You can modify the haproxy_reload() function to use the -sf if you want. I also find it a bit confusing that the documentation uses $(cat /path/to/pidfile) whereas this haproxy_reload() function uses $(<$PIDFILE). Either should work, but really, way to lead by example…



上一篇 2015-04-03 22:07
下一篇 2015-04-03 22:10


  • n25第二周

    linux文件管理类命令     mkdir,rmdir,cp,mv,rm,chwon,chmod     1.mkdir->make directories(创建目录)     用法:    &nbsp…

    Linux干货 2016-12-11
  • 13 文件查找与压缩

    locate find 压缩命令 file-roller、compress/uncompress(.Z)、gzip/gunzip(.gz)、bzip2/bunzip2(.bz2)、xz/unxz(.xz) zip/unzip、tar locate 数据源 1)查询系统上预建的文件索引数据库      2)依赖于事先构建…

    Linux干货 2016-08-18
  • N21沉舟11周作业

    1、详细描述一次加密通讯的过程,结合图示最佳。 2、描述创建私有CA的过程,以及为客户端发来的证书请求进行办法证书。 一、CA服务器端 #进入CA目录:cd    /etc/pki/CA #创建初始文件touch index.txt serialecho 01 >&nbsp…

    Linux干货 2016-09-19
  • 马哥教育网络班20期-第三周课程作业

    Table of Contents 1、列出当前系统上所有已经登录的用户的用户名,注意:同一个用户登录多次,则只显示一次即可。 2、取出最后登录到当前系统的用户的相关信息。 3、取出当前系统上被用户当作其默认shell的最多的那个shell。 4、将/etc/passwd中的第三个字段数值最大的后10个用户的信息全部改为大写后保存至/tmp/maxusers…

    Linux干货 2016-06-26
  • N28 第三周【1】:grep和文本处理工具的使用

    grep一些练习 1、列出当前系统上所有已经登录的用户的用户名,注意:同一个用户登录多次,则只显示一次即可。 [root@localhost ~]# who |cut -d” ” -f1 |sort -u root 2、取出最后登录到当前系统的用户的相关信息。 [root@localhost ~]# last |cut -d” ” -f1|head -1 |…

    Linux干货 2017-12-19
  • 马哥教育网络班22期+第二周课程练习

    一、Linux上的文件管理类命令都有哪些,其常用的使用方法及其相关示例演示? cp命令:copy,有2类复制方法  1、单源复制:如果DEST不存在:则实现创建此文件,并复制源文件的数据流至DEST中;     如果DEST存在:如果DEST是非目录文件,则覆盖目标文件;如果DEST是目录文件,则先DEST目录下创建一个与源文件…

    Linux干货 2016-08-22