redhat 6.4(64位)下HAproxy+Keepalived 负载均衡架构搭建
redhat 6.4（64位）下HAproxy+Keepalived 负载均衡架构搭建（实践版）
一、环境
系统：redhat 6.4 64位
软件：haproxy-1.4.25.tar.gz
keepalived-1.2.12.tar.gz
网络：
MASTER_IP（haproxy）=192.168.101.1
BACKUP_IP（haproxy）=192.168.101.2
VIP（虚拟IP）=192.168.101.4
关闭SELINUX
# vi /etc/sysconfig/config (修改)
SELINUX=disabled
# setenforce 0 (这个为临时生效，上面的为重启后生效)
关闭防火墙
二、haproxy的安装配置（两台Haproxy的安装配置步骤一样）
1、192.168.101.1主机上的安装haproxy
检查系统的内核版本
# uname -a
Linux rac1 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 EST 2013 x86_64 x86_64 x86_64 GNU/Linux
#使用uname -r查看内核，如：2.6.18-371.el5，此时该参数就为linux26
如x86_64 x86_64 x86_64 GNU/Linux，此时该参数就为x86_64
# tar zxvf haproxy-1.4.25.tar.gz
# cd haproxy-1.4.25
# make TARGET=linux2628 ARCH=x86_64 PREFIX=/usr/local/haproxy \\编译参数参考README，其中TARGET是指定内核版本，ARCH指定CPU架构，PREFIX=/usr/local/haproxy安装目录。
gcc -m64 -march=x86-64 -g -o haproxy src/haproxy.o src/sessionhash.o src/base64.o src/protocols.o src/uri_auth.o src/standard.o src/buffers.o src/log.o src/task.o src/time.o src/fd.o src/pipe.o src/regex.o src/cfgparse.o src/server.o src/checks.o src/queue.o src/client.o src/proxy.o src/stick_table.o src/proto_uxst.o src/proto_http.o src/stream_sock.o src/appsession.o src/backend.o src/lb_chash.o src/lb_fwlc.o src/lb_fwrr.o src/lb_map.o src/stream_interface.o src/dumpstats.o src/proto_tcp.o src/session.o src/hdr_idx.o src/ev_select.o src/signal.o src/acl.o src/pattern.o src/memory.o src/freq_ctr.o src/auth.o src/ev_poll.o src/ev_epoll.o src/ev_sepoll.o ebtree/ebtree.o ebtree/eb32tree.o ebtree/eb64tree.o ebtree/ebmbtree.o ebtree/ebsttree.o ebtree/ebimtree.o ebtree/ebistree.o -lcrypt
注：README编译参数参考
Also, you might want to build with libpcre support, which will provide a very
efficient regex implementation and will also fix some badness on Solaris' one.
To build haproxy, you have to choose your target OS amongst the following ones
and assign it to the TARGET variable :
- linux22 for Linux 2.2
- linux24 for Linux 2.4 and above (default)
- linux24e for Linux 2.4 with support for a working epoll (> 0.21)
- linux26 for Linux 2.6 and above
- linux2628 for Linux 2.6.28 and above (enables splice and tproxy)
- solaris for Solaris 8 or 10 (others untested)
- freebsd for FreeBSD 5 to 8.0 (others untested)
- osx for Mac OS/X
- openbsd for OpenBSD 3.1 to 5.2 (others untested)
- aix52 for AIX 5.2
- cygwin for Cygwin
- generic for any other OS.
- custom to manually adjust every setting
# make install PREFIX=/usr/local/haproxy
install -d /usr/local/haproxy/sbin
install haproxy /usr/local/haproxy/sbin
install -d /usr/local/haproxy/share/man/man1
install -m 644 doc/haproxy.1 /usr/local/haproxy/share/man/man1
install -d /usr/local/haproxy/doc/haproxy
for x in configuration architecture haproxy-en haproxy-fr; do \
install -m 644 doc/$x.txt /usr/local/haproxy/doc/haproxy ; \ done
安装完毕，在/usr/local/haproxy/目录下会看到三个目录：doc、sbin、share
2、安装完毕后，创建配置文件和启动文件
# cd /usr/local/haproxy
# cp /home/soft/haproxy-1.4.25/examples/haproxy.cfg /usr/local/haproxy/conf/
默认haproxy配置文件内容：
# more /home/soft/haproxy-1.4.25/examples/haproxy.cfg
# this config needs haproxy-1.1.28 or haproxy-1.2.1
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
#log loghost local0 info
maxconn 4096
chroot /usr/share/haproxy
uid 99
gid 99
daemon
#debug
#quiet
defaults
log global
mode http
option httplog
option dontlognull
retries 3
redispatch
maxconn 2000
contimeout 5000
clitimeout 50000
srvtimeout 50000
listen appli1-rewrite 0.0.0.0:10001
cookie SERVERID rewrite
balance roundrobin
server app1_1 192.168.34.23:8080 cookie app1inst1 check inter 2000 rise
2 fall 5
server app1_2 192.168.34.32:8080 cookie app1inst2 check inter 2000 rise 2 fall 5
server app1_3 192.168.34.27:8080 cookie app1inst3 check inter 2000 rise 2 fall 5
server app1_4 192.168.34.42:8080 cookie app1inst4 check inter 2000 rise 2 fall 5
listen appli2-insert 0.0.0.0:10002
option httpchk
balance roundrobin
cookie SERVERID insert indirect nocache
server inst1 192.168.114.56:80 cookie server01 check inter 2000 fall 3
server inst2 192.168.114.56:81 cookie server02 check inter 2000 fall 3
capture cookie vgnvisitor= len 32
option httpclose # disable keep-alive
rspidel ^Set-cookie:\ IP= # do not let this cookie tell our intern
al IP address
listen appli3-relais 0.0.0.0:10003
dispatch 192.168.135.17:80
listen appli4-backup 0.0.0.0:10004
option httpchk /index.html
option persist
balance roundrobin
server inst1 192.168.114.56:80 check inter 2000 fall 3
server inst2 192.168.114.56:81 check inter 2000 fall 3 backup
listen ssl-relay 0.0.0.0:8443
option ssl-hello-chk
balance source
server inst1 192.168.110.56:443 check inter 2000 fall 3
server inst2 192.168.110.57:443 check inter 2000 fall 3
server back1 192.168.120.58:443 backup
listen appli5-backup 0.0.0.0:10005
option httpchk *
balance roundrobin
cookie SERVERID insert indirect nocache
server inst1 192.168.114.56:80 cookie server01 check inter 2000 fall 3
server inst2 192.168.114.56:81 cookie server02 check inter 2000 fall 3
server inst3 192.168.114.57:80 backup check inter 2000 fall 3
capture cookie ASPSESSION len 32
srvtimeout 20000
option httpclose # disable keep-alive
option checkcache # block response if set-cookie & cacheab le
rspidel ^Set-cookie:\ IP= # do not let this cookie tell our intern
al IP address
errorloc 502 http://192.168.114.58/error502.html
errorfile 503 /etc/haproxy/errors/503.http
编辑配置文件（两台Haproxy配置文件相同）：
# vim /usr/local/haproxy/conf/haproxy.cfg
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
maxconn 2000
contimeout 5000
clitimeout 50000
srvtimeout 50000
listen web 192.168.10.222:80
mode http
option httplog
log global
option httpchk HEAD /test/test.jsp
server ha1 192.168.10.71:8080 check inter 2000 fall 3 weight 20
server ha2 192.168.10.72:8080 check inter 2000 fall 3 weight 10
创建启动文件：
# vim /etc/rc.d/init.d/haproxy
#!/bin/sh
#
# haproxy
#
# chkconfig: 35 85 15
# description: HAProxy is a free, very fast and reliable solution
# offering high availability, load balancing, and
# proxying for TCP and HTTP-based applications
# processname: haproxy
# config: /usr/local/haproxy/conf/haproxy.cfg
# pidfile: /var/run/haproxy.pid
# Source function library.
. /etc/rc.d/init.d/functions
# Source networking configuration.
. /etc/sysconfig/network
# Check that networking is up.
[ "$NETWORKING" = "no" ] && exit 0
config="/usr/local/haproxy/conf/haproxy.cfg"
exec="/usr/local/haproxy/sbin/haproxy"
prog=$(basename $exec)
[ -e /etc/sysconfig/$prog ] && . /etc/sysconfig/$prog
lockfile=/var/lock/subsys/haproxy
check() {
$exec -c -V -f $config
}
start() {
$exec -c -q -f $config
if [ $? -ne 0 ]; then
echo "Errors in configuration file, check with $prog check."
return 1
fi
echo -n $"Starting $prog: "
# start it up here, usually something like "daemon $exec"
daemon $exec -D -f $config -p /var/run/$prog.pid
retval=$?
echo
[ $retval -eq 0 ] && touch $lockfile
return $retval
}
stop() {
echo -n $"Stopping $prog: "
# stop it here, often "killproc $prog"
killproc $prog
retval=$?
echo
[ $retval -eq 0 ] && rm -f $lockfile
return $retval
}
restart() {
$exec -c -q -f $config
if [ $? -ne 0 ]; then
echo "Errors in configuration file, check with $prog check."
return 1
fi
stop
start
}
reload() {
$exec -c -q -f $config
if [ $? -ne 0 ]; then
echo "Errors in configuration file, check with $prog check."
return 1
fi
echo -n $"Reloading $prog: "
$exec -D -f $config -p /var/run/$prog.pid -sf $(cat /var/run/$prog.pid) retval=$?
echo
return $retval
}
force_reload() {
restart
}
fdr_status() {
status $prog
}
case "$1" in
start|stop|restart|reload)
$1
;;
force-reload)
force_reload
;;
checkconfig)
check
;;
status)
fdr_status
;;
condrestart|try-restart)
[ ! -f $lockfile ] || restart
;;
*)
echo $"Usage: $0 {start|stop|status|checkconfig|restart|try-restart|relo
ad|force-reload}"
exit 2
esac
# chmod +x /etc/rc.d/init.d/haproxy
3、启动测试haproxy
# service haproxy start
4、haproxy日志配置
默认情况下，HAProxy为了节省读写IO所消耗的性能，默认情况下没有日志输出，以下是我配置log的过程：
RHEL6.4 默认使用的是rsyslog管理log
rsyslog 默认情况下，需要在514端口监听UDP，所以可以把/etc/rsyslog.conf如下的注释去掉
# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514
新增以下条目：\\ 和haproxy.conf中的要一致
local0.* /var/log/haproxy.log
local1.* /var/log/haproxy.log
# vim /etc/sysconfig/rsyslog
#SYSLOGD_OPTIONS="-c 4"
SYSLOGD_OPTIONS="-c 5"
重启rsyslog服务
# /etc/init.d/rsyslog restart
重启haproxy
# service haproxy stop
# service haproxy start
查看haproxy.log
# tail -f /var/log/haproxy.log
Jun 19 11:18:52 localhost haproxy[9805]: Proxy web started.
三、keepalived安装配置
1、在主服务器上安装keepalived
注：安装keepalived需要gcc gcc-c++ glib* openssl-devel popt-devel libnl-devel make 依赖包。如果系统是64位的，那么可能需要32位、64位的都要安装。
# tar zxvf keepalived-1.2.12.tar.gz
# cd keepalived-1.2.12
# ./configure --prefix=/usr/local/keepalived
configure: creating ./config.status
config.status: creating Makefile
config.status: creating genhash/Makefile
config.status: creating keepalived/core/Makefile
config.status: creating lib/config.h
config.status: creating keepalived.spec
config.status: creating keepalived/Makefile
config.status: creating lib/Makefile
config.status: creating keepalived/vrrp/Makefile
config.status: creating keepalived/check/Makefile
config.status: creating keepalived/libipvs-2.6/Makefile
Keepalived configuration
------------------------
Keepalived version : 1.2.12
Compiler : gcc
Compiler flags : -g -O2 -DFALLBACK_LIBNL1
Extra Lib : -lssl -lcrypto -lcrypt -lnl
Use IPVS Framework : Yes
IPVS sync daemon support : Yes
IPVS use libnl : Yes
Use VRRP Framework : Yes
Use VRRP VMAC : Yes
SNMP support : No
SHA1 support : No
Use Debug flags : No
注：必须看到以下提示，说明配置正确，才能继续安装
Use IPVS Framework : Yes
IPVS sync daemon support : Yes
Use VRRP Framework : Yes
# make
make[2]: Leaving directory `/home/soft/keepalived-1.2.12/keepalived/libipvs-2.6' Building ../bin/keepalived
strip ../bin/keepalived
Make complete
make[1]: Leaving directory `/home/soft/keepalived-1.2.12/keepalived'
make -C genhash
make[1]: Entering directory `/home/soft/keepalived-1.2.12/genhash'
gcc -g -O2 -DFALLBACK_LIBNL1 -I/usr/src/linux/include -I/usr/src/linux/include -I../lib -Wall -Wunused -Wstrict-prototypes -c -o main.o main.c
gcc -g -O2 -DFALLBACK_LIBNL1 -I/usr/src/linux/include -I/usr/src/linux/include -I../lib -Wall -Wunused -Wstrict-prototypes -c -o sock.o sock.c
gcc -g -O2 -DFALLBACK_LIBNL1 -I/usr/src/linux/include -I/usr/src/linux/include -I../lib -Wall -Wunused -Wstrict-prototypes -c -o layer4.o layer4.c
gcc -g -O2 -DFALLBACK_LIBNL1 -I/usr/src/linux/include -I/usr/src/linux/include -I../lib -Wall -Wunused -Wstrict-prototypes -c -o http.o http.c
gcc -g -O2 -DFALLBACK_LIBNL1 -I/usr/src/linux/include -I/usr/src/linux/include -I../lib -Wall -Wunused -Wstrict-prototypes -c -o ssl.o ssl.c
Building ../bin/genhash
strip ../bin/genhash
Make complete
make[1]: Leaving directory `/home/soft/keepalived-1.2.12/genhash'
Make complete
# make install
make -C keepalived install
make[1]: Entering directory `/home/soft/keepalived-1.2.12/keepalived'
install -d /usr/local/keepalived/sbin
install -m 700 ../bin/keepalived /usr/local/keepalived/sbin/
install -d /usr/local/keepalived/etc/rc.d/init.d
install -m 755 etc/init.d/keepalived.init /usr/local/keepalived/etc/rc.d/init.d/keepalived
install -d /usr/local/keepalived/etc/sysconfig
install -m 755 etc/init.d/keepalived.sysconfig /usr/local/keepalived/etc/sysconfig/keepalived install -d /usr/local/keepalived/etc/keepalived/samples
install -m 644 etc/keepalived/keepalived.conf /usr/local/keepalived/etc/keepalived/
install -m 644 ../doc/samples/* /usr/local/keepalived/etc/keepalived/samples/
install -d /usr/local/keepalived/share/man/man5
install -d /usr/local/keepalived/share/man/man8
install -m 644 ../doc/man/man5/keepalived.conf.5 /usr/local/keepalived/share/man/man5 install -m 644 ../doc/man/man8/keepalived.8 /usr/local/keepalived/share/man/man8
make[1]: Leaving directory `/home/soft/keepalived-1.2.12/keepalived'
make -C genhash install
make[1]: Entering directory `/home/soft/keepalived-1.2.12/genhash'
install -d /usr/local/keepalived/bin
install -m 755 ../bin/genhash /usr/local/keepalived/bin/
install -d /usr/local/keepalived/share/man/man1
install -m 644 ../doc/man/man1/genhash.1 /usr/local/keepalived/share/man/man1
make[1]: Leaving directory `/home/soft/keepalived-1.2.12/genhash'
# cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
# cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/ \\ keepalived 启动脚本
# cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/ \\ keepalived启动脚本默认keepalived命令目录
# mkdir /etc/keepalived
# cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/ \\ keepalived配置文件目录
（或者修改keepalived执行命令的目录和keepalived配置文件的目录，
修改/etc/rc.d/init.d/keepalived, 寻找大约15行左右的. /etc/sysconfig/keepalived, 修改为：. /usr/local/keepalived/etc/sysconfig/keepalived, 即指向正确的文件位置\\这个修改根据你的实际目录修改
修改/usr/local/keepalived/etc/sysconfig/keepalived文件，设置正确的服务启动参数\\这项根据你的实际目录修改
KEEPALIVED_OPTIONS="-D -f /usr/local/keepalived/etc/keepalived/keepalived.conf"）
主机keepalived配置文件：
# vim /usr/local/keepalived/etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVA_DEVEL
}
vrrp_script chk_haproxy {
script "/usr/local/haproxy/sbin/check_haproxy.sh"
interval 2
weight 2
}
vrrp_instance VI_1 {
state MASTER
interface bond0
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
track_script {
chk_haproxy
}
virtual_ipaddress {
192.168.10.3
}
}
备机keepalived配置文件：
! Configuration File for keepalived
global_defs {
router_id LVA_DEVEL
}
vrrp_script chk_haproxy {
script "/usr/local/haproxy/sbin/check_haproxy.sh"
interval 2
weight 2
}
vrrp_instance VI_1 {
state BACKUP
interface bond0
virtual_router_id 51
priority 99
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
track_script {
chk_haproxy
}
virtual_ipaddress {
192.168.10.3
}
}