ELK测试部署方案

ELK Stack测试部署方案文档修订摘要亚联（天津）信息技术有限责任公司2016年8月目录一、安装前规划与准备 (3)1.1执行服务器环境初始化脚本 (3)1.2软件下载 (5)二、软件安装 (6)2.1.服务端软件安装 (6)2.1.1 Install JDK (6)2.1.2 Install Elasticsearch (6)2.1.3 Install Logstash (6)2.1.4 Install Redis (7)2.1.5 Install Kibana (8)2.2 客户端软件安装 (8)2.2.1 Install JDK (8)2.2.2 Install Logstash (8)三、Start Service (9)3.1 Start Redis (9)3.2 Start Elasticsearch (9)3.3 Start Kibana (10)四、Monitor Nginx Log (10)4.1 只监控access日志，，从文件获取将access日志导向服务端的redis (10)五、Monitor System Log (17)5.1Client rsyslog Install (If not installed) (17)5.2 The configuration of /etc/rsyslog.conf (17)5.3 Start the instance on the server (17)六、kibana Show 插件安装（可选操作，按个人需求安装与否，安装方式有变，请参考官方文档） (20)6.1Install Plugin (20)6.2 Start Elasticsearch (21)6.3 Kibana Usage (21)七、Redis-browser (22)7.1 Install Ruby (22)7.2 Install redis-browser (22)7.3 Start redis-browser (23)一、安装前规划与准备修改主机名；保证服务器通外网；yum源能使用1.1执行服务器环境初始化脚本change_env.sh# cat change_env.sh#!/bin/bash##this is init_env script#Modify limlt.confecho "* soft nofile 204800" >> /etc/security/limits.confecho "* hard nofile 204800" >> /etc/security/limits.conf#nprocecho "* soft nproc 10240" >> /etc/security/limits.d/90-nproc.conf echo "root soft nproc unlimited" >> /etc/security/limits.d/90-nproc.conf#Add crontabecho "00 00 * * * /usr/sbin/ntpdate -u 195.13.1.153" >> /var/spool/cron/root#ip_conntract# echo "net.ipv4.ip_conntrack_max = 1048576" >> /etc/sysctl.conf# echo "options ip_conntrack hashsize=131072" >> /etc/modprobe.confecho "options nf_conntrack hashsize=131072" >> /etc/modprobe.d/nf_conntrack.conf #Close iptablesservice iptables stopchkconfig iptables off#Close selinuxsetenforce 0/bin/sed -i '7s/enforcing/disabled/' /etc/selinux/config &> /dev/mull#Close sendmail# service sendmail stop# chkconfig sendmail off#charset#export LANG=zh_CN.UTF-8#echo "LANG=zh_CN.UTF-8" > /etc/sysconfig/i18n#Set vimsed -i "8 s/^/alias vi='vim'/" /root/.bashrc#echo 'syntax on' > /root/.vimrc #经验证此行没用#Add DNS Server#cat>>/etc/resolv.conf<<EOF#nameserver 202.106.0.20#nameserver 8.8.8.8#EOF#Close ipv6echo "options ipv6 disable=1" > /etc/modprobe.d/disable-ipv6.confecho "IPV6INIT=no" >> /etc/sysconfig/networkchkconfig ip6tables off#Modify historyecho 'export HISTTIMEFORMAT="%F %T `whoami`"' >> /etc/profilesed -i 's/HISTSIZE\=1000/HISTSIZE\=20000/' /etc/profile#Close default servicefor i in `ls /etc/rc5.d/S*`doCURSRV=`echo $i|cut -c 15-`echo $CURSRVcase $CURSRV in local | crond | network | sshd | certmonger | messagebus | haldaemon | rsyslog | irqbalance | acpid | cpuspeed | rpcbind | spice-vdagentd | udev-post)echo "Base services, Skip!";;*)echo "change $CURSRV to off"chkconfig --level 235 $CURSRV offservice $CURSRV stop;;esacdone#change init_classsed -i "s/^id:5:init/id:3:init/" /etc/inittab#change rm to /root/.trashmkdir -p /root/.trashcat >>/root/.bashrc <<EOFalias rm=trashalias r=trashalias rl='ls ~/.trash'alias ur=undelfileundelfile(){mv -i ~/.trash/\$@ ./}trash(){mv \$@ ~/.trash/}cleartrash(){read -p "clear sure?[n]" confirm[ \$confirm == 'y' ] || [ \$confirm == 'Y' ] && /bin/rm -rf ~/.trash/*}EOFReboot1.2软件下载Elk官方推荐版本组合（也是最新的），注意elk软件版本不可乱搭，最好使用官方推荐的版本组合！https://www.elastic.co/downloads本人下载的是源码的二、软件安装2.1.服务端软件安装# cd /opt/soft/# lselasticsearch-2.3.5.tar.gz jdk-8u101-linux-x64.rpm kibana-4.5.4-linux-x64.tar.gz logstash-2.3.4.tar.gz redis-3.2.3.tar.gz卸载系统自带java ：openjdk（本人习惯）# java -versionjava version "1.7.0_45"OpenJDK Runtime Environment (rhel-2.4.3.3.el6-x86_64 u45-b15)OpenJDK 64-Bit Server VM (build 24.45-b08, mixed mode)# rpm -qa | grep jdkjava-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64# rpm -e --nodeps java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64# rpm -e --nodeps java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_642.1.1 Install JDK# rpm -ivh jdk-8u101-linux-x64.rpm# java -version // 查看是否生效，生效了会显示java版本信息java version "1.8.0_101"Java(TM) SE Runtime Environment (build 1.8.0_101-b13)Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)2.1.2 Install Elasticsearch# groupadd -g 92 elasticsearch# useradd -g 92 -u 92 elasticsearch# tar xf elasticsearch-2.3.5.tar.gz -C /opt/ //解压elasticsearch至指定目录# ln -sv /opt/elasticsearch-2.3.5/bin/elasticsearch /usr/bin/ //将elasticsearch可执行文件链接至/usr/bin目录，方便在任何目录下都可直接执行`elasticsearch`# chown -R elasticsearch.elasticsearch /opt/elasticsearch-2.3.5/2.1.3 Install Logstash# ln -sv /opt/logstash-2.3.4/bin/logstash /usr/bin/ //将logstash可执行文件链接至/usr/bin 目录，方便在任何目录下都可直接执行`logstash`2.1.4 Install Redis# yum –y install tcl gcc //安装测试依赖工具# tar xf redis-3.2.3.tar.gz //解压至当前目录# cd redis-3.2.3//切换至redis源码目录# make //编译# make test //有可能会失败，只是看看，不用在意# make install //安装# mkdir /opt/redis/{db,conf} -pv //创建redis安装目录# cp redis.conf /opt/redis/conf/ //复制配置文件至redis安装目录# cd src# cp redis-benchmark redis-check-aof redis-cli redis-server mkreleasehdr.sh /opt/redis/ //复制各文件至redis安装目录# ln -sv /opt/redis/redis-cli /usr/bin/ //将redis-cli可执行文件链接至/usr/bin目录，方便在任何目录下都可直接执行`redis-cli`# vim /opt/redis/conf/redis.conf //修改redis.conf 中的 `daemonize`为`yes`,让server以守护进程在后台执行，这一步可以不做，因为后面要执行的脚本会自动创建这个文件，且这个值会设置为`yes`daemonize yesmake install仅仅在你的系统上安装了二进制文件，不会替你默认配置init 脚本和配置文件，为了把它用在生产环境而安装它，在源码目录的utils目录下Redis为系统提供了这样的一个脚本install_server.sh# cd /opt/soft/redis-3.2.3/utils# ./install_server.sh //执行sh格式的安装脚本Welcome to the redis service installerThis script will help you easily set up a running redis serverPlease select the redis port for this instance: [6379]Selecting default: 6379Please select the redis config file name [/etc/redis/6379.conf] /opt/redis/conf/redis.conf Please select the redis log file name [/var/log/redis_6379.log]Selected default - /var/log/redis_6379.logPlease select the data directory for this instance [/var/lib/redis/6379] /opt/redis/db/6379.db Please select the redis executable path [/usr/local/bin/redis-server]Selected config:Port : 6379Config file : /opt/redis/conf/redis.confLog file : /var/log/redis_6379.logData dir : /opt/redis/db/6379.dbExecutable : /usr/local/bin/redis-serverCli Executable : /usr/local/bin/redis-cliIs this ok? Then press ENTER to go on or Ctrl-C to abort.Copied /tmp/6379.conf => /etc/init.d/redis_6379Installing service...Successfully added to chkconfig!Successfully added to runlevels 345!Starting Redis server...Installation successful!# chkconfig --add redis_6379 //将redis加入系统服务# chkconfig redis_6379 on //加入开机启动# vim /opt/redis/conf/redis.confrequirepass Carsing2582# //设置密码【坑1】：这步先不做，如果做了后面的测试连接redis的时候就需要认证）bind 0.0.0.0 //redis要打开远程访问允许，在 redis.conf 中 bind 这一行改成 0.0.0.0(运行任何机器连接，为了测试先这样)【此为坑2】# /etc/init.d/redis_6379 restart# ss -an| grep 6379 //端口已经被监听，证明服务已提供LISTEN 0 128 127.0.0.1:6379 *:*TIME-WAIT 0 0 127.0.0.1:6379 127.0.0.1:345062.1.5 Install Kibana# groupadd -g 56 kibana# useradd -g 56 -u 56 kibana# tar xf kibana-4.5.4-linux-x64.tar.gz -C /opt/ //解压kibana至指定目录# ln -sv /opt/kibana-4.5.4-linux-x64/bin/kibana /usr/bin/ //将kibana可执行文件链接至/usr/bin目录，方便在任何目录下都可直接执行`kibana`# chown -R kibana.kibana /opt/kibana-4.5.4-linux-x64/2.2 客户端软件安装2.2.1 Install JDK同样卸载openjdk（不再赘述）# rpm -ivh jdk-8u101-linux-x64.rpm# java -version // 查看是否生效，生效了会显示java版本信息java version "1.8.0_101"Java(TM) SE Runtime Environment (build 1.8.0_101-b13)Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)2.2.2 Install Logstash# ln -sv /opt/logstash-2.3.4/bin/logstash /usr/bin/ //将logstash可执行文件链接至/usr/bin 目录，方便在任何目录下都可直接执行`logstash`三、Start Service服务端3.1 Start Redis# /etc/init.d/redis_6379 start# netstat -tnlp //查看是否有6379端口3.2 Start Elasticsearch# vi /opt/elasticsearch-2.3.5/config/elasticsearch.yml //在文件最后加入: es_cluster: node0network.host: discovery.zen.ping.unicast.hosts: [""]elasticsearch只能以普通用户运行# su -l -c "su elasticsearch /opt/elasticsearch-2.3.5/bin/elasticsearch >/dev/null 2>&1 &" //启动并放至后台运行# vim /etc/rc.local //上面命令复制，设置开机启动# netstat -tnlp //ElasticSearch默认的对外服务的HTTP端口是9200，节点间交互的TCP端口是9300，注意打开tcp端口# exit //退出当前用户http://60.32.0.23:9200//可以看到如下信息3.3 Start Kibana# vi /opt/kibana-4.5.4-linux-x64/config/kibana.yml //找到这2行，修改为server.host: "63.32.0.23"elasticsearch.url: "http://63.32.0.23:9200"启动并放至后台运行su -l -c "su kibana /opt/kibana-4.5.4-linux-x64/bin/kibana >/dev/null 2>&1 &"# vim /etc/rc.local //复制添加设置开机启动# netstat -tnlp //启用端口为5601http://60.32.0.23:5601//可以看到网页内容四、Monitor Nginx Log4.1 只监控access日志，，从文件获取将access日志导向服务端的redis客户端安装一个最简单的nginx，目的是为了产生访问日志。