前些天发现了一个巨牛的人工智能学习网站通俗易懂风趣幽默忍不住分享一下给大家。点击跳转到网站。http://my.oschina.net/itblog/blog/547250http://baidu.blog.51cto.com/71938/16767981、ELK由ElasticSearch、Logstash和Kiabana三个开源工具组成。官方网站https://www.elastic.co/products• Elasticsearch是个开源分布式搜索引擎它的特点有分布式零配置自动发现索引自动分片索引副本机制restful风格接口多数据源自动搜索负载等。• Logstash是一个完全开源的工具他可以对你的日志进行收集、过滤并将其存储供以后使用如搜索。• Kibana 也是一个开源和免费的工具它Kibana可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面可以帮助您汇总、分析和搜索重要数据日志。• 画了一个ELK工作的原理图• 如图Logstash收集AppServer产生的Log并存放到ElasticSearch集群中而Kibana则从ES集群中查询数据生成图表再返回给Browser。2、环境Linux:Elasticsearchelasticsearch-2.3.4.tar.gzLogstashlogstash-2.3.4.tar.gzKibanakibana-4.5.2-linux-x64.tar.gzJdkjdk-8u91-linux-x64.tar.gz3、jdk的安装mkdir -p /usr/local/java/tar -zxvf jdk-8u91-linux-x64.tar.gz -C /usr/local/java/vim /etc/profile加入JAVA_HOME/usr/local/java/jdk1.8.0_91PATHJ A V A H O M E / b i n : JAVA_HOME/bin:JAVAH​OME/bin:PATHCLASSPATH.:J A V A H O M E / l i b / d t . j a r : JAVA_HOME/lib/dt.jar:JAVAH​OME/lib/dt.jar:JAVA_HOME/lib/tools.jarexport JAVA_HOMEexport PATHexport CLASSPATH执行. /etc/profile #生效配置java -version4、安装配置Elasticsearch4.1、安装配置Elasticsearchtar -zxvf elasticsearch-2.3.4.tar.gzcd elasticsearch-2.3.4然后编辑ES的配置文件vim config/elasticsearch.yml修改以下配置项cluster.name: wennode.name: node-1path.data: /path/to/datapath.logs: /path/to/logs当前hostname或IP这里使用ipnetwork.host: 10.10.20.210http.port: 92004.2、启动elasticsearch./bin/elasticsearch报错不能使用root用户启动elasticsearch解决方法: http://my.oschina.net/topeagle/blog/591451?fromerrmzOr2qzZ将/data/soft/ch/elasticsearch-2.3.4/所属用户和组改为另外一个非root账户:4.2.1、创建esgp组groupadd esgp4.2.2、创建用户es 添加到esgp组设置用户的密码为es123useradd es -g esgp -p es1234.2.3、更改elasticsearch-2.3.4文件夹及内部文件的所属用户及组为es:esgpchown -R es:esgp elasticsearch-2.3.44.2.4、切换到es用户su es4.2.5、启动elasticsearch./elasticsearch-2.3.4/bin/elasticsearch修改/path/to/logs/和/path/to/data/的权限设置/path的用户为es,组为esgpsu rootchown -R es:esgp /path再启动elasticsearch使用ctrlC停止。当然也可以使用后台进程的方式启动ES./elasticsearch-2.3.4/bin/elasticsearch –d或者./elasticsearch-2.3.4/bin/elasticsearch 4.2.6、可以打开页面10.10.20.210:920将会看到以下内容5、安装elasticsearch的插件elasticsearch-headhttps://github.com/mobz/elasticsearch-headhttp://blog.csdn.net/july_2/article/details/244819355.1、插件安装方法15.1.1.#./elasticsearch-2.3.4/bin/plugin -install mobz/elasticsearch-head5.1.2.运行es5.1.3.打开http://10.10.20.210:9200/_plugin/head/5.2、插件安装方法25.2.1.https://github.com/mobz/elasticsearch-head下载zip 解压5.2.2.建立elasticsearch-2.3.4/plugins/head/_site文件mkdir -p ./elasticsearch-2.3.4/plugins/head/_site5.2.3.将解压后的elasticsearch-head-master文件夹下的文件copy到_sitesu root #切换到root用户unzip elasticsearch-head-master.zip #解压cp -r /data/soft/ch/elasticsearch-head-master/* /data/soft/ch/elasticsearch-2.3.4/plugins/head/_site/5.2.4.运行essu es #切换到es用户/data/soft/ch/elasticsearch-2.3.4/bin/elasticsearch会报错如上拷贝plugin-descriptor.properties文件cp /data/soft/ch/elasticsearch-2.3.4/plugins/head/_site/plugin-descriptor.properties /data/soft/ch/elasticsearch-2.3.4/plugins/head/再启动elasticsearch5.2.5.打开http://10.10.20.210:9200/_plugin/head/6、安装logstash-2.3.4.tar.gztar -zxvf logstash-2.3.4.tar.gzcd logstash-2.3.4mkdir configtouch file_to_es.conf #创建file_to_es.conf文件vim file_to_es.conf输入内容#For detail structure of this file#Set: https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.htmlinput {For detail config for file as input,See: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.htmlfile{path “/data/java/apache-tomcat-ware/logs/catalina.out”}For detail config for log4j as input,See: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html#log4j {mode “server”host “10.10.20.210”port 4567#}}filter {#Only matched data are send to output.}output {For detail config for elasticsearch as output,See: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.htmlelasticsearch {action “index” #The operation on EShosts “10.10.20.210:9200” #ElasticSearch host, can be array.index “ware-manager-dev” #The index to write data to.}}启动logstash./logstash-2.3.4/bin/logstash agent -f ./logstash-2.3.4/config/file_to_es.conf后台启动加上./logstash-2.3.4/bin/logstash agent -f ./logstash-2.3.4/config/file_to_es.conf 查看进程可以在http://10.10.20.210:9200/_plugin/head/中看到自动创建了ware-manager-dev这个索引里面保存的是ware-manager这个项目的日志注意如果在同一个服务器对不同的文件设置不同的索引那么可以增加配置文件然后制定配置文件启动即可./logstash-2.3.4/bin/logstash agent -f ./logstash-2.3.4/config/order_to_es.conf #./logstash-2.3.4/bin/logstash agent -f ./logstash-2.3.4/config/ware_to_es.conf order_to_es.conf的内容 # For detail structure of this file # Set: https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html input { file{ path /data/applogs/**order**/wrapper.*.log } # For detail config for log4j as input, # See: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html #log4j { # mode server # host 10.10.20.206 # port 4567 #} } filter { #Only matched data are send to output. } output { # For detail config for elasticsearch as output, # See: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html elasticsearch { action index #The operation on ES hosts 10.10.20.206:9200 #ElasticSearch host, can be array. index **order-server-test** #The index to write data to. } } ware_to_es.conf的内容 # For detail structure of this file # Set: https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html input { file{ path /data/applogs/**ware**/wrapper.*.log } # For detail config for log4j as input, # See: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html #log4j { # mode server # host 10.10.20.206 # port 4567 #} } filter { #Only matched data are send to output. } output { # For detail config for elasticsearch as output, # See: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html elasticsearch { action index #The operation on ES hosts 10.10.20.206:9200 #ElasticSearch host, can be array. index **ware-server-test** #The index to write data to. } }7、安装kibanatar -zxvf kibana-4.5.2-linux-x64.tar.gzcd kibana-4.5.2-linux-x64vim config/kibana.yml修改内容如下server.port: 5601server.host: “10.10.20.210”elasticsearch.url: “http://10.10.20.210:9200”kibana.index: “.kibana”注意其中server.host是安装kibana服务的地址elasticsearch.url为elasticsearch安装的地址启动kibana/data/soft/ch/kibana-4.5.2-linux-x64/bin/kibana可以通过http://10.10.20.206:5601访问到。也可以后台启动kibana(加上)/data/soft/ch/kibana-4.5.2-linux-x64/bin/kibana注意这时加上了虽然执行了后台启动但是还是有日志打印出来使用ctrlc可以退出。但是如果直接关闭了Xshell,这时服务也会停止访问http://10.10.20.206:5601就失败了。解决方法执行了/data/soft/ch/kibana-4.5.2-linux-x64/bin/kibana 命令后不使用ctrlc去退出日志而是使用exit;这样即使关闭了Xshell窗口服务也不会挂了。配置kibana这时在elasticsearch中多了一个kibana的索引为了后续使用Kibana需要配置至少一个Index名字或者Pattern它用于在分析时确定ES中的Index。这里我输入之前配置的Index 名字ware-manager-devKibana会自动加载该Index下doc的field并自动选择合适的field用于图标中的时间字段点击Discover查看日志设置只查看message信息:设置自动刷新更多配置请查看http://my.oschina.net/itblog/blog/547250