ElasticSearch及其插件安装

ElasticSearch,强大的数据搜索应用,可以当成数据库来使用,接下来演示如何安装和安装它的插件。

安装Elasticsearch

  1. 安装Java
    Java安装就不多说了,记得安装完后设置JAVA_HOME环境变量,我使用的1.8版本。
  2. 下载
    https://www.elastic.co/downloads/elasticsearch 下载最新版本,解压,假设elasticsearch解压后的根目录为$ES_HOME。
  3. 修改配置
    配置文件在 $ES_HOME/config/elasticsearch.yml,设置基本的集群名称,节点名称,和网络监听地址:

    1
    2
    3
    cluster.name: myes
    node.name: n1
    network.host: 0.0.0.0
  4. 运行
    进入$ES_HOME/bin目录,执行elasticsearch即可。

  5. 验证
    打开:http://localhost:9200/
    install-ok

我使用的是5.x版本,以前的插件安装方法,现在很多用不上了,先看看能在线安装的插件:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
ESHOME # bin/elasticsearch-plugin install -h
Install a plugin
The following official plugins may be installed by name:
analysis-icu
analysis-kuromoji
analysis-phonetic
analysis-smartcn
analysis-stempel
analysis-ukrainian
discovery-azure-classic
discovery-ec2
discovery-file
discovery-gce
ingest-attachment
ingest-geoip
ingest-user-agent
lang-javascript
lang-python
mapper-attachments
mapper-murmur3
mapper-size
repository-azure
repository-gcs
repository-hdfs
repository-s3
store-smb
x-pack

Elasticsearch运行的服务器上,需要重新修改一下配置,对所有插件都是这样,以便支持跨域和跨主机访问:

1
2
3
4
ESHOME # vi config/elasticsearch.yml
http.cors.enabled: true # 开启跨域访问
http.cors.allow-origin: "*" # 允许任意来源的跨域访问
network.host: 0.0.0.0 # 配置BIND地址,默认是127.0.0.1,不能跨主机访问

不然会出现类似错误:

1
No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://do:8000' is therefore not allowed access.

安装elasticsearch-head docker版本

此插件用于浏览当前有哪些indices,支持语句执行,并不是很易用,参考文档:https://github.com/mobz/elasticsearch-head
我将elasticsearch-head安装到另外一台机器,使用docker一条命令解决:

1
2
3
4
5
6
7
8
9
10
11
do ~ # docker run -p 9100:9100 mobz/elasticsearch-head:5
Unable to find image 'mobz/elasticsearch-head:5' locally
5: Pulling from mobz/elasticsearch-head
32734c6d04f6: Pull complete
...
61c321193443: Pull complete
Digest: sha256:19cf8b27a5e71a79af27c29ff20ffa18f27306e2489dcfd88b9ec5708f2bffea
Status: Downloaded newer image for mobz/elasticsearch-head:5
Running "connect:server" (connect) task
Waiting forever...
Started connect web server on http://localhost:9100

从浏览器打开Elasticsearch-head监听的9100端口,连接服务器地址,即可:
elasticsearch-head-ui

安装elasticsearch-head windows版本

相比docker安装,安装在windows就麻烦多了,先安装依赖。

  1. 安装NodeJS
    head插件是NodeJS源码,从 https://nodejs.org/en/ 下载并安装。安装后记得设置NODE_HOME到环境变量。

    我的环境,需要设置代理才能使用nodejs的npm来安装软件:

    1
    2
    npm config set proxy http://do:3128
    npm config set https-proxy http://do:3128
  2. 安装grunt
    grunt是Javascript的构建工具,有点像maven之于Java,因为head插件是NodeJS源码,所以需要这东西。

    1
    2
    3
    4
    5
    6
    7
    > npm install -g grunt-cli
    C:\Users\w7\AppData\Roaming\npm\grunt -> C:\Users\w7\AppData\Roaming\npm\node_modules\grunt-cli\bin\grunt
    + grunt-cli@1.2.0
    added 16 packages in 60.156s
    > grunt -version
    grunt-cli v1.2.0
  3. 安装head

    1. 下载
      https://github.com/mobz/elasticsearch-head 下载源码,解压。

    2. 修改源码
      如果想跨主机访问,需要修改:

      修改 Gruntfile.js,添加hostname: ‘*’:

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      connect: {
      server: {
      options: {
      port: 9100,
      hostname: '*',
      base: '.',
      keepalive: true
      }
      }
      }

      修改_site/app.js

      1
      this.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http://localhost:9200";

      把localhost修改成你es服务器地址,例如:

      1
      this.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http://esserver:9200";
    3. 修改ElasticSearch
      ElasticSearch配置文件,添加head需要的选项:

      1
      2
      http.cors.enabled: true
      http.cors.allow-origin: "*"

      启动ElasticSearch。

    4. 安装elasticsearch-head
      进入 elasticsearch-head 源码目录,执行安装。
      第一次执行报错:

      1
      2
      3
      4
      5
      6
      7
      8
      9
      > npm install
      npm WARN elasticsearch-head@0.0.0 license should be a valid SPDX license express
      ion
      npm ERR! code ECONNRESET
      npm ERR! errno ECONNRESET
      npm ERR! network request to https://registry.npm.taobao.org/phantomjs-prebuilt/d
      ownload/phantomjs-prebuilt-2.1.15.tgz failed, reason: read ECONNRESET
      npm ERR! network This is a problem related to network connectivity.

      看样子是我之前设置过taobao作为registry,有问题,删除掉,用官方版本,还是有问题,网上说还是网络不好,又换行淘宝源:

      1
      2
      3
      > npm config set registry https://registry.npm.taobao.org
      > npm config list
      registry = "https://registry.npm.taobao.org/"

      还是不行,后来发现是网络不好,单独打开 https://github.com/Medium/phantomjs/releases/download/v2.1.1/phantom
      js-2.1.1-windows.zip 也是几byte/s,于是找了个csdn上的版本,解压,将其bin目录放到PATH环境变量下,安装成功:

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      > npm install
      - phantomjs-prebuilt@2.1.15 install D:\Work\CodeRead\DB\es\elasticsearch-head-master\node_modules\phantomjs-prebuilt
      - node install.js
      Considering PhantomJS found at D:\Software\NodeJS\phantomjs-2.1.1\bin\phantomjs.EXE
      Found PhantomJS at D:\Software\NodeJS\phantomjs-2.1.1\bin\phantomjs.EXE ...verifying
      Writing location.js file PhantomJS is already installed on PATH at D:\Software\NodeJS\phantomjs-2.1.1\bin\phantomjs.EXE
      npm notice created a lockfile as package-lock.json. You should commit this file.
      npm WARN elasticsearch-head@0.0.0 license should be a valid SPDX license expression
      npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.1.2 (node_modules\fsevents):
      npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@
      1.1.2: wanted {"os":"darwin","arch":"any"} (current: {"os":"win32","arch":"x64"})
      added 190 packages in 28.8s

      终于可以执行:

      1
      2
      3
      4
      > grunt server
      Running "connect:server" (connect) task
      Waiting forever...
      Started connect web server on http://localhost:9100

安装cerebro

cerebro提供的功能和head差不多,但是UI设计要好一些,直接到这里下载二进制: https://github.com/lmenezes/cerebro/releases

要求JAVA 8,我的CentOS之前安装的是JAVA 7:

1
2
3
4
~/cerebro-0.6.7 # java -version
java version "1.7.0_151"
OpenJDK Runtime Environment (rhel-2.6.11.0.el6_9-x86_64 u151-b00)
OpenJDK 64-Bit Server VM (build 24.151-b00, mixed mode)

先卸载掉JAVA 7:

1
2
3
~/cerebro-0.6.7 # rpm -e --nodeps `rpm -qa | grep java`
~/cerebro-0.6.7 # java -version
-bash: /usr/bin/java: No such file or directory

然后直接去官网下载JDK 8(也可以只下载JRE)最新版本: http://www.oracle.com/technetwork/java/javase/downloads/index.html
下载的rpm包,安装:

1
2
3
4
5
~/cerebro-0.6.7 # rpm -ivh jdk-8u144-linux-x64.rpm
~/cerebro-0.6.7 # java -version
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

运行:

1
2
3
~/cerebro-0.6.7 # bin/cerebro
[info] play.api.Play - Application started (Prod)
[info] p.c.s.NettyServer - Listening for HTTP on /0:0:0:0:0:0:0:0:9000

前端访问9000端口,连接到ES地址即可,效果如图:
cerebro

安装bigdesk

bigdesk和前面两个的作用不太一样,主要用来将_cluster/stats,_cluster/health, _cluster/state/nodes,routing_table,routing_nodes几个API的几个可视化展示出来,包括CPU/内存/Indices/Thread Pools/OS & JVM & Process & Transport,这个版本支持5.x系列,下载:

1
git clone https://github.com/hlstudio/bigdesk

将_site目录移动到web服务器下或者直接用Python创建一个简单web服务器都可以运行,但是不能用https:

1
python -m SimpleHTTPServer

根据控制台输出,可以看出默认启动在8000端口,如果你的防火墙阻止了8000端口,还需要放开防火墙,例如我的CentOS,我直接删除了拒绝规则:

1
2
3
4
5
do ~ # iptables --line-number -L -nv
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
12 46598 4092K REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
do ~ # iptables -D INPUT 12

网页打开后,指定你要连接到的服务器地址,马上就会出数据:
bigdesk

安装kibana

kibana的主要用于数据可视化,将数据化成图标展示给用户,同样,也使用docker运行:

1
docker run -v /root/esvolumn/public/kibana.104.yml:/usr/share/kibana/config/kibana.yml -p 5601:5601 kibana:5.3

上面,我将配置文件/usr/share/kibana/config/kibana.yml用外部的覆盖了,可以参考 https://www.elastic.co/guide/en/kibana/5.6/settings.html 修改 kibana.yml,例如设置要bind的地址为0.0.0.0,连接的URL为你想连接的远程服务器:

1
2
3
server.host: "0.0.0.0"
elasticsearch.url: "http://200.200.88.104:9200"
elasticsearch.requestTimeout: 300000

安装好后出现错误:
kibana-error
原因是ES服务器端上次创建过.kibana index,将其删除:

1
2
3
# curl http://127.0.0.1:9200/_cat/indices/.kibana?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
red open .kibana 9qewgRIcSpSisOyCjI5Yfg 1 1

我删除超时了,但是的确成功了:

1
2
3
4
5
6
7
8
9
# curl -XDELETE http://127.0.0.1:9200/.kibana
{"acknowledged":false}
# curl http://127.0.0.1:9200/_cat/indices/.kibana?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
red open .kibana 9qewgRIcSpSisOyCjI5Yfg 1 1
# curl -XDELETE http://127.0.0.1:9200/.kibana
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","index_uuid":"9qewgRIcSpSisOyCjI5Yfg","index":".kibana"}],"type":"index_not_found_exception","reason":"no such index","index_uuid":"9qewgRIcSpSisOyCjI5Yfg","index":".kibana"},"status":404}

重新访问,刚才显示Elasticsearch is still initializing the kibana index的地方,变成了如下错误:

1
plugin:elasticsearch@5.3.2 [search_phase_execution_exception] all shards failed

过几分钟,此错误消失,可以使用了:
kibana