elasticsearch 集群

集群内的原理

介绍 cluster 、 node 、 shard 等常用术语,Elastisearch 的扩容机制, 以及如何处理硬件故障的内容。

ElasticSearch 的主旨是随时可用和按需扩容。 而扩容可以通过购买性能更强大( 垂直扩容 ,或 纵向扩容 ) 或者数量更多的服务器( 水平扩容 ,或 横向扩容 )来实现。

虽然 Elasticsearch 可以获益于更强大的硬件设备,但是垂直扩容是有极限的。 真正的扩容能力是来自于水平扩容–为集群添加更多的节点,并且将负载压力和稳定性分散到这些节点中。

(1) 空集群

一个运行中的 Elasticsearch 实例称为一个 节点,而集群是由一个或者多个拥有相同 cluster.name 配置的节点组成, 它们共同承担数据和负载的压力。当有节点加入集群中或者从集群中移除节点时,集群将会重新平均分布所有的数据。

(2) 集群健康

查看集群健康状态

http://localhost:9200/_cluster/health?pretty

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ curl -X GET "localhost:9200/_cluster/health?pretty"
{
"cluster_name" : "my-application",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 5,
"active_shards" : 5,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 5,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 50.0
}

status 字段指示着当前集群在总体上是否工作正常。它的三种颜色含义如下:
green 所有的主分片和副本分片都正常运行。
yellow 所有的主分片都正常运行,但不是所有的副本分片都正常运行。
red 有主分片没能正常运行。

(3) 添加索引

我们往 Elasticsearch 添加数据时需要用到 索引 —— 保存相关数据的地方。 索引实际上是指向一个或者多个物理 分片 的 逻辑命名空间 。

一个 分片 是一个底层的 工作单元 ,它仅保存了全部数据中的一部分。 在分片内部机制中,我们将详细介绍分片是如何工作的,而现在我们只需知道一个分片是一个 Lucene 的实例,以及它本身就是一个完整的搜索引擎。 我们的文档被存储和索引到分片内,但是应用程序是直接与索引而不是与分片进行交互。

Elasticsearch 是利用分片将数据分发到集群内各处的。分片是数据的容器,文档保存在分片内,分片又被分配到集群内的各个节点里。 当你的集群规模扩大或者缩小时, Elasticsearch 会自动的在各节点中迁移分片,使得数据仍然均匀分布在集群里。

一个分片可以是 主 分片或者 副本 分片。 索引内任意一个文档都归属于一个主分片,所以主分片的数目决定着索引能够保存的最大数据量。

技术上来说,一个主分片最大能够存储 Integer.MAX_VALUE - 128 个文档,但是实际最大值还需要参考你的使用场景:包括你使用的硬件, 文档的大小和复杂程度,索引和查询文档的方式以及你期望的响应时长。

一个副本分片只是一个主分片的拷贝。 副本分片作为硬件故障时保护数据不丢失的冗余备份,并为搜索和返回文档等读操作提供服务。
在索引建立的时候就已经确定了主分片数,但是副本分片数可以随时修改。

索引在默认情况下会被分配5个主分片

创建一个blogs索引,设置3个主分片和一份副本(每个主分片拥有一个副本分片)

设置分片数和副本数

1
2
3
4
5
6
7
8
9
10
11
12
13
$ curl -X PUT "localhost:9200/blogs?pretty" -H 'Content-Type: application/json' -d'
> {
> "settings" : {
> "number_of_shards" : 3,
> "number_of_replicas" : 1
> }
> }
> '
{
"acknowledged" : true,
"shards_acknowledged" : true,
"index" : "blogs"
}
1
2
3
4
5
6
7
8
9
10
WKQ@WKQ-PC MINGW64 /c/ProfessionalSoftWare/ElasticSearch/elasticsearch-5.6.10
$ curl -X PUT "localhost:9200/blogs" -H 'Content-Type: application/json' -d'
> {
> "settings" : {
> "number_of_shards" : 3,
> "number_of_replicas" : 1
> }
> }
> '
{"acknowledged":true,"shards_acknowledged":true,"index":"blogs"}

(4) 添加故障转移

当集群中只有一个节点在运行时,意味着会有一个单点故障问题——没有冗余。 幸运的是,我们只需再启动一个节点即可防止数据丢失。

为了测试第二个节点启动后的情况,你可以在同一个目录内,完全依照启动第一个节点的方式来启动一个新节点。多个节点可以共享同一个目录。

当你在同一台机器上启动了第二个节点时,只要它和第一个节点有同样的 cluster.name 配置,它就会自动发现集群并加入到其中。 但是在不同机器上启动节点的时候,为了加入到同一集群,你需要配置一个可连接到的单播主机列表。

(4.1) 自己搭建一个3个节点的集群

elasticsearch-5.6.10解压三份,然后把3个的配置文件里 cluster.name 全改成统一的一个名字,这儿我写的是 elasticsearch_production ,然后分别启动3个ES

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
λ elasticsearch
[2018-07-22T17:38:20,925][INFO ][o.e.n.Node ] [elasticsearch_node_001] initializing ...
[2018-07-22T17:38:21,034][INFO ][o.e.e.NodeEnvironment ] [elasticsearch_node_001] using [1] data paths, mounts [[disk-C (C:)]], net usable_space [81.2gb], net total_space [118.7gb], spins? [unknown], types [NTFS]
[2018-07-22T17:38:21,034][INFO ][o.e.e.NodeEnvironment ] [elasticsearch_node_001] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-07-22T17:38:21,164][INFO ][o.e.n.Node ] [elasticsearch_node_001] node name [elasticsearch_node_001], node ID [gjy4N2RCQ4mLxp1lxdyZ8w]
[2018-07-22T17:38:21,164][INFO ][o.e.n.Node ] [elasticsearch_node_001] version[5.6.10], pid[9412], build[b727a60/2018-06-06T15:48:34.860Z], OS[Windows 10/10.0/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_131/25.131-b11]
[2018-07-22T17:38:21,164][INFO ][o.e.n.Node ] [elasticsearch_node_001] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Delasticsearch, -Des.path.home=C:\ProfessionalSoftWare\ElasticSearch\elasticsearch-5.6.10]
[2018-07-22T17:38:22,254][INFO ][o.e.p.PluginsService ] [elasticsearch_node_001] loaded module [aggs-matrix-stats]
[2018-07-22T17:38:22,256][INFO ][o.e.p.PluginsService ] [elasticsearch_node_001] loaded module [ingest-common]
[2018-07-22T17:38:22,257][INFO ][o.e.p.PluginsService ] [elasticsearch_node_001] loaded module [lang-expression]
[2018-07-22T17:38:22,258][INFO ][o.e.p.PluginsService ] [elasticsearch_node_001] loaded module [lang-groovy]
[2018-07-22T17:38:22,258][INFO ][o.e.p.PluginsService ] [elasticsearch_node_001] loaded module [lang-mustache]
[2018-07-22T17:38:22,258][INFO ][o.e.p.PluginsService ] [elasticsearch_node_001] loaded module [lang-painless]
[2018-07-22T17:38:22,259][INFO ][o.e.p.PluginsService ] [elasticsearch_node_001] loaded module [parent-join]
[2018-07-22T17:38:22,259][INFO ][o.e.p.PluginsService ] [elasticsearch_node_001] loaded module [percolator]
[2018-07-22T17:38:22,259][INFO ][o.e.p.PluginsService ] [elasticsearch_node_001] loaded module [reindex]
[2018-07-22T17:38:22,260][INFO ][o.e.p.PluginsService ] [elasticsearch_node_001] loaded module [transport-netty3]
[2018-07-22T17:38:22,260][INFO ][o.e.p.PluginsService ] [elasticsearch_node_001] loaded module [transport-netty4]
[2018-07-22T17:38:22,261][INFO ][o.e.p.PluginsService ] [elasticsearch_node_001] no plugins loaded
[2018-07-22T17:38:24,294][INFO ][o.e.d.DiscoveryModule ] [elasticsearch_node_001] using discovery type [zen]
[2018-07-22T17:38:24,844][INFO ][o.e.n.Node ] [elasticsearch_node_001] initialized
[2018-07-22T17:38:24,844][INFO ][o.e.n.Node ] [elasticsearch_node_001] starting ...
[2018-07-22T17:38:25,488][INFO ][o.e.t.TransportService ] [elasticsearch_node_001] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2018-07-22T17:38:28,621][INFO ][o.e.c.s.ClusterService ] [elasticsearch_node_001] new_master {elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)[, ]
[2018-07-22T17:38:28,887][INFO ][o.e.g.GatewayService ] [elasticsearch_node_001] recovered [2] indices into cluster_state
[2018-07-22T17:38:29,246][INFO ][o.e.c.r.a.AllocationService] [elasticsearch_node_001] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[megacorp][4], [megacorp][3], [megacorp][0]] ...]).
[2018-07-22T17:38:29,360][INFO ][o.e.h.n.Netty4HttpServerTransport] [elasticsearch_node_001] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}
[2018-07-22T17:38:29,360][INFO ][o.e.n.Node ] [elasticsearch_node_001] started
[2018-07-22T17:38:56,302][INFO ][o.e.c.s.ClusterService ] [elasticsearch_node_001] added {{elasticsearch_node_002}{3ezLh-3jTB-c0pQXFXnr5Q}{Y4i8X_t7QhWDBoP6MoDuvw}{127.0.0.1}{127.0.0.1:9301},}, reason: zen-disco-node-join
[2018-07-22T17:38:56,505][WARN ][o.e.d.z.ElectMasterService] [elasticsearch_node_001] value for setting "discovery.zen.minimum_master_nodes" is too low. This can result in data loss! Please set it to at least a quorum of master-eligible nodes (current value: [-1], total number of master-eligible nodes used for publishing in this round: [2])
[2018-07-22T17:38:58,989][INFO ][o.e.c.r.a.AllocationService] [elasticsearch_node_001] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[megacorp][4]] ...]).
[2018-07-22T17:39:43,600][INFO ][o.e.c.s.ClusterService ] [elasticsearch_node_001] added {{elasticsearch_node_003}{upVg0NksTRi8LH8mj9xqpg}{ydGaCWORSlWf09TTjpUBQg}{127.0.0.1}{127.0.0.1:9302},}, reason: zen-disco-node-join
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
C:\ProfessionalSoftWare\ElasticSearch\elasticsearch-5.6.10-node2\bin
λ elasticsearch
[2018-07-22T17:38:47,851][INFO ][o.e.n.Node ] [elasticsearch_node_002] initializing ...
[2018-07-22T17:38:47,960][INFO ][o.e.e.NodeEnvironment ] [elasticsearch_node_002] using [1] data paths, mounts [[disk-C (C:)]], net usable_space [78.9gb], net total_space [118.7gb], spins? [unknown], types [NTFS]
[2018-07-22T17:38:47,960][INFO ][o.e.e.NodeEnvironment ] [elasticsearch_node_002] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-07-22T17:38:47,960][INFO ][o.e.n.Node ] [elasticsearch_node_002] node name [elasticsearch_node_002], node ID [3ezLh-3jTB-c0pQXFXnr5Q]
[2018-07-22T17:38:47,960][INFO ][o.e.n.Node ] [elasticsearch_node_002] version[5.6.10], pid[13072], build[b727a60/2018-06-06T15:48:34.860Z], OS[Windows 10/10.0/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_131/25.131-b11]
[2018-07-22T17:38:47,960][INFO ][o.e.n.Node ] [elasticsearch_node_002] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Delasticsearch, -Des.path.home=C:\ProfessionalSoftWare\ElasticSearch\elasticsearch-5.6.10-node2]
[2018-07-22T17:38:49,179][INFO ][o.e.p.PluginsService ] [elasticsearch_node_002] loaded module [aggs-matrix-stats]
[2018-07-22T17:38:49,179][INFO ][o.e.p.PluginsService ] [elasticsearch_node_002] loaded module [ingest-common]
[2018-07-22T17:38:49,179][INFO ][o.e.p.PluginsService ] [elasticsearch_node_002] loaded module [lang-expression]
[2018-07-22T17:38:49,179][INFO ][o.e.p.PluginsService ] [elasticsearch_node_002] loaded module [lang-groovy]
[2018-07-22T17:38:49,179][INFO ][o.e.p.PluginsService ] [elasticsearch_node_002] loaded module [lang-mustache]
[2018-07-22T17:38:49,179][INFO ][o.e.p.PluginsService ] [elasticsearch_node_002] loaded module [lang-painless]
[2018-07-22T17:38:49,179][INFO ][o.e.p.PluginsService ] [elasticsearch_node_002] loaded module [parent-join]
[2018-07-22T17:38:49,179][INFO ][o.e.p.PluginsService ] [elasticsearch_node_002] loaded module [percolator]
[2018-07-22T17:38:49,179][INFO ][o.e.p.PluginsService ] [elasticsearch_node_002] loaded module [reindex]
[2018-07-22T17:38:49,179][INFO ][o.e.p.PluginsService ] [elasticsearch_node_002] loaded module [transport-netty3]
[2018-07-22T17:38:49,179][INFO ][o.e.p.PluginsService ] [elasticsearch_node_002] loaded module [transport-netty4]
[2018-07-22T17:38:49,179][INFO ][o.e.p.PluginsService ] [elasticsearch_node_002] no plugins loaded
[2018-07-22T17:38:51,724][INFO ][o.e.d.DiscoveryModule ] [elasticsearch_node_002] using discovery type [zen]
[2018-07-22T17:38:52,224][INFO ][o.e.n.Node ] [elasticsearch_node_002] initialized
[2018-07-22T17:38:52,240][INFO ][o.e.n.Node ] [elasticsearch_node_002] starting ...
[2018-07-22T17:38:52,929][INFO ][o.e.t.TransportService ] [elasticsearch_node_002] publish_address {127.0.0.1:9301}, bound_addresses {127.0.0.1:9301}, {[::1]:9301}
[2018-07-22T17:38:56,365][INFO ][o.e.c.s.ClusterService ] [elasticsearch_node_002] detected_master {elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300}, added {{elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300},}, reason: zen-disco-receive(from master [master {elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300} committed version [7]])
[2018-07-22T17:38:57,005][INFO ][o.e.h.n.Netty4HttpServerTransport] [elasticsearch_node_002] publish_address {127.0.0.1:9201}, bound_addresses {127.0.0.1:9201}, {[::1]:9201}
[2018-07-22T17:38:57,005][INFO ][o.e.n.Node ] [elasticsearch_node_002] started
[2018-07-22T17:39:43,600][INFO ][o.e.c.s.ClusterService ] [elasticsearch_node_002] added {{elasticsearch_node_003}{upVg0NksTRi8LH8mj9xqpg}{ydGaCWORSlWf09TTjpUBQg}{127.0.0.1}{127.0.0.1:9302},}, reason: zen-disco-receive(from master [master {elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300} committed version [16]])
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
λ elasticsearch
[2018-07-22T17:39:34,786][INFO ][o.e.n.Node ] [elasticsearch_node_003] initializing ...
[2018-07-22T17:39:35,020][INFO ][o.e.e.NodeEnvironment ] [elasticsearch_node_003] using [1] data paths, mounts [[disk-C (C:)]], net usable_space [76.8gb], net total_space [118.7gb], spins? [unknown], types [NTFS]
[2018-07-22T17:39:35,020][INFO ][o.e.e.NodeEnvironment ] [elasticsearch_node_003] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-07-22T17:39:35,020][INFO ][o.e.n.Node ] [elasticsearch_node_003] node name [elasticsearch_node_003], node ID [upVg0NksTRi8LH8mj9xqpg]
[2018-07-22T17:39:35,020][INFO ][o.e.n.Node ] [elasticsearch_node_003] version[5.6.10], pid[2316], build[b727a60/2018-06-06T15:48:34.860Z], OS[Windows 10/10.0/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_131/25.131-b11]
[2018-07-22T17:39:35,020][INFO ][o.e.n.Node ] [elasticsearch_node_003] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Delasticsearch, -Des.path.home=C:\ProfessionalSoftWare\ElasticSearch\elasticsearch-5.6.10-node3]
[2018-07-22T17:39:36,389][INFO ][o.e.p.PluginsService ] [elasticsearch_node_003] loaded module [aggs-matrix-stats]
[2018-07-22T17:39:36,389][INFO ][o.e.p.PluginsService ] [elasticsearch_node_003] loaded module [ingest-common]
[2018-07-22T17:39:36,389][INFO ][o.e.p.PluginsService ] [elasticsearch_node_003] loaded module [lang-expression]
[2018-07-22T17:39:36,389][INFO ][o.e.p.PluginsService ] [elasticsearch_node_003] loaded module [lang-groovy]
[2018-07-22T17:39:36,389][INFO ][o.e.p.PluginsService ] [elasticsearch_node_003] loaded module [lang-mustache]
[2018-07-22T17:39:36,389][INFO ][o.e.p.PluginsService ] [elasticsearch_node_003] loaded module [lang-painless]
[2018-07-22T17:39:36,389][INFO ][o.e.p.PluginsService ] [elasticsearch_node_003] loaded module [parent-join]
[2018-07-22T17:39:36,389][INFO ][o.e.p.PluginsService ] [elasticsearch_node_003] loaded module [percolator]
[2018-07-22T17:39:36,389][INFO ][o.e.p.PluginsService ] [elasticsearch_node_003] loaded module [reindex]
[2018-07-22T17:39:36,389][INFO ][o.e.p.PluginsService ] [elasticsearch_node_003] loaded module [transport-netty3]
[2018-07-22T17:39:36,404][INFO ][o.e.p.PluginsService ] [elasticsearch_node_003] loaded module [transport-netty4]
[2018-07-22T17:39:36,404][INFO ][o.e.p.PluginsService ] [elasticsearch_node_003] no plugins loaded
[2018-07-22T17:39:38,995][INFO ][o.e.d.DiscoveryModule ] [elasticsearch_node_003] using discovery type [zen]
[2018-07-22T17:39:39,605][INFO ][o.e.n.Node ] [elasticsearch_node_003] initialized
[2018-07-22T17:39:39,605][INFO ][o.e.n.Node ] [elasticsearch_node_003] starting ...
[2018-07-22T17:39:40,386][INFO ][o.e.t.TransportService ] [elasticsearch_node_003] publish_address {127.0.0.1:9302}, bound_addresses {127.0.0.1:9302}, {[::1]:9302}
[2018-07-22T17:39:43,631][INFO ][o.e.c.s.ClusterService ] [elasticsearch_node_003] detected_master {elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300}, added {{elasticsearch_node_002}{3ezLh-3jTB-c0pQXFXnr5Q}{Y4i8X_t7QhWDBoP6MoDuvw}{127.0.0.1}{127.0.0.1:9301},{elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300},}, reason: zen-disco-receive(from master [master {elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300} committed version [16]])
[2018-07-22T17:39:44,308][INFO ][o.e.h.n.Netty4HttpServerTransport] [elasticsearch_node_003] publish_address {127.0.0.1:9202}, bound_addresses {127.0.0.1:9202}, {[::1]:9202}
[2018-07-22T17:39:44,308][INFO ][o.e.n.Node ] [elasticsearch_node_003] started
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
WKQ@WKQ-PC MINGW64 /c/ProfessionalSoftWare/ElasticSearch/elasticsearch-5.6.10
$ curl -X GET "localhost:9200/_cluster/health"
{
"cluster_name":"elasticsearch_production",
"status":"green",
"timed_out":false,
"number_of_nodes":3,
"number_of_data_nodes":3,
"active_primary_shards":8,
"active_shards":16,
"relocating_shards":0,
"initializing_shards":0,
"unassigned_shards":0,
"delayed_unassigned_shards":0,
"number_of_pending_tasks":0,
"number_of_in_flight_fetch":0,
"task_max_waiting_in_queue_millis":0,
"active_shards_percent_as_number":100
}

(5) 水平扩容

每个节点的硬件资源(CPU, RAM, I/O)将被更少的分片所共享,每个分片的性能将会得到提升。
分片是一个功能完整的搜索引擎,它拥有使用一个节点上的所有资源的能力。
主分片的数目在索引创建时 就已经确定了下来。实际上,这个数目定义了这个索引能够 存储 的最大数据量。(实际大小取决于你的数据、硬件和使用场景。)
但是,读操作——搜索和返回数据——可以同时被主分片 或 副本分片所处理,所以当你拥有越多的副本分片时,也将拥有越高的吞吐量。

(5.1) 设置副本数

1
2
3
4
5
6
7
8
$ curl -X PUT "localhost:9200/blogs/_settings?pretty" -H 'Content-Type: application/json' -d'
> {
> "number_of_replicas" : 2
> }
> '
{
"acknowledged" : true
}

blogs 索引现在拥有9个分片:3个主分片和6个副本分片。 这意味着我们可以将集群扩容到9个节点,每个节点上一个分片。相比原来3个节点时,集群搜索性能可以提升 3 倍。

当然,如果只是在相同节点数目的集群上增加更多的副本分片并不能提高性能,因为每个分片从节点上获得的资源会变少。 你需要增加更多的硬件资源来提升吞吐量。
但是更多的副本分片数提高了数据冗余量:按照上面的节点配置,我们可以在失去2个节点的情况下不丢失任何数据。

(6) 应对故障

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
[2018-07-22T17:59:38,513][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [elasticsearch_node_001] updating number_of_replicas to [2] for indices [blogs]
[2018-07-22T17:59:39,286][INFO ][o.e.c.r.a.AllocationService] [elasticsearch_node_001] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[blogs][2]] ...]).
[2018-07-22T18:05:07,871][INFO ][o.e.n.Node ] [elasticsearch_node_001] stopping ...
[2018-07-22T18:05:07,943][INFO ][o.e.n.Node ] [elasticsearch_node_001] stopped
[2018-07-22T18:05:08,051][INFO ][o.e.n.Node ] [elasticsearch_node_001] closing ...
[2018-07-22T18:05:08,065][INFO ][o.e.n.Node ] [elasticsearch_node_001] closed
终止批处理操作吗(Y/N)? y


[2018-07-22T18:05:07,887][INFO ][o.e.d.z.ZenDiscovery ] [elasticsearch_node_002] master_left [{elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300}], reason [shut_down]
[2018-07-22T18:05:07,910][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch_node_002] master left (reason = shut_down), current nodes: nodes:
{elasticsearch_node_003}{upVg0NksTRi8LH8mj9xqpg}{ydGaCWORSlWf09TTjpUBQg}{127.0.0.1}{127.0.0.1:9302}
{elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300}, master
{elasticsearch_node_002}{3ezLh-3jTB-c0pQXFXnr5Q}{Y4i8X_t7QhWDBoP6MoDuvw}{127.0.0.1}{127.0.0.1:9301}, local

[2018-07-22T18:05:07,911][INFO ][o.e.d.z.ZenDiscovery ] [elasticsearch_node_002] master_left [{elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300}], reason [transport disconnected]
[2018-07-22T18:05:07,925][WARN ][o.e.t.n.Netty4Transport ] [elasticsearch_node_002] write and flush on the network layer failed (channel: [id: 0xee4a8fab, L:0.0.0.0/0.0.0.0:9301 ! R:/127.0.0.1:53349])
java.nio.channels.ClosedChannelException: null
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
[2018-07-22T18:05:08,929][WARN ][o.e.c.NodeConnectionsService] [elasticsearch_node_002] failed to connect to node {elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [elasticsearch_node_001][127.0.0.1:9300] connect_timeout[30s]
at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:363) ~[?:?]
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:570) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:473) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:342) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:154) [elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:107) [elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:674) [elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.10.jar:5.6.10]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: 127.0.0.1/127.0.0.1:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
... 1 more
Caused by: java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
... 1 more
[2018-07-22T18:05:10,961][INFO ][o.e.c.s.ClusterService ] [elasticsearch_node_002] new_master {elasticsearch_node_002}{3ezLh-3jTB-c0pQXFXnr5Q}{Y4i8X_t7QhWDBoP6MoDuvw}{127.0.0.1}{127.0.0.1:9301}, reason: zen-disco-elected-as-master ([0] nodes joined)[, ]
[2018-07-22T18:05:13,003][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [elasticsearch_node_002] failed to execute on node [gjy4N2RCQ4mLxp1lxdyZ8w]
org.elasticsearch.transport.NodeNotConnectedException: [elasticsearch_node_001][127.0.0.1:9300] Node not connected
at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:516) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:730) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.nodesStats(AbstractClient.java:826) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.updateNodeStats(InternalClusterInfoService.java:256) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.refresh(InternalClusterInfoService.java:292) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.maybeRefresh(InternalClusterInfoService.java:277) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.lambda$onMaster$0(InternalClusterInfoService.java:137) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.10.jar:5.6.10]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
[2018-07-22T18:05:13,015][DEBUG][o.e.a.a.i.s.TransportIndicesStatsAction] [elasticsearch_node_002] failed to execute [indices:monitor/stats] on node [gjy4N2RCQ4mLxp1lxdyZ8w]
org.elasticsearch.transport.NodeNotConnectedException: [elasticsearch_node_001][127.0.0.1:9300] Node not connected
at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:503) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.sendNodeRequest(TransportBroadcastByNodeAction.java:322) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.start(TransportBroadcastByNodeAction.java:311) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:234) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:79) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1256) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.stats(AbstractClient.java:1577) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.updateIndicesStats(InternalClusterInfoService.java:270) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.refresh(InternalClusterInfoService.java:321) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.maybeRefresh(InternalClusterInfoService.java:277) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.lambda$onMaster$0(InternalClusterInfoService.java:137) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.10.jar:5.6.10]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
[2018-07-22T18:05:13,057][INFO ][o.e.c.r.a.AllocationService] [elasticsearch_node_002] Cluster health status changed from [GREEN] to [YELLOW] (reason: []).
[2018-07-22T18:05:13,058][INFO ][o.e.c.s.ClusterService ] [elasticsearch_node_002] removed {{elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300},}, reason: zen-disco-node-failed({elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300}), reason(transport disconnected)
[2018-07-22T18:05:14,096][INFO ][o.e.c.r.DelayedAllocationService] [elasticsearch_node_002] scheduling reroute for delayed shards in [58.8s] (6 delayed shards)



[2018-07-22T18:05:07,882][INFO ][o.e.d.z.ZenDiscovery ] [elasticsearch_node_003] master_left [{elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300}], reason [shut_down]
[2018-07-22T18:05:07,891][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch_node_003] master left (reason = shut_down), current nodes: nodes:
{elasticsearch_node_002}{3ezLh-3jTB-c0pQXFXnr5Q}{Y4i8X_t7QhWDBoP6MoDuvw}{127.0.0.1}{127.0.0.1:9301}
{elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300}, master
{elasticsearch_node_003}{upVg0NksTRi8LH8mj9xqpg}{ydGaCWORSlWf09TTjpUBQg}{127.0.0.1}{127.0.0.1:9302}, local

[2018-07-22T18:05:07,904][INFO ][o.e.d.z.ZenDiscovery ] [elasticsearch_node_003] master_left [{elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300}], reason [transport disconnected]
[2018-07-22T18:05:08,917][WARN ][o.e.c.NodeConnectionsService] [elasticsearch_node_003] failed to connect to node {elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [elasticsearch_node_001][127.0.0.1:9300] connect_timeout[30s]
at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:363) ~[?:?]
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:570) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:473) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:342) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:154) [elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:107) [elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:674) [elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.10.jar:5.6.10]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: 127.0.0.1/127.0.0.1:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
... 1 more
Caused by: java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
... 1 more
[2018-07-22T18:05:11,977][INFO ][o.e.c.s.ClusterService ] [elasticsearch_node_003] detected_master {elasticsearch_node_002}{3ezLh-3jTB-c0pQXFXnr5Q}{Y4i8X_t7QhWDBoP6MoDuvw}{127.0.0.1}{127.0.0.1:9301}, reason: zen-disco-receive(from master [master {elasticsearch_node_002}{3ezLh-3jTB-c0pQXFXnr5Q}{Y4i8X_t7QhWDBoP6MoDuvw}{127.0.0.1}{127.0.0.1:9301} committed version [27]])
[2018-07-22T18:05:13,066][INFO ][o.e.c.s.ClusterService ] [elasticsearch_node_003] removed {{elasticsearch_node_001}{gjy4N2RCQ4mLxp1lxdyZ8w}{81yv2TghSQSrqvbPWszDTg}{127.0.0.1}{127.0.0.1:9300},}, reason: zen-disco-receive(from master [master {elasticsearch_node_002}{3ezLh-3jTB-c0pQXFXnr5Q}{Y4i8X_t7QhWDBoP6MoDuvw}{127.0.0.1}{127.0.0.1:9301} committed version [28]])
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
[2018-07-22T18:11:53,988][INFO ][o.e.n.Node               ] [elasticsearch_node_002] stopping ...
[2018-07-22T18:11:54,051][INFO ][o.e.n.Node ] [elasticsearch_node_002] stopped
[2018-07-22T18:11:54,051][INFO ][o.e.n.Node ] [elasticsearch_node_002] closing ...
[2018-07-22T18:11:54,059][INFO ][o.e.n.Node ] [elasticsearch_node_002] closed
终止批处理操作吗(Y/N)? y


[2018-07-22T18:11:53,998][INFO ][o.e.d.z.ZenDiscovery ] [elasticsearch_node_003] master_left [{elasticsearch_node_002}{3ezLh-3jTB-c0pQXFXnr5Q}{Y4i8X_t7QhWDBoP6MoDuvw}{127.0.0.1}{127.0.0.1:9301}], reason [shut_down]
[2018-07-22T18:11:54,002][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch_node_003] master left (reason = shut_down), current nodes: nodes:
{elasticsearch_node_002}{3ezLh-3jTB-c0pQXFXnr5Q}{Y4i8X_t7QhWDBoP6MoDuvw}{127.0.0.1}{127.0.0.1:9301}, master
{elasticsearch_node_003}{upVg0NksTRi8LH8mj9xqpg}{ydGaCWORSlWf09TTjpUBQg}{127.0.0.1}{127.0.0.1:9302}, local

[2018-07-22T18:11:57,059][INFO ][o.e.c.s.ClusterService ] [elasticsearch_node_003] new_master {elasticsearch_node_003}{upVg0NksTRi8LH8mj9xqpg}{ydGaCWORSlWf09TTjpUBQg}{127.0.0.1}{127.0.0.1:9302}, reason: zen-disco-elected-as-master ([0] nodes joined)[, ]
[2018-07-22T18:11:58,069][WARN ][o.e.c.NodeConnectionsService] [elasticsearch_node_003] failed to connect to node {elasticsearch_node_002}{3ezLh-3jTB-c0pQXFXnr5Q}{Y4i8X_t7QhWDBoP6MoDuvw}{127.0.0.1}{127.0.0.1:9301} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [elasticsearch_node_002][127.0.0.1:9301] connect_timeout[30s]
at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:363) ~[?:?]
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:570) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:473) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:342) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:154) [elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:107) [elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:674) [elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.10.jar:5.6.10]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: 127.0.0.1/127.0.0.1:9301
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
... 1 more
Caused by: java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
... 1 more
[2018-07-22T18:11:58,100][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [elasticsearch_node_003] failed to execute on node [3ezLh-3jTB-c0pQXFXnr5Q]
org.elasticsearch.transport.NodeNotConnectedException: [elasticsearch_node_002][127.0.0.1:9301] Node not connected
at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:516) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:730) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.nodesStats(AbstractClient.java:826) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.updateNodeStats(InternalClusterInfoService.java:256) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.refresh(InternalClusterInfoService.java:292) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.maybeRefresh(InternalClusterInfoService.java:277) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.lambda$onMaster$0(InternalClusterInfoService.java:137) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.10.jar:5.6.10]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
[2018-07-22T18:11:58,155][DEBUG][o.e.a.a.i.s.TransportIndicesStatsAction] [elasticsearch_node_003] failed to execute [indices:monitor/stats] on node [3ezLh-3jTB-c0pQXFXnr5Q]
org.elasticsearch.transport.NodeNotConnectedException: [elasticsearch_node_002][127.0.0.1:9301] Node not connected
at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:503) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.sendNodeRequest(TransportBroadcastByNodeAction.java:322) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.start(TransportBroadcastByNodeAction.java:311) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:234) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:79) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1256) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.stats(AbstractClient.java:1577) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.updateIndicesStats(InternalClusterInfoService.java:270) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.refresh(InternalClusterInfoService.java:321) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.maybeRefresh(InternalClusterInfoService.java:277) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.InternalClusterInfoService.lambda$onMaster$0(InternalClusterInfoService.java:137) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.10.jar:5.6.10]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
[2018-07-22T18:11:58,170][INFO ][o.e.c.s.ClusterService ] [elasticsearch_node_003] removed {{elasticsearch_node_002}{3ezLh-3jTB-c0pQXFXnr5Q}{Y4i8X_t7QhWDBoP6MoDuvw}{127.0.0.1}{127.0.0.1:9301},}, reason: zen-disco-node-failed({elasticsearch_node_002}{3ezLh-3jTB-c0pQXFXnr5Q}{Y4i8X_t7QhWDBoP6MoDuvw}{127.0.0.1}{127.0.0.1:9301}), reason(transport disconnected)
[2018-07-22T18:11:58,220][INFO ][o.e.c.r.DelayedAllocationService] [elasticsearch_node_003] scheduling reroute for delayed shards in [59.8s] (8 delayed shards)
[2018-07-22T18:12:02,864][INFO ][o.e.n.Node ] [elasticsearch_node_003] stopping ...
[2018-07-22T18:12:02,893][INFO ][o.e.n.Node ] [elasticsearch_node_003] stopped
[2018-07-22T18:12:02,894][INFO ][o.e.n.Node ] [elasticsearch_node_003] closing ...
[2018-07-22T18:12:02,902][INFO ][o.e.n.Node ] [elasticsearch_node_003] closed
终止批处理操作吗(Y/N)? y

References

[1] 空集群
[2] 集群健康
[3] 添加索引
[4] 添加故障转移
[5] 水平扩容
[6] 应对故障