ES索引管理工具 – curator
- 2019 年 11 月 5 日
- 笔记
官方文档：https://www.elastic.co/guide/en/elasticsearch/client/curator/current/actions.html
社区好文：https://blog.csdn.net/laoyang360/article/details/85882832
curator与es版本的兼容性：https://www.elastic.co/guide/en/elasticsearch/client/curator/current/version-compatibility.html
curator允许对索引和快照执行许多不同的操作，包括：
1. 从别名添加或删除索引（或两者！） 2. 更改分片路由分配更改分片路由分配 3. 关闭索引关闭索引 4. 创建索引创建索引 5. 删除索引删除索引 6. 删除快照删除快照 7. 打开被关闭的索引打开被关闭的索引 8. 对索引执行forcemerge段合并操作对索引执行forcemerge段合并操作 9. reindex索引，包括来自远程集群的索引reindex索引，包括来自远程集群的索引 10. 更改索引的每个分片的副本数 更改索引的每个分片的副本数 11. rollover索引rollover索引 12. 生成索引的快照（备份）生成索引的快照（备份） 13. 还原快照还原快照
使用方法
安装
pip install elasticsearch-curator
命令行下单次执行
curator_cli --host 127.0.0.1 --port 9200 close --filter_list '[{"filtertype":"age","source":"creation_date","direction":"older","unit":"days","unit_count":1},{"filtertype":"pattern","kind":"prefix","value":"logs_"}]'
更常用的是，定时任务执行：
mkdir /root/.curator cd /root/.curator
cat curator.yml 内容如下：
client: hosts: 192.168.2.187 port: 9200 url_prefix: use_ssl: False certificate: client_cert: client_key: ssl_no_validate: False http_auth: timeout: 30 master_only: False logging: loglevel: INFO logfile: /var/log/curator.log logformat: default blacklist: ['elasticsearch', 'urllib3']
cat action.yml 内容如下：
# 下面我这里演示的是关闭7天前的索引
actions: 1: action: close description: "Close selected indices" options: ignore_empty_list: True disable_action: false skip_flush: false delete_aliases: false filters: - filtertype: pattern kind: prefix value: 'logs_|filebeat-|logstash-services-|metricsbeat-' - filtertype: age source: name direction: older timestring: '%Y-%m-%d' unit: days unit_count: 7
由于支持的action操作非常多，建议直接参考官网配置即可：
https://www.elastic.co/guide/en/elasticsearch/client/curator/current/actions.html https://www.elastic.co/guide/en/elasticsearch/client/curator/current/examples.html
执行：
格式：curator [--config CONFIG.YML] [--dry-run] ACTION_FILE.YML curator --config curator.yml action.yml
添加定时任务：
50 23 * * * cd /root/.curator/ && curator --config curator.yml action.yml tail -F /var/log/curator.log 日志类似如下：
2019-11-04 23:14:13,089 INFO Action ID: 1, "close" completed. 2019-11-04 23:14:13,089 INFO Job completed. 2019-11-04 23:14:30,934 INFO Preparing Action ID: 1, "close" 2019-11-04 23:14:30,935 INFO Creating client object and testing connection 2019-11-04 23:14:30,941 INFO Instantiating client object 2019-11-04 23:14:30,942 INFO Testing client connectivity 2019-11-04 23:14:30,949 INFO Successfully created Elasticsearch client object with provided settings 2019-11-04 23:14:30,953 INFO Trying Action ID: 1, "close": Close selected indices 2019-11-04 23:14:30,995 INFO Closing 1 selected indices: ['metricsbeat-2019-11-01'] 2019-11-04 23:14:31,164 INFO Action ID: 1, "close" completed. 2019-11-04 23:14:31,165 INFO Job completed. 2019-11-04 23:18:06,032 INFO Instantiating client object 2019-11-04 23:18:06,033 INFO Testing client connectivity 2019-11-04 23:18:06,041 INFO Successfully created Elasticsearch client object with provided settings
一个组合的写法(也可以将多个action拆分成多个脚本来执行)：
actions: 1: action: create_index description: "Create index as named" options: continue_if_exception: True name: '<logstash-{now/d+1d}>' extra_settings: settings: number_of_shards: 2 number_of_replicas: 0 2: action: close description: "Close selected indices" options: ignore_empty_list: True disable_action: false skip_flush: false delete_aliases: false filters: - filtertype: pattern kind: prefix value: 'logs_|filebeat-|logstash-services-|metricsbeat-' - filtertype: age source: name direction: older timestring: '%Y-%m-%d' unit: days unit_count: 7 3: action: delete_indices description: "Delete selected indices" options: continue_if_exception: False timeout_override: 300 filters: - filtertype: pattern kind: prefix value: 'logs_|filebeat-|logstash-services-|metricsbeat-' - filtertype: age source: name direction: older timestring: '%Y-%m-%d' unit: days unit_count: 14
脚本正常执行时候，日志如下：
2019-11-05 00:42:06,377 INFO Preparing Action ID: 1, "create_index" 2019-11-05 00:42:06,377 INFO Creating client object and testing connection 2019-11-05 00:42:06,384 INFO Instantiating client object 2019-11-05 00:42:06,385 INFO Testing client connectivity 2019-11-05 00:42:06,392 INFO Successfully created Elasticsearch client object with provided settings 2019-11-05 00:42:06,396 INFO Trying Action ID: 1, "create_index": Create index as named 2019-11-05 00:42:06,396 INFO "<logstash-{now/d+2d}>" is using Elasticsearch date math. 2019-11-05 00:42:06,396 INFO Creating index "<logstash-{now/d+2d}>" with settings: {'settings': {'number_of_shards': 2, 'number_of_replicas': 0}} 2019-11-05 00:42:06,608 INFO Action ID: 1, "create_index" completed. 2019-11-05 00:42:06,608 INFO Preparing Action ID: 2, "close" 2019-11-05 00:42:06,608 INFO Creating client object and testing connection 2019-11-05 00:42:06,609 INFO Instantiating client object 2019-11-05 00:42:06,609 INFO Testing client connectivity 2019-11-05 00:42:06,615 INFO Successfully created Elasticsearch client object with provided settings 2019-11-05 00:42:06,619 INFO Trying Action ID: 2, "close": Close selected indices 2019-11-05 00:42:06,672 INFO Closing 3 selected indices: ['filebeat-2019-10-02', 'filebeat-2018-10-25', 'filebeat-2019-10-25'] 2019-11-05 00:42:06,875 INFO Action ID: 2, "close" completed. 2019-11-05 00:42:06,875 INFO Preparing Action ID: 3, "delete_indices" 2019-11-05 00:42:06,875 INFO Creating client object and testing connection 2019-11-05 00:42:06,876 INFO Instantiating client object 2019-11-05 00:42:06,876 INFO Testing client connectivity 2019-11-05 00:42:06,881 INFO Successfully created Elasticsearch client object with provided settings 2019-11-05 00:42:06,885 INFO Trying Action ID: 3, "delete_indices": Delete selected indices 2019-11-05 00:42:06,927 INFO Deleting 2 selected indices: ['filebeat-2018-10-25', 'filebeat-2019-10-02'] 2019-11-05 00:42:06,927 INFO ---deleting index filebeat-2018-10-25 2019-11-05 00:42:06,927 INFO ---deleting index filebeat-2019-10-02 2019-11-05 00:42:07,007 INFO Action ID: 3, "delete_indices" completed. 2019-11-05 00:42:07,007 INFO Job completed.
切记：
curator适用于基于时间或者template其他方式创建的索引，不适合单一索引存储N久历史数据的操作的场景。
