ES索引管理工具 – curator
- 2019 年 11 月 5 日
- 筆記
官方文檔：https://www.elastic.co/guide/en/elasticsearch/client/curator/current/actions.html
社區好文：https://blog.csdn.net/laoyang360/article/details/85882832
curator與es版本的兼容性：https://www.elastic.co/guide/en/elasticsearch/client/curator/current/version-compatibility.html
curator允許對索引和快照執行許多不同的操作，包括：
1. 從別名添加或刪除索引（或兩者！） 2. 更改分片路由分配更改分片路由分配 3. 關閉索引關閉索引 4. 創建索引創建索引 5. 刪除索引刪除索引 6. 刪除快照刪除快照 7. 打開被關閉的索引打開被關閉的索引 8. 對索引執行forcemerge段合併操作對索引執行forcemerge段合併操作 9. reindex索引，包括來自遠程集群的索引reindex索引，包括來自遠程集群的索引 10. 更改索引的每個分片的副本數 更改索引的每個分片的副本數 11. rollover索引rollover索引 12. 生成索引的快照（備份）生成索引的快照（備份） 13. 還原快照還原快照
使用方法
安裝
pip install elasticsearch-curator
命令行下單次執行
curator_cli --host 127.0.0.1 --port 9200 close --filter_list '[{"filtertype":"age","source":"creation_date","direction":"older","unit":"days","unit_count":1},{"filtertype":"pattern","kind":"prefix","value":"logs_"}]'
更常用的是，定時任務執行：
mkdir /root/.curator cd /root/.curator
cat curator.yml 內容如下：
client: hosts: 192.168.2.187 port: 9200 url_prefix: use_ssl: False certificate: client_cert: client_key: ssl_no_validate: False http_auth: timeout: 30 master_only: False logging: loglevel: INFO logfile: /var/log/curator.log logformat: default blacklist: ['elasticsearch', 'urllib3']
cat action.yml 內容如下：
# 下面我這裡演示的是關閉7天前的索引
actions: 1: action: close description: "Close selected indices" options: ignore_empty_list: True disable_action: false skip_flush: false delete_aliases: false filters: - filtertype: pattern kind: prefix value: 'logs_|filebeat-|logstash-services-|metricsbeat-' - filtertype: age source: name direction: older timestring: '%Y-%m-%d' unit: days unit_count: 7
由於支援的action操作非常多，建議直接參考官網配置即可：
https://www.elastic.co/guide/en/elasticsearch/client/curator/current/actions.html https://www.elastic.co/guide/en/elasticsearch/client/curator/current/examples.html
執行：
格式：curator [--config CONFIG.YML] [--dry-run] ACTION_FILE.YML curator --config curator.yml action.yml
添加定時任務：
50 23 * * * cd /root/.curator/ && curator --config curator.yml action.yml tail -F /var/log/curator.log 日誌類似如下：
2019-11-04 23:14:13,089 INFO Action ID: 1, "close" completed. 2019-11-04 23:14:13,089 INFO Job completed. 2019-11-04 23:14:30,934 INFO Preparing Action ID: 1, "close" 2019-11-04 23:14:30,935 INFO Creating client object and testing connection 2019-11-04 23:14:30,941 INFO Instantiating client object 2019-11-04 23:14:30,942 INFO Testing client connectivity 2019-11-04 23:14:30,949 INFO Successfully created Elasticsearch client object with provided settings 2019-11-04 23:14:30,953 INFO Trying Action ID: 1, "close": Close selected indices 2019-11-04 23:14:30,995 INFO Closing 1 selected indices: ['metricsbeat-2019-11-01'] 2019-11-04 23:14:31,164 INFO Action ID: 1, "close" completed. 2019-11-04 23:14:31,165 INFO Job completed. 2019-11-04 23:18:06,032 INFO Instantiating client object 2019-11-04 23:18:06,033 INFO Testing client connectivity 2019-11-04 23:18:06,041 INFO Successfully created Elasticsearch client object with provided settings
一個組合的寫法(也可以將多個action拆分成多個腳本來執行)：
actions: 1: action: create_index description: "Create index as named" options: continue_if_exception: True name: '<logstash-{now/d+1d}>' extra_settings: settings: number_of_shards: 2 number_of_replicas: 0 2: action: close description: "Close selected indices" options: ignore_empty_list: True disable_action: false skip_flush: false delete_aliases: false filters: - filtertype: pattern kind: prefix value: 'logs_|filebeat-|logstash-services-|metricsbeat-' - filtertype: age source: name direction: older timestring: '%Y-%m-%d' unit: days unit_count: 7 3: action: delete_indices description: "Delete selected indices" options: continue_if_exception: False timeout_override: 300 filters: - filtertype: pattern kind: prefix value: 'logs_|filebeat-|logstash-services-|metricsbeat-' - filtertype: age source: name direction: older timestring: '%Y-%m-%d' unit: days unit_count: 14
腳本正常執行時候，日誌如下：
2019-11-05 00:42:06,377 INFO Preparing Action ID: 1, "create_index" 2019-11-05 00:42:06,377 INFO Creating client object and testing connection 2019-11-05 00:42:06,384 INFO Instantiating client object 2019-11-05 00:42:06,385 INFO Testing client connectivity 2019-11-05 00:42:06,392 INFO Successfully created Elasticsearch client object with provided settings 2019-11-05 00:42:06,396 INFO Trying Action ID: 1, "create_index": Create index as named 2019-11-05 00:42:06,396 INFO "<logstash-{now/d+2d}>" is using Elasticsearch date math. 2019-11-05 00:42:06,396 INFO Creating index "<logstash-{now/d+2d}>" with settings: {'settings': {'number_of_shards': 2, 'number_of_replicas': 0}} 2019-11-05 00:42:06,608 INFO Action ID: 1, "create_index" completed. 2019-11-05 00:42:06,608 INFO Preparing Action ID: 2, "close" 2019-11-05 00:42:06,608 INFO Creating client object and testing connection 2019-11-05 00:42:06,609 INFO Instantiating client object 2019-11-05 00:42:06,609 INFO Testing client connectivity 2019-11-05 00:42:06,615 INFO Successfully created Elasticsearch client object with provided settings 2019-11-05 00:42:06,619 INFO Trying Action ID: 2, "close": Close selected indices 2019-11-05 00:42:06,672 INFO Closing 3 selected indices: ['filebeat-2019-10-02', 'filebeat-2018-10-25', 'filebeat-2019-10-25'] 2019-11-05 00:42:06,875 INFO Action ID: 2, "close" completed. 2019-11-05 00:42:06,875 INFO Preparing Action ID: 3, "delete_indices" 2019-11-05 00:42:06,875 INFO Creating client object and testing connection 2019-11-05 00:42:06,876 INFO Instantiating client object 2019-11-05 00:42:06,876 INFO Testing client connectivity 2019-11-05 00:42:06,881 INFO Successfully created Elasticsearch client object with provided settings 2019-11-05 00:42:06,885 INFO Trying Action ID: 3, "delete_indices": Delete selected indices 2019-11-05 00:42:06,927 INFO Deleting 2 selected indices: ['filebeat-2018-10-25', 'filebeat-2019-10-02'] 2019-11-05 00:42:06,927 INFO ---deleting index filebeat-2018-10-25 2019-11-05 00:42:06,927 INFO ---deleting index filebeat-2019-10-02 2019-11-05 00:42:07,007 INFO Action ID: 3, "delete_indices" completed. 2019-11-05 00:42:07,007 INFO Job completed.
切記：
curator適用於基於時間或者template其他方式創建的索引，不適合單一索引存儲N久歷史數據的操作的場景。
