Elasticsearch資料庫 | Elasticsearch-7.5.0應用搭建實戰
- 2020 年 11 月 6 日
- 筆記
Elasticsearch 是一個可用於分散式以及符合RESTful 風格的搜索和數據分析引擎。—— Elastic Stack 官網
搭建Elasticsearch的「那些事兒”
有一天，在黃金梅麗號的甲板上，韋柏告訴薩博，需要在接下來的項目開發過程中，運用到Elasticsearch資料庫，主要用於黃金梅麗號上的各種設備採集數據實時查詢，或許後期還會運用於分散式日誌系統的搭建運用等，讓薩博先做一次技術預研。於是，在薩博查詢大量的資料發現，關於Elasticsearch資料庫的搭建，網上的資料幾乎是千篇一律，或者多數都是沒有一個完整的流程。甚至，還發現，對於Elasticsearch資料庫實際應用方面，大多數海賊只局限於ELK等這樣的情況，就像是大多數海賊提到Redis，就只覺得這個玩意兒，只能用於快取層面，但是實際上人家的功能強大到超出了海賊的想像空間。甚至於，薩博在一個阿里巴巴的地方，找到了關於Elasticsearch資料庫免費試用的資源，但是對於免費的午餐，也許那一天收費之後，那就只能用一首「浪浪」來祭奠這萬惡的黃金之惡。於是在薩博精心研究和分析發現，除了傳統部署Elasticsearch資料庫之外，有個叫Docker 的東西，除了能夠穩定運行Elasticsearch資料庫之外，還能節省不少的伺服器記憶體。於是呀，薩博就開始了探索在Docker上搭建Elasticsearch之之路……
Docker部署Elasticsearch的「那些事兒”
1.基礎先決條件
阿里雲或者虛擬機VM主機一個，並且自行裝有Docker容器：
ps[⚠️注意事項]：
對於Docker安裝請參考：Linux環境基於CentOS7 搭建部署Docker容器
2.最大文件打開數
[2.1]參數優化：ulimit -a |grep open
[root@centos-meteor ~]# ulimit -a |grep open
open files (-n) 65535
[root@centos-meteor ~]#
[2.2] 修改配置文件： /etc/security/limits.conf
#在最下邊添加如下兩行
* soft nofile 65536
* hard nofile 65536
* soft nproc 2048
* hard nproc 4096
[2.3] 確認是否修改： ulimit -a |grep open
[root@centos-meteor ~]# ulimit -a |grep open
open files (-n) 65535
[root@centos-meteor ~]#
3.調整系統進程數
修改方法：vim /etc/security/limits.d/20-nproc.conf
#調整成以下配置
* soft nproc 4096
root soft nproc unlimited
4.調整虛擬記憶體和最大並發連接數
虛擬記憶體（Virtual memory）->vm.max_map_count
Elasticsearch默認使用 hybrid mmapfs / niofs 目錄來存儲索引。默認作業系統對mmap計數的限制太低，可能引發記憶體不足的異常
[1]臨時解決方法：
sysctl –w vm.max_map_count=262144
sysctl –w vm.max_map_count=262144
如果不優化會出現啟動Elasticsearch的時候會出現如下提示：
max virtual memory areas vm.max_map_count [65535] is too low, increase to at least [262144]
[2]永久生效方案：更新系統文件/etc/sysctl.conf的vm.max_map_count欄位
執行命令：vim /etc/sysctl.conf
[root@turbo-master ~]# vim /etc/sysctl.conf
[root@turbo-master ~]#
#Settings Elasticsearch Virtual Memory—>655360
vm.max_map_count=655360
fs.file-max=655360
vm.overcommit_memory=1
載入到系統：sysctl -p –load /etc/sysctl.conf
[root@centos-meteor elasticsearch]# sysctl -p --load /etc/sysctl.conf
vm.swappiness = 0
net.ipv4.neigh.default.gc_stale_time = 120
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_announce = 2
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_synack_retries = 2
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
kernel.sysrq = 1
vm.max_map_count = 655360
fs.file-max = 655360
vm.overcommit_memory = 1
[root@centos-meteor elasticsearch]#
5.關閉elasticsearch自檢測
在elasticsearch.yml中添加配置項：
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
ps[⚠️注意事項]：
主要是避免以下問題：
[1]最大文件打開數: max file descriptors [1024] for elasticsearch process is too low, increase to at least [65536]
[2]調整虛擬記憶體和最大並發連接數: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[3]關閉elasticsearch自檢測: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[4]調整系統進程數: max number of threads [1024] for user [lish] likely too low, increase to at least [2048]
實戰部署ElasticSearch服務
Docker 部署應用服務的基本步驟：Search[查詢鏡像]->Pull[拉取鏡像]->Run[部署鏡像]
1.查詢Elasticsearch 鏡像：
docker search elasticsearch
ps[注意事項]：
- 一般拉取鏡像資源都是從Docker官方倉庫[docker-hub]拉取，或者自己構建的Docker雲倉庫aliyun-docker 等
- 本教程選取的ELK鏡像均是基於ELK官方Docker倉庫elastic-io
2.拉取Elasticsearch 鏡像：
docker pull docker.elastic.co/elasticsearch/elasticsearch:7.3.1
ps[注意事項]：
1.本教程採用7.3.x版本，目前最新版本7.4.x[主要用7.3.x版本在阿里雲搭建過，避免入坑問題]
2.拉取的過程中可能會出現[net/http: TLS handshake timeout]問題,多嘗試幾次,主要是網路頻寬限制問題
3.修改鏡像名稱:
docker tag docker.elastic.co/elasticsearch/elasticsearch:7.3.1 elasticsearch:latest
ps[注意事項]：
1.名稱過長導致查看些許不便，通過docker tag source-image[來源鏡像] target-image[目標鏡像]，推薦統一採用[target-image:target-version]格式定義，且不佔用空間，相當於重命名鏡像
2.對於拉取kibana[docker.elastic.co/kibana/kibana:7.3.1]和logstash[docker.elastic.co/logstash/logstash:7.3.1] 都建議修改。
4.部署鏡像服務：
部署命令：
docker run -itd -p 9200:9200 -p 9300:9300 --restart=always --privileged=true --name elasticsearch-server -e "discovery.type=single-node" -e ES_JAVA_OPTS="-Xms=512m -Xms=512m" elasticsearch:latest
默認配置目錄：
/usr/share/elasticsearch/config
/usr/share/elasticsearch/logs
查看容器列表：
docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Ports}}"
![docker ps]
(//ftp.bmp.ovh/imgs/2019/11/d7e09058af3fadbd.png)
ps[注意事項]:
1.需要開放埠[9200和9300]->9200作為Http協議，主要用於外部通訊,9300作為Tcp協議，jar之間就是通過tcp協議通訊,通常部署集群就是通過9300通訊。推薦[宿主機自定義埠:9200]
2.–restart=always :配置容器重啟策略,當宿主機重啟由於配置了開機自啟動，不用手動啟動
3.–privileged：配置容器操作許可權[true-root操作許可權，false-當前容器用戶操作許可權]
4.對於部署網路模式推薦默認橋接模式,也自定義可以host模式等
5.修改配置：
進入容器：docker exec -it container-id[容器id] or container-name[容器名稱] /bin/bash
例如：docker exec -it f2d2e97da375 /bin/bash #f2d2e97da375-> container-id
修改配置文件:
[root@f2d2e97da375 elasticsearch]# ls
LICENSE.txt NOTICE.txt README.textile bin config data jdk lib logs modules plugins
[root@f2d2e97da375 elasticsearch]#
[root@f2d2e97da375 elasticsearch]# cd config
[root@f2d2e97da375 config]# ls
elasticsearch.keystore elasticsearch.yml jvm.options log4j2.properties role_mapping.yml roles.yml users users_roles
[root@f2d2e97da375 config]# vi elasticsearch.yml
添加跨域配置：http.cors.enabled: true && http.cors.allow-origin: "*"
cluster.name: "docker-cluster"
network.host: 0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: "*"
然後退出exit容器，在宿主機重啟容器：docker restart container-id[容器id] or container-name[容器名稱]
docker restart f2d2e97da375
[root@f2d2e97da375 config]# exit
exit
[root@centos-meteor ~]# docker restart f2d2e97da375
f2d2e97da375
[root@centos-meteor ~]#
ps[注意事項]：
1.進入容器方式：包括使用 docker attach 命令或 docker exec 命令，
推薦使用 docker exec 命令。原因：
- docker attach: 使用exit退出容器，會導致容器的停止
- docker exec：使用exit退出容器，不會導致容器的停止
- 參考docker進入容器的幾種方法部落格-docker進入容器的幾種方法
2.如果Docker安裝了可視化介面 Portainer，推薦採用這種方式進入容器：
搭建部署ElasticSearch-Head服務
ElasticSearch-Head：彈性搜索集群的Web前端介面，是使用Nodjs構建的，主要用於查看ElasticSearch相關資訊
1.拉取Elasticsearch-Head 鏡像：docker pull mobz/elasticsearch-head:5
[root@centos-amber ~]# docker pull mobz/elasticsearch-head:5
5: Pulling from mobz/elasticsearch-head
75a822cd7888: Pull complete
57de64c72267: Pull complete
4306be1e8943: Pull complete
871436ab7225: Pull complete
0110c26a367a: Pull complete
1f04fe713f1b: Pull complete
723bac39028e: Pull complete
7d8cb47f1c60: Pull complete
7328dcf65c42: Pull complete
b451f2ccfb9a: Pull complete
304d5c28a4cf: Pull complete
4cf804850db1: Pull complete
Digest: sha256:55a3c82dd4ba776e304b09308411edd85de0dc9719f9d97a2f33baa320223f34
Status: Downloaded newer image for mobz/elasticsearch-head:5
docker.io/mobz/elasticsearch-head:5
[root@centos-amber ~]#
2.修改Elasticsearch-Head 鏡像名稱：docker tag mobz/elasticsearch-head:5 elasticsearch-head:latest
[root@centos-amber ~]# docker tag mobz/elasticsearch-head:5 elasticsearch-head:latest
[root@centos-amber ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
grafana/grafana latest 05d1bcf30d16 7 days ago 207MB
nginx latest 540a289bab6c 3 weeks ago 126MB
prom/prometheus latest 2c8e464e47f4 3 weeks ago 129MB
moxm/sentinel-dashboard latest 0ccaac81584e 4 weeks ago 167MB
portainer latest 4cda95efb0e4 4 weeks ago 80.6MB
portainer/portainer latest 4cda95efb0e4 4 weeks ago 80.6MB
apache/skywalking-ui latest fa66ca9c9862 2 months ago 123MB
apache/skywalking-oap-server latest 376a37cdf65c 2 months ago 190MB
docker.elastic.co/kibana/kibana 7.3.1 b54865ba6b0b 2 months ago 1.01GB
docker.elastic.co/elasticsearch/elasticsearch 7.3.1 3d3aa92f641f 2 months ago 807MB
elasticsearch latest 3d3aa92f641f 2 months ago 807MB
prom/node-exporter latest e5a616e4b9cf 5 months ago 22.9MB
google/cadvisor latest eb1210707573 12 months ago 69.6MB
elasticsearch-head latest b19a5c98e43b 2 years ago 824MB
mobz/elasticsearch-head 5 b19a5c98e43b 2 years ago 824MB
tutum/influxdb latest c061e5808198 3 years ago 290MB
[root@centos-amber ~]#
3.部署Elasticsearch-Head 容器：
docker run -itd --restart=always --privileged=true -p 9100:9100 --name elasticsearch-head-server elasticsearch-head:latest
查看容器服務：
docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Ports}}"
4.瀏覽器訪問：//remote-ip:9100/
實戰測試ElasticSearch-Head服務
1.準備數據：
{
"companyId": "ebcb4f99e0cb4ad781278ae636c1031f",
"classifyName": "激素檢測",
"deviceType": "激素檢測",
"data": {
"deviceId": "20000060100000002",
"deviceNo": "QC00020000060100000002",
"appid": "00000000",
"sim": "89860404191792655118",
"csq": "30",
"electric": "98",
"voltage": "13279",
"softVer": "5143000500010601-01020006|5143000500010601-01010003",
"hardVer": "5143000500010601-00000002|5143000500010601-00000002",
"status": "0",
"date": "2020-09-13 11:23:52",
"elements": [
{
"key": "20",
"value": "10",
"time": "2020-09-13 11:23:52"
},
{
"key": "21",
"value": "11",
"time": "2020-09-13 11:23:52"
},
{
"key": "22",
"value": "12",
"time": "2020-09-13 11:23:52"
},
{
"key": "23",
"value": "13",
"time": "2020-09-13 11:23:52"
}
]
}
}
2.設置請求頭：application/json;charset=UTF-8
3.發送PUT請求：//remote-ip/device/deviceData/QC00020000060100000001
{
"_index": "device",
"_type": "deviceData",
"_id": "QC00020000060100000002",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"created": true
}
4.打開瀏覽器查看結果：
實戰搭建ElasticSearch總結
1.部署基本命令：
docker run -itd -p 9200:9200 -p 9300:9300 --restart=always --privileged=true --name elasticsearch-server --network-alias elasticsearch-server --hostname elasticsearch-server -v /docker/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /docker/elasticsearch/data:/usr/share/elasticsearch/data -v /docker/elasticsearch/logs:/usr/share/elasticsearch/logs -e "discovery.type=single-node" -e ES_JAVA_OPTS="-server -Xms512m -Xmx512m -Xmn256m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m -XX:+AlwaysPreTouch -XX:-UseBiasedLocking " -e TZ="Asia/Shanghai" elasticsearch:latest
2.配置文件[elasticsearch.yml ]:
cluster.name: elasticsearch-cluster
node.name: elasticsearch-server
network.host: 0.0.0.0
network.bind_host: 0.0.0.0
network.publish_host: 0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: "*"
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
transport.tcp.port: 9300
transport.tcp.compress: true
http.max_content_length: 128mb
3.分詞插件安裝：
./bin/elasticsearch-plugin install //github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.5.0/elasticsearch-analysis-ik-7.5.0.zip
4.Docker 部署 ElasticSearch-Head：
docker run -itd --restart=always --privileged=true -p 9100:9100 --network-alias elasticsearch-head-server --name elasticsearch-head-server --hostname elasticsearch-head-server -v /docker/elasticsearch-head/app:/usr/src/app elasticsearch-head:latest
5.ElasticSearch-Head 使用406錯誤：
ajaxSettings部分json配置：
ajaxSettings: {
url: ajaxLocation,
isLocal: rlocalProtocol.test(ajaxLocParts[1]),
global: true,
type: "GET",
//contentType: "application/x-www-form-urlencoded",
contentType: "application/json;charset=UTF-8",
processData: true,
async: true,
/*
timeout: 0,
data: null,
dataType: null,
username: null,
password: null,
cache: null,
traditional: false,
headers: {},
*/
accepts: {
xml: "application/xml, text/xml",
html: "text/html",
text: "text/plain",
json: "application/json, text/javascript",
"*": "*/*",
},
inspectData 部分json配置：
// Detect, normalize options and install callbacks for jsonp requests
jQuery.ajaxPrefilter("json jsonp", function (s, originalSettings, jqXHR) {
//var inspectData = s.contentType === "application/x-www-form-urlencoded" &&
var inspectData =
(s.contentType === "application/x-www-form-urlencoded" &&
typeof s.data === "string") ||
(s.contentType === "application/json;charset=UTF-8" &&
typeof s.data === "string");
版權聲明：本文為部落客原創文章，遵循相關版權協議，如若轉載或者分享請附上原文出處鏈接和鏈接來源。
