安裝nginx-kafka插件

  • 2019 年 10 月 5 日
  • 筆記

版權聲明：本文為部落客原創文章，遵循 CC 4.0 BY-SA 版權協議，轉載請附上原文出處鏈接和本聲明。

本文鏈接：https://blog.csdn.net/qq_37933685/article/details/82284930

個人部落格：https://suveng.github.io/blog/​​​​​​​ 安裝nginx-kafka插件

nginx可以直接把數據寫到kafka裡面去。

1.安裝git

	yum install -y git

2.切換到/usr/local/src目錄，然後將kafka的c客戶端源碼clone到本地

	cd /usr/local/src  	git clone https://github.com/edenhill/librdkafka

3.進入到librdkafka，然後進行編譯

	cd librdkafka  	yum install -y gcc gcc-c++ pcre-devel zlib-devel  	./configure  	make && make install

4.安裝nginx整合kafka的插件，進入到/usr/local/src，clone nginx整合kafka的源碼

	cd /usr/local/src  	git clone https://github.com/brg-liuwei/ngx_kafka_module

5.進入到nginx的源碼包目錄下 （編譯nginx，然後將將插件同時編譯）

	cd /usr/local/src/nginx-1.12.2  	./configure --add-module=/usr/local/src/ngx_kafka_module/  	make  	make install

6.修改nginx的配置文件，詳情請查看當前目錄的nginx.conf

#user  nobody;  worker_processes  1;    #error_log  logs/error.log;  #error_log  logs/error.log  notice;  #error_log  logs/error.log  info;    #pid        logs/nginx.pid;      events {      worker_connections  1024;  }      http {      include       mime.types;      default_type  application/octet-stream;        #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '      #                  '$status $body_bytes_sent "$http_referer" '      #                  '"$http_user_agent" "$http_x_forwarded_for"';      #access_log  logs/access.log  main;      sendfile        on;      #tcp_nopush     on;      #keepalive_timeout  0;      keepalive_timeout  65;      #gzip  on;        kafka;      kafka_broker_list node-1.xiaoniu.com:9092 node-2.xiaoniu.com:9092 node-3.xiaoniu.com:9092;        server {          listen       80;          server_name  node-6.xiaoniu.com;          #charset koi8-r;          #access_log  logs/host.access.log  main;        	location = /kafka/track {                  kafka_topic track;          }        	location = /kafka/user {                  kafka_topic user;          }            #error_page  404              /404.html;            # redirect server error pages to the static page /50x.html          #          error_page   500 502 503 504  /50x.html;          location = /50x.html {              root   html;          }        }    }

主要是添加kafka 和 location，在liuwei的git倉庫裡面的用法說明有提到。

7.啟動zk和kafka集群(創建topic)

	/bigdata/zookeeper-3.4.9/bin/zkServer.sh start  	/bigdata/kafka_2.11-0.10.2.1/bin/kafka-server-start.sh -daemon /bigdata/kafka_2.11-0.10.2.1/config/server.properties

8.啟動nginx，報錯，找不到kafka.so.1的文件

	error while loading shared libraries: librdkafka.so.1: cannot open shared object file: No such file or directory

原因是沒有載入庫編譯

9.載入so庫

	echo "/usr/local/lib" >> /etc/ld.so.conf  	ldconfig

10.測試前把nginx開啟，記得要ping通才能測試，而且開啟相應的埠，開始測試：向nginx中寫入數據，然後觀察kafka的消費者能不能消費到數據

	curl localhost/kafka/track -d "message send to kafka topic"

Tags:

VirMach 便宜 VPS