負載均衡服務之HAProxy https配置、四層負載均衡以及訪問控制
前文我們聊了下haproxy的訪問控制ACL的配置，回顧請參考//www.cnblogs.com/qiuhom-1874/p/12817773.html；今天我們來聊一聊haproxy的https的配置、基於tcp四層負載均衡以及訪問控制的配置說明；
首先我們來看一下haproxy的https的配置；https是什麼我這裡就不過多闡述了，有關證書的申請相關說明請參考//www.cnblogs.com/qiuhom-1874/p/12237944.html；在haproxy的配置文件中，我們要明確的聲明監聽某個埠，該埠需要ssl協議來訪問，類似nginx里的listen 443 ssl配置；除此之外我們還需要用crt來指定證書；不同於nginx里的配置是haproxy的證書內容包含私鑰資訊；所以在我們申請好證書後，還需要把證書文件內容同私鑰文件做合併；
示例：配置haproxy監聽443埠，並支援https
提示：紅框中的內容表示監聽443埠，並明確指定使用ssl協議訪問；證書文件是/etc/haproxy/ssl/haproxy.pem
創建ssl目錄和證書文件
[root@docker_node1 ~]# mkdir -p /etc/haproxy/ssl
[root@docker_node1 ~]# cd /etc/haproxy/ssl
[root@docker_node1 ssl]# ls
[root@docker_node1 ssl]# (umask 066;openssl genrsa -out haproxy.key 2048)
Generating RSA private key, 2048 bit long modulus
.......+++
...+++
e is 65537 (0x10001)
[root@docker_node1 ssl]# ls
haproxy.key
[root@docker_node1 ssl]# openssl req -new -x509 -key haproxy.key -out haproxy.crt -subj "/CN=www.test.com"
[root@docker_node1 ssl]# ls
haproxy.crt haproxy.key
[root@docker_node1 ssl]# cat haproxy.crt haproxy.key > haproxy.pem
[root@docker_node1 ssl]# ls
haproxy.crt haproxy.key haproxy.pem
[root@docker_node1 ssl]# openssl x509 -in haproxy.pem -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
e6:c5:30:f9:10:e5:da:cf
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=www.test.com
Validity
Not Before: May 2 14:41:50 2020 GMT
Not After : Jun 1 14:41:50 2020 GMT
Subject: CN=www.test.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:e6:bd:db:2d:37:1e:b3:26:21:45:37:c3:bc:c2:
f8:f0:35:31:7c:af:f4:22:be:bc:f2:01:a1:ce:31:
32:91:37:06:14:a8:eb:ec:99:30:ae:8f:66:6b:51:
03:94:bb:d1:ae:7d:15:23:ea:9a:83:74:6d:d0:be:
30:5d:bf:4c:dd:79:c6:8d:51:01:ea:d7:a8:e5:93:
f9:11:d9:75:cc:2c:65:d6:31:db:15:20:7f:5e:9f:
56:15:3e:17:b9:82:a8:25:ba:40:17:1c:ef:f2:fc:
11:cb:72:ce:07:5b:57:5d:c3:f1:f6:42:1f:02:63:
b0:33:de:87:a3:43:c7:a0:1b:03:c4:ab:09:f4:67:
ce:ec:ef:1f:88:7a:c2:aa:ca:41:ed:1d:78:60:b1:
41:6d:b9:46:67:1e:c9:ce:43:d8:d6:36:be:ba:ed:
d9:2b:01:9a:0c:8e:64:59:07:3a:c2:2c:55:2b:9b:
06:e5:5b:c7:ba:e2:9f:ea:be:71:9f:76:da:79:c6:
f0:b8:94:70:c0:4f:7b:57:53:5c:30:71:c5:82:d0:
7a:a8:d0:71:32:2c:f0:38:61:69:9b:9e:ac:da:45:
74:63:25:76:78:91:c3:be:6c:7c:72:1c:54:4a:ac:
10:24:45:9c:89:6a:1f:5b:00:22:3a:b6:fe:3f:b8:
73:7b
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Key Identifier:
B2:EC:53:AE:68:E0:65:1D:E6:DB:C8:16:E3:BA:D3:70:BC:E7:79:81
X509v3 Authority Key Identifier:
keyid:B2:EC:53:AE:68:E0:65:1D:E6:DB:C8:16:E3:BA:D3:70:BC:E7:79:81
X509v3 Basic Constraints:
CA:TRUE
Signature Algorithm: sha256WithRSAEncryption
91:a2:36:9b:b8:d3:fd:89:ea:6c:b4:b9:3c:b6:a1:f3:86:87:
71:13:8b:10:a0:73:e9:96:b7:1b:dd:7e:91:64:41:a4:c3:80:
7c:b6:f2:ce:a7:77:28:74:51:da:a6:52:98:a7:88:48:41:2f:
ad:3d:cb:36:bd:94:f9:27:72:f1:e8:ae:f8:bd:38:2e:d6:ea:
0b:26:2d:8b:49:3e:bc:4e:58:9e:bf:79:99:5b:88:23:da:98:
e4:45:79:9c:d2:c9:05:05:9f:23:8a:0f:38:db:9a:c4:5c:e9:
a0:f5:e5:aa:02:bc:df:a7:5d:11:cd:35:08:cc:41:08:40:b6:
0b:e4:ad:79:df:9a:1d:7b:70:8b:65:e5:dc:85:34:55:5c:f2:
94:dc:07:91:43:d0:bb:b4:fb:31:b9:74:19:7a:69:43:11:70:
41:41:86:93:ad:83:42:62:e2:67:69:38:cd:18:c7:e9:f7:f4:
be:78:22:ea:ee:20:db:27:1f:06:87:4c:51:67:19:0a:64:97:
3a:e6:2c:32:bd:84:91:88:96:d2:01:e5:c3:62:59:11:c8:20:
7d:a1:c8:5a:3d:8a:fc:f5:14:fe:41:15:97:ee:47:ec:e5:19:
49:0b:c1:8a:c9:3f:10:4d:66:bf:d5:01:21:2d:fd:8b:a7:95:
17:08:7e:46
[root@docker_node1 ssl]#
提示：我這裡就直接用-subj來指定證書/CN來生成自簽名證書；haproxy的證書要求是把證書資訊和私鑰資訊放在一起，所以我們需要把證書文件資訊和私鑰新資訊通過重定向的方式合併在一起；
測試：重啟haproxy,用瀏覽器訪問
提示：可以看到我們用https訪問是可以正常的訪問，只是證書瀏覽器不信任，所以會提示我們不安全；
把80埠的請求重向定443
提示：紅框中的配置表示把http請求重定向為https，這個類似nginx里的重寫url；
測試：用瀏覽器訪問//192.168.0.22看看是否會把http重寫為https?
提示：可以看到我們訪問用http去訪問，在響應報文中會用location告訴瀏覽器去訪問//192.168.0.22；這樣一來就實現了全站https；
向後端傳遞用戶請求的協議和埠（frontend或backend）
提示：紅框中的內容就表示向後端請求報文中設置X-Forwarded-port首部的值為dst_port變數的值；添加X-Forwared-Proto首部，其值為https，在請求協議是https的請求時；通常這兩個設置是後端應用server要求使用https訪問時需要把前端https請求通過X-Forwared-Proto傳遞給後端；
測試：在後端server上配置日誌格式，分別將這兩個首部記錄到日誌，然後通過訪問，查看日誌中記錄這兩個首部的值是否是我們訪問的443埠和https協議？
提示：以上定義httpd的日誌格式，分別記錄{X-Forwarded-Port和X-Forwared-Proto首部的值
用瀏覽器訪問測試，看看日誌中記錄的資訊
提示：可以看到日誌中把對應訪問埠和協議都傳給了後端server；
haproxy基於4層代理做tcp負載均衡
示例：基於tcp對mysql做負載均衡
提示：以上配置表示用tcp協議來代理後端server；
後端server環境準備
[root@docker_node1 ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE wordpress latest c3fa1c8546fb 44 hours ago 540MB mysql 5.7 f965319e89de 4 days ago 448MB httpd 2.4.37-alpine dfd436f9a5d8 16 months ago 91.8MB [root@docker_node1 ~]# docker run --name db1 -d --net bridge -e MYSQL_ROOT_PASSWORD=admin mysql:5.7 8cade03e4f28f40e7d7e970355735c8b881892754b18a637fef0d956f3f88877 [root@docker_node1 ~]# docker run --name db2 -d --net bridge -e MYSQL_ROOT_PASSWORD=admin mysql:5.7 1bff841c226abe88441103764655dc86257d5b99079eaa113384fd04b62cf0f8 [root@docker_node1 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1bff841c226a mysql:5.7 "docker-entrypoint.s…" 7 seconds ago Up 6 seconds 3306/tcp, 33060/tcp db2 8cade03e4f28 mysql:5.7 "docker-entrypoint.s…" 12 seconds ago Up 12 seconds 3306/tcp, 33060/tcp db1 3572c621f827 wordpress "docker-entrypoint.s…" 9 hours ago Up 9 hours 80/tcp wordpress f07288d607e5 httpd:2.4.37-alpine "httpd-foreground" 7 days ago Up 13 hours 80/tcp web3 971595b7f409 httpd:2.4.37-alpine "httpd-foreground" 7 days ago Up 13 hours 80/tcp web2 5c74f3be1868 httpd:2.4.37-alpine "httpd-foreground" 7 days ago Up 13 hours 80/tcp web1 [root@docker_node1 ~]# docker exec -it db1 /bin/sh # mysql -padmin mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 2 Server version: 5.7.30 MySQL Community Server (GPL) Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> grant all on *.* to "myuser"@'172.17.0.%' identified by 'admin'; Query OK, 0 rows affected, 1 warning (0.00 sec) mysql> \q Bye # exit [root@docker_node1 ~]# docker exec -it db2 /bin/sh # mysql -padmin mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 2 Server version: 5.7.30 MySQL Community Server (GPL) Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> grant all on *.* to "myuser"@'172.17.0.%' identified by 'admin'; Query OK, 0 rows affected, 1 warning (0.01 sec) mysql> \q Bye # exit [root@docker_node1 ~]#
提示：啟動mysql容器時，我們需要對root設置密碼；新建用戶時，我們指定的主機IP地址應該是docker0橋的地址，雖然是基於4層tcp代理，但它還是要更改源ip；
測試：重啟haproxy，用mysql客戶端訪問haproxy所監聽的3306埠，看看是否能夠正常的連接到後端mysql容器里？
提示：可以看到用別的主機上的mysql客戶端是可以正常訪問haproxy監聽的3306埠；
haproxy基於4層tcp做訪問控制
提示：tcp-request connection 表示根據第4層條件對傳入連接執行操作；以上配置表示拒絕源地址為192.168.0.21的連接
測試：重啟haproxy，用192.168.0.21上的mysql客戶端連接192.168.0.22：3306看看是否能夠正常連接？
提示：可以看到在192.168.0.21上用mysql工具就不能夠連接192.168.0.22上的mysql了；
示例：僅開放源地址為192.168.0.21的主機訪問，拒絕其他主機的訪問
提示：tcp-request connection 允許訪問用accept來標識，拒絕用reject；如果是聯合使用後面範圍大的可以不用寫條件，表示除上面匹配到的ACL條目以外的所有條目；同http-request 類似，不同的是http-request允許是用allow標識，拒絕用deny標識；
測試：分別用源地址為192.168.0.21的主機上的mysql客戶端去連接192.168.0.22上的mysql和源地址非192.168.0.21上的mysql客戶端連接192.168.0.22上的mysql看看是否都可以連接？
提示：可以看到在源地址非192.168.0.21上使用mysql客戶端是不能夠連接到後端mysql容器的；這是因為它沒有被我們定義ACL匹配，所以就會被tcp-request connect reject匹配；從而拒絕連接；
最後一個話題，在前邊的部落格中我們聊的了haproxy的狀態頁的配置；其中我們演示的是通過頁面展示給我們，通過滑鼠點點點去管理後端server；這樣一來在對於我們想要監控haproxy本身以及後端server就是一個難題；聰明的你一定注意到haproxy的全局配置段中有一個stat socket /var/lib/haproxy/stats這樣的配置；我們接下來說說這個配置是幹嘛用的；
提示：以上配置是表示將監控頁面資訊綁定到一個socket文件上；我們可以通過對該socket文件發送特定指令，實現操作haproxy的目的
示例：通過socat向/var/lib/haproxy/stats傳遞help資訊，讓其列印幫助頁面
[root@docker_node1 ~]# echo "help"|socat stdio /var/lib/haproxy/stats Unknown command. Please enter one of the following commands only : clear counters : clear max statistics counters (add 'all' for all counters) clear table : remove an entry from a table help : this message prompt : toggle interactive mode with prompt quit : disconnect show info : report information about the running process show pools : report information about the memory pools usage show stat : report counters for each proxy and server show errors : report last request and response errors for each proxy show sess [id] : report the list of current sessions or dump this session show table [id]: report table usage stats or dump this table's contents get weight : report a server's current weight set weight : change a server's weight set server : change a server's state or weight set table [id] : update or create a table entry's data set timeout : change a timeout setting set maxconn : change a maxconn setting set rate-limit : change a rate limiting value disable : put a server or frontend in maintenance mode enable : re-enable a server or frontend which is in maintenance mode shutdown : kill a session or a frontend (eg:to release listening ports) show acl [id] : report available acls or dump an acl's contents get acl : reports the patterns matching a sample for an ACL add acl : add acl entry del acl : delete acl entry clear acl <id> : clear the content of this acl show map [id] : report available maps or dump a map's contents get map : reports the keys and values matching a sample for a map set map : modify map entry add map : add map entry del map : delete map entry clear map <id> : clear the content of this map set ssl <stmt> : set statement for ssl [root@docker_node1 ~]#
提示：從上面的幫助資訊，我們可以了解到我們可以通過管道把對應指令傳給/var/lib/haproxy/stats文件，從而實現管理後端server的目的；
示例：通過socat命令把web1標記為disable
提示：我們執行把web1標記為disable，系統提示我們沒有許可權；接下來我們來配置許可權即可
提示：以上配置表示該socket文件具有admin許可權，接下來在來把web1標記為disable（修改haproxy配置文件，需要重啟haproxy ）
提示：我們修改了許可權以後，現在執行以上命令就沒有報什麼許可權之類的錯誤了，接下來我們打開狀態頁，看看web1的狀態是什麼
提示：可以看到web1變成了維護的模式；
示例：列出監控頁面指標數據的資訊
[root@docker_node1 ~]# echo "show info " |socat stdio /var/lib/haproxy/stats Name: HAProxy Version: 1.5.18 Release_date: 2016/05/10 Nbproc: 4 Process_num: 3 Pid: 6374 Uptime: 0d 0h05m04s Uptime_sec: 304 Memmax_MB: 0 Ulimit-n: 8039 Maxsock: 8039 Maxconn: 4000 Hard_maxconn: 4000 CurrConns: 0 CumConns: 3 CumReq: 10 MaxSslConns: 0 CurrSslConns: 0 CumSslConns: 0 Maxpipes: 0 PipesUsed: 0 PipesFree: 0 ConnRate: 0 ConnRateLimit: 0 MaxConnRate: 1 SessRate: 0 SessRateLimit: 0 MaxSessRate: 1 SslRate: 0 SslRateLimit: 0 MaxSslRate: 0 SslFrontendKeyRate: 0 SslFrontendMaxKeyRate: 0 SslFrontendSessionReuse_pct: 0 SslBackendKeyRate: 0 SslBackendMaxKeyRate: 0 SslCacheLookups: 0 SslCacheMisses: 0 CompressBpsIn: 0 CompressBpsOut: 0 CompressBpsRateLim: 0 ZlibMemUsage: 0 MaxZlibMemUsage: 0 Tasks: 14 Run_queue: 1 Idle_pct: 100 node: docker_node1 description: [root@docker_node1 ~]#
提示：以上資訊就是監控頁面中的指標數據；
示例：上線web1
在監控頁面看web1的狀態
提示：可以看到web1已經正常上線了；聰明的你一定想到了通過這樣的方式動態的去管理後端server；這樣就可以實現很多功能，比如通過zibbix去監控haproxy里的狀態指標；通過腳本動態的上線和下線server、動態的修改後端server的權重（這個需要根據演算法來，如果調度演算法是動態的就支援，靜態的就不支援）；
