nginx單機1w並發優化

  • 2019 年 10 月 3 日
  • 筆記

目錄

ab工具
整體優化思路
具體的優化思路
編寫腳本完成並發優化配置
性能統計工具
tips
總結

ab工具

ab -c 10000 -n 200000 http://localhost/index.html

[root@study02 ~]# ab -c 10000 -n 100000 http://192.168.0.217/index.html  This is ApacheBench, Version 2.3 <$Revision: 1430300 $>  Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/  Licensed to The Apache Software Foundation, http://www.apache.org/    Benchmarking 192.168.0.217 (be patient)  socket: Too many open files (24)

整體優化思路

  • 允許建立更多的socket連接
  • 允許打開更多的文件

具體的優化思路

1、socket層面

  • 系統層面
  • 不做洪水抵禦
  • 最大連接數somaxconn
  • 加快tcp連接回收 recyle
  • 空的tcp連接是否允許回收利用reuse
  • nginx
  • 每個子進程允許打開的連接(work_connections)
  • 加快http連接，快速關閉,keepalive_timeout 0

2、文件層面

  • nginx層面
  • 子進程允許打開的文件 worker_rlimit_nofile
  • 系統層面
  • ulimit -n 10000(設置一個比較大的值,允許打開文件數)

具體的配置操作

1、系統配置

查看系統允許打開的最大連接數

more /proc/sys/net/core/somaxconn  echo 50000 > /proc/sys/net/core/somaxconn

打開系統快速連接回收

cat /proc/sys/net/ipv4/tcp_tw_recycle  echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle

打開空的tcp連接允許回收利用

cat /proc/sys/net/ipv4/tcp_tw_reuse  echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse

不做洪水抵禦

cat /proc/sys/net/ipv4/tcp_syncookies  echo 0 > /proc/sys/net/ipv4/tcp_syncookies

2、nginx配置

  • http 1.0 client server 請求-應答-斷開
  • http 1.1 防止頻繁的握手,nginx設置keepalive_timeout參數是一個請求完成之後還要保持連接多久減少tcp的連接次數，在高並發的網站中，keepalived是一個需要嚴重注意的選項，需要將其設置為0不做保持連接提高並發
worker_rlimit_nofile 10000;  events {      worker_connections 10000;  }  keepalive_timeout 0;

編寫腳本完成並發優化配置

echo 50000 > /proc/sys/net/core/somaxconn  echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle  echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse  echo 0 > /proc/sys/net/ipv4/tcp_syncookies

排除問題：

  • 系統的dmesg觀察
  • nginx的error.log來觀察

dmesg|tail

性能統計工具

安裝統計模組http_stub_status_module，便於觀察nginx的狀態

  • 1.進入之前nginx的安裝包目錄，查找是否有統計模組可以安裝

刪選出所有可以安裝的包

    root@STUDY3 nginx-1.14.2]# cat auto/options |grep YES      HTTP=YES      HTTP_CACHE=YES      HTTP_CHARSET=YES      HTTP_GZIP=YES      HTTP_SSI=YES      HTTP_ACCESS=YES      HTTP_AUTH_BASIC=YES      HTTP_MIRROR=YES      HTTP_USERID=YES      HTTP_AUTOINDEX=YES      HTTP_GEO=YES      HTTP_MAP=YES      HTTP_SPLIT_CLIENTS=YES      HTTP_REFERER=YES      HTTP_REWRITE=YES      HTTP_PROXY=YES      HTTP_FASTCGI=YES      HTTP_UWSGI=YES      HTTP_SCGI=YES      HTTP_GRPC=YES      HTTP_MEMCACHED=YES      HTTP_LIMIT_CONN=YES      HTTP_LIMIT_REQ=YES      HTTP_EMPTY_GIF=YES      HTTP_BROWSER=YES      HTTP_UPSTREAM_HASH=YES      HTTP_UPSTREAM_IP_HASH=YES      HTTP_UPSTREAM_LEAST_CONN=YES      HTTP_UPSTREAM_KEEPALIVE=YES      HTTP_UPSTREAM_ZONE=YES      MAIL_POP3=YES      MAIL_IMAP=YES      MAIL_SMTP=YES      STREAM_LIMIT_CONN=YES      STREAM_ACCESS=YES      STREAM_GEO=YES      STREAM_MAP=YES      STREAM_SPLIT_CLIENTS=YES      STREAM_RETURN=YES      STREAM_UPSTREAM_HASH=YES      STREAM_UPSTREAM_LEAST_CONN=YES      STREAM_UPSTREAM_ZONE=YES              --with-select_module)            EVENT_SELECT=YES           ;;              --with-poll_module)              EVENT_POLL=YES             ;;              --with-threads)                  USE_THREADS=YES            ;;              --with-file-aio)                 NGX_FILE_AIO=YES           ;;              --with-http_ssl_module)          HTTP_SSL=YES               ;;              --with-http_v2_module)           HTTP_V2=YES                ;;              --with-http_realip_module)       HTTP_REALIP=YES            ;;              --with-http_addition_module)     HTTP_ADDITION=YES          ;;              --with-http_xslt_module)         HTTP_XSLT=YES              ;;              --with-http_image_filter_module) HTTP_IMAGE_FILTER=YES      ;;              --with-http_geoip_module)        HTTP_GEOIP=YES             ;;              --with-http_sub_module)          HTTP_SUB=YES               ;;              --with-http_dav_module)          HTTP_DAV=YES               ;;              --with-http_flv_module)          HTTP_FLV=YES               ;;              --with-http_mp4_module)          HTTP_MP4=YES               ;;              --with-http_gunzip_module)       HTTP_GUNZIP=YES            ;;              --with-http_gzip_static_module)  HTTP_GZIP_STATIC=YES       ;;              --with-http_auth_request_module) HTTP_AUTH_REQUEST=YES      ;;              --with-http_random_index_module) HTTP_RANDOM_INDEX=YES      ;;              --with-http_secure_link_module)  HTTP_SECURE_LINK=YES       ;;              --with-http_degradation_module)  HTTP_DEGRADATION=YES       ;;              --with-http_slice_module)        HTTP_SLICE=YES             ;;              --with-http_perl_module)         HTTP_PERL=YES              ;;              --with-http_stub_status_module)  HTTP_STUB_STATUS=YES       ;;              --with-mail)                     MAIL=YES                   ;;              --with-mail_ssl_module)          MAIL_SSL=YES               ;;                  MAIL=YES                  MAIL_SSL=YES              --with-stream)                   STREAM=YES                 ;;              --with-stream_ssl_module)        STREAM_SSL=YES             ;;              --with-stream_realip_module)     STREAM_REALIP=YES          ;;              --with-stream_geoip_module)      STREAM_GEOIP=YES           ;;                                               STREAM_SSL_PREREAD=YES     ;;              --with-google_perftools_module)  NGX_GOOGLE_PERFTOOLS=YES   ;;              --with-cpp_test_module)          NGX_CPP_TEST=YES           ;;              --with-compat)                   NGX_COMPAT=YES             ;;              --with-debug)                    NGX_DEBUG=YES              ;;              --with-pcre)                     USE_PCRE=YES               ;;              --with-pcre-jit)                 PCRE_JIT=YES               ;;              --with-libatomic)                NGX_LIBATOMIC=YES          ;;              --test-build-devpoll)            NGX_TEST_BUILD_DEVPOLL=YES ;;              --test-build-eventport)          NGX_TEST_BUILD_EVENTPORT=YES ;;              --test-build-epoll)              NGX_TEST_BUILD_EPOLL=YES   ;;              --test-build-solaris-sendfilev)  NGX_TEST_BUILD_SOLARIS_SENDFILEV=YES ;;

查看是否有http_stub_status_module模組

[root@STUDY3 nginx-1.14.2]# cat auto/options |grep YES|grep 'http_stub_status_module'          --with-http_stub_status_module)  HTTP_STUB_STATUS=YES       ;;    make && make install
  • 2.安裝nginx的性能統計工具
    ./configure --prefix=/usr/local/nginx/ --with-http_stub_status_module
  • 3.查看模組是否安裝成功
    [root@STUDY3 nginx-1.14.2]# /usr/local/nginx/sbin/nginx -V      nginx version: nginx/1.14.2      built by gcc 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC)      configure arguments: --prefix=/usr/local/nginx/ --with-http_stub_status_module

說明模組已經安裝成功了

  • 4.在nginx的server配置項裡面加入如下配置開啟性能統計工具
    location /nginx_status {          stub_status on;          access_log off;          allow 127.0.0.1;      }
  • 5.查看,刷新訪問頁面http://192.168.0.217/nginx_status查看狀態
Active connections: 2020  server accepts handled requests   897553 897553 442986  Reading: 0 Writing: 1 Waiting: 2019
  • 6.ab測試
    [root@study02 ~]# ab -c 10000 -n 200000 http://192.168.0.217/index.html      This is ApacheBench, Version 2.3 <$Revision: 1430300 $>      Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/      Licensed to The Apache Software Foundation, http://www.apache.org/        Benchmarking 192.168.0.217 (be patient)      Completed 20000 requests      Completed 40000 requests      Completed 60000 requests      Completed 80000 requests      Completed 100000 requests      Completed 120000 requests      Completed 140000 requests      Completed 160000 requests      Completed 180000 requests      Completed 200000 requests      Finished 200000 requests          Server Software:        nginx/1.14.2      Server Hostname:        192.168.0.217      Server Port:            80        Document Path:          /index.html      Document Length:        612 bytes        Concurrency Level:      10000      Time taken for tests:   13.268 seconds      Complete requests:      200000      Failed requests:        345710         (Connect: 0, Receive: 0, Length: 174517, Exceptions: 171193)      Write errors:           0      Non-2xx responses:      21      Total transferred:      24276700 bytes      HTML transferred:       17581305 bytes      Requests per second:    15074.19 [#/sec] (mean)      Time per request:       663.386 [ms] (mean)      Time per request:       0.066 [ms] (mean, across all concurrent requests)      Transfer rate:          1786.87 [Kbytes/sec] received        Connection Times (ms)                    min  mean[+/-sd] median   max      Connect:        0  322  85.2    324    1238      Processing:    69  322 108.4    315     692      Waiting:        0   38 103.0      0     503      Total:        373  643  97.8    625    1651        Percentage of the requests served within a certain time (ms)        50%    625        66%    640        75%    643        80%    646        90%    739        95%    883        98%    976        99%   1015       100%   1651 (longest request)

tips

  • 測試機器也需要配置ulimit -n 一個較大的數字
  • 測試機器需要配置echo 50000 > /proc/sys/net/core/somaxconn

總結

在做伺服器部署的時候，首先必須要了解伺服器的配置和伺服器所能夠處理的極限，最先測試的時候可以先從nginx的只跑html靜態頁面測試開始，不連資料庫不做快取，不做邏輯處理，不做cdn來測試nginx的最大能力。大家知道如果加入php，連接了資料庫，做了資料庫快取做了圖片cdn並發會有有影響，那之後的事情也是一樣逐個調試壓榨伺服器的最大性能，有針對性的優化才是正確的。

Tags:

VirMach 便宜 VPS