python的Tqdm模組

• 2019 年 10 月 5 日
• 筆記

版權聲明：本文為部落客原創文章，遵循 CC 4.0 BY-SA 版權協議，轉載請附上原文出處鏈接和本聲明。

本文鏈接：https://blog.csdn.net/weixin_36670529/article/details/100975882

Tqdm 是一個快速，可擴展的Python進度條，可以在 Python 長循環中添加一個進度提示資訊，用戶只需要封裝任意的迭代器 tqdm(iterator)。

我的系統是window環境，首先安裝python，接下來就是pip。

pip安裝：

在python根目錄下創建一個get-pip.py的文件，內容：

https://bootstrap.pypa.io/get-pip.py

然後在CMD窗口進入python下面：

輸出：

python -m pip install -U pip

由於Tqdm要求的pip版本是9.0所以需要手動安裝pip9.0 http://pypi.python.org/pypi/pip

下載安裝包9.0

然後解壓進入，CMD窗口輸入：python setup.py install

然後就可以安裝Tqdm了，

pip install tqdm

安裝最新的開發版的話

pip install -e git+https://github.com/tqdm/tqdm.git@master#egg=tqdm

最後看看怎麼用呢？https://pypi.python.org/pypi/tqdm

基本用法：

from tqdm import tqdm    for i in tqdm(range(10000)):       sleep(0.01) 

當然除了tqdm，還有trange,使用方式完全相同

for i in trange(100):        sleep(0.1) 

只要傳入list都可以：

pbar = tqdm(["a", "b", "c", "d"])    for char in pbar:      pbar.set_description("Processing %s" % char)

也可以手動控制更新

with tqdm(total=100) as pbar:      for i in range(10):        pbar.update(10) 

也可以這樣：

pbar = tqdm(total=100)    for i in range(10):      pbar.update(10)    pbar.close() 

在Shell的tqdm用法

統計所有python腳本的行數：

$ time find . -name '*.py' -exec cat {} ; | wc -l    857365        real  0m3.458s    user  0m0.274s    sys   0m3.325s        $ time find . -name '*.py' -exec cat {} ; | tqdm | wc -l    857366it [00:03, 246471.31it/s]    857365        real  0m3.585s    user  0m0.862s    sys   0m3.358s

使用參數：

$ find . -name '*.py' -exec cat {} ; |      tqdm --unit loc --unit_scale --total 857366 >> /dev/null    100%|███████████████████████████████████| 857K/857K [00:04<00:00, 246Kloc/s]

備份一個目錄：

$ 7z a -bd -r backup.7z docs/ | grep Compressing |      tqdm --total $(find docs/ -type f | wc -l) --unit files >> backup.log    100%|███████████████████████████████▉| 8014/8014 [01:37<00:00, 82.29files/s]

通過看示範的程式碼，我們能發現使用的核心是tqdm和trange這兩個函數，從程式碼層面分析tqdm的功能，那首先是init.py

__all__ = ['tqdm', 'tqdm_gui', 'trange', 'tgrange', 'tqdm_pandas',          'tqdm_notebook', 'tnrange', 'main', 'TqdmKeyError', 'TqdmTypeError',          '__version__']

跟蹤到_tqdm.py，能看到tqdm類的聲明，首先是初始化

def __init__(self, iterable=None, desc=None, total=None, leave=True,             file=sys.stderr, ncols=None, mininterval=0.1,             maxinterval=10.0, miniters=None, ascii=None, disable=False,             unit='it', unit_scale=False, dynamic_ncols=False,             smoothing=0.3, bar_format=None, initial=0, position=None,             gui=False, **kwargs):

Parameters        iterable : iterable, optional    Iterable to decorate with a progressbar.    可迭代的進度條。    Leave blank to manually manage the updates.    留空手動管理更新？？    desc : str, optional    Prefix for the progressbar.    進度條的描述    total : int, optional    The number of expected iterations. If unspecified,    len(iterable) is used if possible. As a last resort, only basic    progress statistics are displayed (no ETA, no progressbar).    If gui is True and this parameter needs subsequent updating,    specify an initial arbitrary large positive integer,    e.g. int(9e9).    預期的迭代數目，默認為None，則儘可能的迭代下去，如果gui設置為True，這裡則需要後續的更新，將需要指定為一個初始隨意值較大的正整數，例如int(9e9)    leave : bool, optional    If [default: True], keeps all traces of the progressbar    upon termination of iteration.    保留進度條存在的痕迹，簡單來說就是會把進度條的最終形態保留下來，默認為True    file : io.TextIOWrapper or io.StringIO, optional    Specifies where to output the progress messages    [default: sys.stderr]. Uses file.write(str) and file.flush()    methods.    指定消息的輸出    ncols : int, optional    The width of the entire output message. If specified,    dynamically resizes the progressbar to stay within this bound.    If unspecified, attempts to use environment width. The    fallback is a meter width of 10 and no limit for the counter and    statistics. If 0, will not print any meter (only stats).    整個輸出消息的寬度。如果指定，動態調整的進度停留在這個邊界。如果未指定，嘗試使用環境的寬度。如果為0，將不列印任何東西（只統計）。    mininterval : float, optional    Minimum progress update interval, in seconds [default: 0.1].    最小進度更新間隔，以秒為單位（默認值：0.1）。    maxinterval : float, optional    Maximum progress update interval, in seconds [default: 10.0].    最大進度更新間隔，以秒為單位（默認值：10）。    miniters : int, optional    Minimum progress update interval, in iterations.    If specified, will set mininterval to 0.    最小進度更新周期    ascii : bool, optional    If unspecified or False, use unicode (smooth blocks) to fill    the meter. The fallback is to use ASCII characters 1-9 #.    如果不設置，默認為unicode編碼    disable : bool, optional    Whether to disable the entire progressbar wrapper    [default: False].    是否禁用整個進度條包裝(如果為True，進度條不顯示)    unit : str, optional    String that will be used to define the unit of each iteration    [default: it].    將被用來定義每個單元的字元串？？？    unit_scale : bool, optional    If set, the number of iterations will be reduced/scaled    automatically and a metric prefix following the    International System of Units standard will be added    (kilo, mega, etc.) [default: False].    如果設置，迭代的次數會自動按照十、百、千來添加前綴，默認為false    dynamic_ncols : bool, optional    If set, constantly alters ncols to the environment (allowing    for window resizes) [default: False].    不斷改變ncols環境，允許調整窗口大小    smoothing : float, optional    Exponential moving average smoothing factor for speed estimates    (ignored in GUI mode). Ranges from 0 (average speed) to 1    (current/instantaneous speed) [default: 0.3].        bar_format : str, optional    Specify a custom bar string formatting. May impact performance.    If unspecified, will use 『{l_bar}{bar}{r_bar}', where l_bar is    『{desc}{percentage:3.0f}%|' and r_bar is    『| {n_fmt}/{total_fmt} [{elapsed_str}<{remaining_str}, {rate_fmt}]'    Possible vars: bar, n, n_fmt, total, total_fmt, percentage,    rate, rate_fmt, elapsed, remaining, l_bar, r_bar, desc.    自定義欄字元串格式化…默認會使用{l_bar}{bar}{r_bar}的格式，格式同上        initial : int, optional    The initial counter value. Useful when restarting a progress    bar [default: 0].    初始計數器值，默認為0    position : int, optional    Specify the line offset to print this bar (starting from 0)    Automatic if unspecified.    Useful to manage multiple bars at once (eg, from threads).    指定偏移，這個功能在多個條中有用    gui : bool, optional    WARNING: internal parameter - do not use.    Use tqdm_gui(…) instead. If set, will attempt to use    matplotlib animations for a graphical output [default: False].    內部參數…    Returns    out : decorated iterator.    返回為一個迭代器

其實不用分析更多程式碼，多看看幾個例子：(官網的例子)

7zx.py壓縮進度條

# -*- coding: utf-8 -*-    """Usage:     7zx.py [--help | options] <zipfiles>...    Options:     -h, --help   Print this help and exit     -v, --version Print version and exit     -c, --compressed    Use compressed (instead of uncompressed) file sizes     -s, --silent  Do not print one row per zip file     -y, --yes   Assume yes to all queries (for extraction)     -D=<level>, --debug=<level>             Print various types of debugging information. Choices:                 CRITICAL|FATAL                 ERROR                 WARN(ING)                 [default: INFO]                 DEBUG                 NOTSET     -d, --debug-trace   Print lots of debugging information (-D NOTSET)    """    from __future__ import print_function    from docopt import docopt    import logging as log    import subprocess    import re    from tqdm import tqdm    import pty    import os    import io    __author__ = "Casper da Costa-Luis <[email protected]>"    __licence__ = "MPLv2.0"    __version__ = "0.2.0"    __license__ = __licence__            RE_SCN = re.compile("([0-9]+)s+([0-9]+)s+(.*)$", flags=re.M)            def main():      args = docopt(__doc__, version=__version__)      if args.pop('--debug-trace', False):        args['--debug'] = "NOTSET"      log.basicConfig(level=getattr(log, args['--debug'], log.INFO),              format='%(levelname)s: %(message)s')      log.debug(args)          # Get compressed sizes      zips = {}      for fn in args['<zipfiles>']:        info = subprocess.check_output(["7z", "l", fn]).strip()        finfo = RE_SCN.findall(info)            # builtin test: last line should be total sizes        log.debug(finfo)        totals = map(int, finfo[-1][:2])        # log.debug(totals)        for s in range(2):          assert(sum(map(int, (inf[s] for inf in finfo[:-1]))) == totals[s])        fcomp = dict((n, int(c if args['--compressed'] else u))               for (u, c, n) in finfo[:-1])        # log.debug(fcomp)        # zips : {'zipname' : {'filename' : int(size)}}        zips[fn] = fcomp          # Extract      cmd7zx = ["7z", "x", "-bd"]      if args['--yes']:        cmd7zx += ["-y"]      log.info("Extracting from {:d} file(s)".format(len(zips)))      with tqdm(total=sum(sum(fcomp.values()) for fcomp in zips.values()),           unit="B", unit_scale=True) as tall:        for fn, fcomp in zips.items():          md, sd = pty.openpty()          ex = subprocess.Popen(cmd7zx + [fn],                     bufsize=1,                     stdout=md, # subprocess.PIPE,                     stderr=subprocess.STDOUT)          os.close(sd)          with io.open(md, mode="rU", buffering=1) as m:            with tqdm(total=sum(fcomp.values()), disable=len(zips) < 2,                 leave=False, unit="B", unit_scale=True) as t:              while True:                try:                  l_raw = m.readline()                except IOError:                  break                l = l_raw.strip()                if l.startswith("Extracting"):                  exname = l.lstrip("Extracting").lstrip()                  s = fcomp.get(exname, 0) # 0 is likely folders                  t.update(s)                  tall.update(s)                elif l:                  if not any(l.startswith(i) for i in                        ("7-Zip ",                        "p7zip Version ",                        "Everything is Ok",                        "Folders: ",                        "Files: ",                        "Size: ",                        "Compressed: ")):                    if l.startswith("Processing archive: "):                      if not args['--silent']:                        t.write(t.format_interval(                          t.start_t - tall.start_t) + ' ' +                          l.lstrip("Processing archive: "))                    else:                      t.write(l)          ex.wait()            main.__doc__ = __doc__            if __name__ == "__main__":      main()

tqdm_wget.py

"""An example of wrapping manual tqdm updates for urllib reporthook.    # urllib.urlretrieve documentation    > If present, the hook function will be called once    > on establishment of the network connection and once after each block read    > thereafter. The hook will be passed three arguments; a count of blocks    > transferred so far, a block size in bytes, and the total size of the file.    Usage:      tqdm_wget.py [options]    Options:    -h, --help      Print this help message and exit    -u URL, --url URL : string, optional      The url to fetch.      [default: http://www.doc.ic.ac.uk/~cod11/matryoshka.zip]    -o FILE, --output FILE : string, optional      The local file path in which to save the url [default: /dev/null].    """        import urllib    from tqdm import tqdm    from docopt import docopt            def my_hook(t):      """      Wraps tqdm instance. Don't forget to close() or __exit__()      the tqdm instance once you're done with it (easiest using `with` syntax).      Example      -------      >>> with tqdm(...) as t:      ...   reporthook = my_hook(t)      ...   urllib.urlretrieve(..., reporthook=reporthook)      """      last_b = [0]          def inner(b=1, bsize=1, tsize=None):        """        b : int, optional          Number of blocks just transferred [default: 1].        bsize : int, optional          Size of each block (in tqdm units) [default: 1].        tsize : int, optional          Total size (in tqdm units). If [default: None] remains unchanged.        """        if tsize is not None:          t.total = tsize        t.update((b - last_b[0]) * bsize)        last_b[0] = b      return inner            opts = docopt(__doc__)        eg_link = opts['--url']    eg_file = eg_link.replace('/', ' ').split()[-1]    with tqdm(unit='B', unit_scale=True, leave=True, miniters=1,         desc=eg_file) as t: # all optional kwargs      urllib.urlretrieve(eg_link, filename=opts['--output'],                reporthook=my_hook(t), data=None)

examples.py

"""    # Simple tqdm examples and profiling    # Benchmark    for i in _range(int(1e8)):      pass    # Basic demo    import tqdm    for i in tqdm.trange(int(1e8)):      pass    # Some decorations    import tqdm    for i in tqdm.trange(int(1e8), miniters=int(1e6), ascii=True,               desc="cool", dynamic_ncols=True):      pass    # Nested bars    from tqdm import trange    for i in trange(10):      for j in trange(int(1e7), leave=False, unit_scale=True):        pass    # Experimental GUI demo    import tqdm    for i in tqdm.tgrange(int(1e8)):      pass    # Comparison to https://code.google.com/p/python-progressbar/    try:      from progressbar.progressbar import ProgressBar    except ImportError:      pass    else:      for i in ProgressBar()(_range(int(1e8))):        pass    # Dynamic miniters benchmark    from tqdm import trange    for i in trange(int(1e8), miniters=None, mininterval=0.1, smoothing=0):      pass    # Fixed miniters benchmark    from tqdm import trange    for i in trange(int(1e8), miniters=4500000, mininterval=0.1, smoothing=0):      pass    """        from time import sleep    from timeit import timeit    import re        # Simple demo    from tqdm import trange    for i in trange(16, leave=True):      sleep(0.1)        # Profiling/overhead tests    stmts = filter(None, re.split(r'ns*#.*?n', __doc__))    for s in stmts:      print(s.replace('import tqdmn', ''))      print(timeit(stmt='try:nt_range = xrange'               'nexcept:nt_range = rangen' + s, number=1),         'seconds') 

pandas_progress_apply.py

import pandas as pd    import numpy as np    from tqdm import tqdm        df = pd.DataFrame(np.random.randint(0, 100, (100000, 6)))        # Register `pandas.progress_apply` and `pandas.Series.map_apply` with `tqdm`    # (can use `tqdm_gui`, `tqdm_notebook`, optional kwargs, etc.)    tqdm.pandas(desc="my bar!")        # Now you can use `progress_apply` instead of `apply`    # and `progress_map` instead of `map`    df.progress_apply(lambda x: x**2)    # can also groupby:    # df.groupby(0).progress_apply(lambda x: x**2)            # -- Source code for `tqdm_pandas` (really simple!)    # def tqdm_pandas(t):    #  from pandas.core.frame import DataFrame    #  def inner(df, func, *args, **kwargs):    #    t.total = groups.size // len(groups)    #    def wrapper(*args, **kwargs):    #      t.update(1)    #      return func(*args, **kwargs)    #    result = df.apply(wrapper, *args, **kwargs)    #    t.close()    #    return result    #  DataFrame.progress_apply = inner 

引用tqdm並非強製作為依賴：

include_no_requirements.py

# How to import tqdm without enforcing it as a dependency    try:      from tqdm import tqdm    except ImportError:      def tqdm(*args, **kwargs):        if args:          return args[0]        return kwargs.get('iterable', None)