【Python爬蟲】：使用高性能非同步多進程爬蟲獲取豆瓣電影Top250

2021 年 1 月 31 日
筆記
Python爬蟲

在本篇博文當中，將會教會大家如何使用高性能爬蟲，快速爬取並解析頁面當中的資訊。一般情況下，如果我們請求網頁的次數太多，每次都要發出一次請求，進行串列執行的話，那麼請求將會佔用我們大量的時間，這樣得不償失。因此我們可以i使用高性能爬蟲，也就是採用多進程，非同步的方式對數據進行爬取和解析，這樣就可以在更快的時間內得到我們想要的結果。本篇博文給出有關爬取豆瓣電影的例子，以此來教會大家如何使用高性能爬蟲。

一.網頁分析

首先我們來分析豆瓣電影的網頁程式碼，在本次的案例當中。我們需要爬取豆瓣電影top250當中的標題title和星數star。

發現，豆瓣電影當中的所有有關電影的資訊全部都隱藏在< ol class=”grid view”>這個標籤，當中，因此我們在編寫xpath的時候，可以利用對它做一個循環。然後又發現，對於電影的title而言，有兩個地方出現，一個地方是在圖片上，另一個地方是在span標籤下的class = title處，但是在span標籤下具有多個標題，為了以免引起混，因此我們使用圖片當中所暗含的標題title文字，使用xpath進行定位即可。

對於star而言，就更加簡單了。我們發現每次一個star的分數出現，就會有又一個<div class=”star”>的標籤在前面，然後再出現了與span有關的標籤，因此我們編寫xpath表達式為：//ol[@class=”grid_view”]//div[@class=”star”]/span[@class=”rating_num”]/text()

這樣就可以得到一整個頁面的star的數值了。當然這樣我們只能獲取第一頁的我們想要得到的數據，怎麼得到第二頁的數據呢？

二.翻頁處理

翻頁處理對於豆瓣電影這個網站還是比較簡單的。我們分別查看第一，二，三頁的url，就會驚奇的發現它的網址如下：

https://movie.douban.com/top250?start=0&filter=
https://movie.douban.com/top250?start=25&filter=
https://movie.douban.com/top250?start=50&filter=

十分明顯，這個網址後面有問號說明想要獲取頁面內容肯定需要發起get請求，都沒有做有關post請求的加密，這樣看來這也太簡單了吧！

同樣的我們發現裡面的參數start在不斷的變化，而filter卻保持不變。因此我們只需要得到start參數的規律就知道該怎麼編寫爬蟲了。

對於start而言，每跳轉一頁，就會增加25的數值，因為每一個頁面裡面均僅有25部電影。這樣我們就找到了start參數的規律，開始編寫爬蟲。

三.爬蟲程式碼的編寫

在編寫的程式碼時候，我們導入了多進程的庫，使用這個庫進行爬蟲，也就只需要在原本程式碼的基礎之上多添加兩行程式碼即可，如下所示：

pool=Pool(4)
pool.map(get_information,number_ls)

這兩行程式碼當中，第一個參數的4表示了我們使用4個進程的進程池進行數據的抓取。數值越大，爬取的效率就越高，這取決於你CPU的數量，數值不能超過CPU核心數的數量，因為一個一個CPU核心同時只能夠運行單個進程。

第二行程式碼使用了map函數，第一個參數填寫我們進行爬蟲的函數，第二個參數填寫爬蟲函數所需要的參數。把這兩個東西放到map函數里，就可以開始高性能爬蟲了。

Remark：
在進行進程池爬蟲的時候，我們放入的參數number_ls一定是一個列表，同時我們在get_imformation函數里使用得到的參數時，每次系統會調用這個參數列表當中的任意一個數值，而不是對整個列表進行調用。

由於整個原因，因此我們編寫整個的程式碼·如下所示：

import requests
from lxml import etree
from multiprocessing.dummy import Pool
cookie='bid=N3Zqe_FFUKc; douban-fav-remind=1; viewed="27093751"; _vwo_uuid_v2=D401F17C96234AE149C4E04B78C3C8066|6fcc3cefe576bff2b89cdf28c4c5f597; __gads=ID=21cdec44606b00df-2250ba4d7ac4009b:T=1604034713:RT=1604034713:S=ALNI_Mb6iYJKYfbUjLxlisTQX5HCODTGKg; gr_user_id=fb6ac40c-94c3-400e-b170-47e126a9b78a; _gid=GA1.2.1520341169.1612004212; _ga=GA1.2.645228582.1602221486; ll="108288"; UM_distinctid=17752f076e4530-0b6eef25ebabba-f7b1332-1fa400-17752f076e57f0; Hm_lvt_19fc7b106453f97b6a84d64302f21a04=1612004228; Hm_lpvt_19fc7b106453f97b6a84d64302f21a04=1612004253; ap_v=0,6.0; _pk_ref.100001.4cf6=%5B%22%22%2C%22%22%2C1612004299%2C%22https%3A%2F%2Fwww.google.com%2F%22%5D; _pk_ses.100001.4cf6=*; __utma=30149280.645228582.1602221486.1611225800.1612004300.9; __utmb=30149280.0.10.1612004300; __utmc=30149280; __utmz=30149280.1612004300.9.9.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); __utma=223695111.645228582.1602221486.1612004300.1612004300.1; __utmb=223695111.0.10.1612004300; __utmc=223695111; __utmz=223695111.1612004300.1.1.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); _pk_id.100001.4cf6=9a1bb1df4597b334.1612004299.1.1612005471.1612004299.'

headers={
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36',
}

url='//movie.douban.com/top250'


number_ls=[]
for i in range(0,251,25):
    number_ls.append(i)

print(number_ls)


def get_information(number_ls):
    param={
        'start':number_ls,
        'filter' :''
    }
    page_content=requests.get(url=url,headers=headers,params=param).text

    with open('douban.html','w',encoding='utf-8') as fp:
        fp.write(page_content)

    tree=etree.HTML(page_content)
    vedio_title=tree.xpath('//ol[@class="grid_view"]//div[@class="pic"]//a/img/@alt')
    star=tree.xpath('//ol[@class="grid_view"]//div[@class="star"]/span[@class="rating_num"]/text()')

    vedio_title_ls=[]
    star_ls=[]
    for i in vedio_title:
        vedio_title_ls.append(i)
    for i in star:
        star_ls.append(i)

    j=0
    while j<len(star_ls):
        print("the movie is ",vedio_title_ls[j])
        print("the star is ",star_ls[j])
        print()
        j+=1

pool=Pool(4)
pool.map(get_information,number_ls)

四.輸出的結果

輸出的結果十分完美，一共有250份電影，如下圖所示：

[0, 25, 50, 75, 100, 125, 150, 175, 200, 225, 250]
the movie is  搏擊俱樂部
the star is  9.0

the movie is  教父2
the star is  9.2

the movie is  獅子王
the star is  9.0

the movie is  指環王2：雙塔奇兵
the star is  9.1

the movie is  死亡詩社
the star is  9.1

the movie is  鋼琴家
the star is  9.2

the movie is  黑客帝國
the star is  9.0

the movie is  指環王1：魔戒再現
the star is  9.0

the movie is  飲食男女
the star is  9.1

the movie is  竊聽風暴
the star is  9.1

the movie is  美麗心靈
the star is  9.0

the movie is  讓子彈飛
the star is  8.8

the movie is  綠皮書
the star is  8.9

the movie is  兩桿大煙槍
the star is  9.1

the movie is  本傑明·巴頓奇事
the star is  8.9

the movie is  海蒂和爺爺
the star is  9.2

the movie is  飛越瘋人院
the star is  9.1

the movie is  看不見的客人
the star is  8.8

the movie is  西西里的美麗傳說
the star is  8.9

the movie is  拯救大兵瑞恩
the star is  9.0

the movie is  穿條紋睡衣的男孩
the star is  9.1

the movie is  小鞋子
the star is  9.2

the movie is  音樂之聲
the star is  9.0

the movie is  情書
the star is  8.9

the movie is  海豚灣
the star is  9.3

the movie is  美國往事
the star is  9.2

the movie is  致命魔術
the star is  8.9

the movie is  沉默的羔羊
the star is  8.9

the movie is  低俗小說
the star is  8.9

the movie is  禁閉島
the star is  8.8

the movie is  蝴蝶效應
the star is  8.8

the movie is  七宗罪
the star is  8.8

the movie is  心靈捕手
the star is  8.9

the movie is  布達佩斯大飯店
the star is  8.9

the movie is  春光乍泄
the star is  8.9

the movie is  摩登時代
the star is  9.3

the movie is  被嫌棄的松子的一生
the star is  8.9

the movie is  哈利·波特與死亡聖器(下)
the star is  8.9

the movie is  阿凡達
the star is  8.7

the movie is  喜劇之王
the star is  8.8

the movie is  致命ID
the star is  8.8

the movie is  剪刀手愛德華
the star is  8.7

the movie is  勇敢的心
the star is  8.9

the movie is  加勒比海盜
the star is  8.8

the movie is  殺人回憶
the star is  8.9

the movie is  狩獵
the star is  9.1

the movie is  請以你的名字呼喚我
the star is  8.9

the movie is  天使愛美麗
the star is  8.7

the movie is  斷背山
the star is  8.8

the movie is  紅辣椒
the star is  9.0

the movie is  觸不可及
the star is  9.2

the movie is  蝙蝠俠：黑暗騎士
the star is  9.2

the movie is  末代皇帝
the star is  9.3

the movie is  活著
the star is  9.3

the movie is  尋夢環遊記
the star is  9.1

the movie is  亂世佳人
the star is  9.3

the movie is  何以為家
the star is  9.1

the movie is  指環王3：王者無敵
the star is  9.2

the movie is  飛屋環遊記
the star is  9.0

the movie is  摔跤吧！爸爸
the star is  9.0

the movie is  哈利·波特與魔法石
the star is  9.1

the movie is  素媛
the star is  9.3

the movie is  少年派的奇幻漂流
the star is  9.1

the movie is  十二怒漢
the star is  9.4

the movie is  哈爾的移動城堡
the star is  9.1

the movie is  鬼子來了
the star is  9.3

the movie is  天空之城
the star is  9.1

the movie is  大話西遊之月光寶盒
the star is  9.0

the movie is  我不是葯神
the star is  9.0

the movie is  聞香識女人
the star is  9.1

the movie is  羅馬假日
the star is  9.0

the movie is  天堂電影院
the star is  9.2

the movie is  辯護人
the star is  9.2

the movie is  貓鼠遊戲
the star is  9.0

the movie is  大鬧天宮
the star is  9.4

the movie is  肖申克的救贖
the star is  9.7

the movie is  霸王別姬
the star is  9.6

the movie is  阿甘正傳
the star is  9.5

the movie is  這個殺手不太冷
the star is  9.4

the movie is  泰坦尼克號
the star is  9.4

the movie is  美麗人生
the star is  9.5

the movie is  千與千尋
the star is  9.4

the movie is  辛德勒的名單
the star is  9.5

the movie is  盜夢空間
the star is  9.3

the movie is  忠犬八公的故事
the star is  9.4

the movie is  星際穿越
the star is  9.3

the movie is  海上鋼琴師
the star is  9.3

the movie is  楚門的世界
the star is  9.3

the movie is  三傻大鬧寶萊塢
the star is  9.2

the movie is  機器人總動員
the star is  9.3

the movie is  放牛班的春天
the star is  9.3

the movie is  大話西遊之大聖娶親
the star is  9.2

the movie is  瘋狂動物城
the star is  9.2

the movie is  無間道
the star is  9.2

the movie is  熔爐
the star is  9.3

the movie is  教父
the star is  9.3

the movie is  當幸福來敲門
the star is  9.1

the movie is  龍貓
the star is  9.2

the movie is  怦然心動
the star is  9.1

the movie is  控方證人
the star is  9.6

the movie is  7號房的禮物
the star is  8.9

the movie is  幽靈公主
the star is  8.9

the movie is  小森林 夏秋篇
the star is  9.0

the movie is  陽光燦爛的日子
the star is  8.8

the movie is  第六感
the star is  8.9

the movie is  重慶森林
the star is  8.8

the movie is  入殮師
the star is  8.9

the movie is  唐伯虎點秋香
the star is  8.7

the movie is  小森林 冬春篇
the star is  9.0

the movie is  愛在黎明破曉前
the star is  8.8

the movie is  超脫
the star is  8.9

the movie is  消失的愛人
the star is  8.7

the movie is  一一
the star is  9.0

the movie is  菊次郎的夏天
the star is  8.8

the movie is  蝙蝠俠：黑暗騎士崛起
the star is  8.8

the movie is  側耳傾聽
the star is  8.9

the movie is  倩女幽魂
the star is  8.7

the movie is  功夫
the star is  8.6

the movie is  超能陸戰隊
the star is  8.7

the movie is  無人知曉
the star is  9.1

the movie is  人生果實
the star is  9.5

the movie is  螢火之森
the star is  8.9

the movie is  甜蜜蜜
the star is  8.8

the movie is  借東西的小人阿莉埃蒂
the star is  8.8

the movie is  瑪麗和馬克思
the star is  8.9

the movie is  愛在日落黃昏時
the star is  8.8

the movie is  馴龍高手
the star is  8.7

the movie is  完美的世界
the star is  9.1

the movie is  幸福終點站
the star is  8.8

the movie is  告白
the star is  8.7

the movie is  大魚
the star is  8.8

the movie is  陽光姐妹淘
the star is  8.8

the movie is  射鵰英雄傳之東成西就
the star is  8.7

the movie is  哈利·波特與阿茲卡班的囚徒
the star is  8.8

the movie is  恐怖直播
the star is  8.8

the movie is  天書奇譚
the star is  9.2

the movie is  怪獸電力公司
the star is  8.7

the movie is  神偷奶爸
the star is  8.6

the movie is  玩具總動員3
the star is  8.8

the movie is  傲慢與偏見
the star is  8.6

the movie is  時空戀旅人
the star is  8.8

the movie is  哈利·波特與密室
the star is  8.7

the movie is  教父3
the star is  8.9

the movie is  釜山行
the star is  8.6

the movie is  血戰鋼鋸嶺
the star is  8.7

the movie is  哪吒鬧海
the star is  9.1

the movie is  被解救的姜戈
the star is  8.7

the movie is  七武士
the star is  9.3

the movie is  喜宴
the star is  8.9

the movie is  電鋸驚魂
the star is  8.7

the movie is  爆裂鼓手
the star is  8.7

the movie is  貧民窟的百萬富翁
the star is  8.6

the movie is  螢火蟲之墓
the star is  8.7

the movie is  東邪西毒
the star is  8.6

the movie is  海街日記
the star is  8.8

the movie is  黑天鵝
the star is  8.6

the movie is  驚魂記
the star is  9.0

the movie is  無敵破壞王
the star is  8.7

the movie is  你看起來好像很好吃
the star is  8.9

the movie is  冰川時代
the star is  8.6

the movie is  雨人
the star is  8.7

the movie is  小偷家族
the star is  8.7

the movie is  綠里奇蹟
the star is  8.9

the movie is  戀戀筆記型電腦
the star is  8.5

the movie is  愛在午夜降臨前
the star is  8.8

the movie is  瘋狂的石頭
the star is  8.5

the movie is  哈利·波特與火焰杯
the star is  8.6

the movie is  寄生蟲
the star is  8.7

the movie is  恐怖游輪
the star is  8.5

the movie is  奇蹟男孩
the star is  8.6

the movie is  雨中曲
the star is  9.0

the movie is  魔女宅急便
the star is  8.7

the movie is  二十二
the star is  8.7

the movie is  海邊的曼徹斯特
the star is  8.6

the movie is  房間
the star is  8.8

the movie is  風之谷
the star is  8.9

the movie is  一個叫歐維的男人決定去死
the star is  8.9

the movie is  我是山姆
the star is  8.9

the movie is  頭號玩家
the star is  8.7

the movie is  英雄本色
the star is  8.7

the movie is  上帝之城
the star is  9.0

the movie is  諜影重重3
the star is  8.8

the movie is  瘋狂原始人
the star is  8.7

the movie is  未麻的部屋
the star is  9.0

the movie is  歲月神偷
the star is  8.7

the movie is  盧安達飯店
the star is  8.9

the movie is  縱橫四海
the star is  8.8

the movie is  三塊廣告牌
the star is  8.7

the movie is  達拉斯買傢俱樂部
the star is  8.8

the movie is  花樣年華
the star is  8.7

the movie is  心迷宮
the star is  8.7

the movie is  記憶碎片
the star is  8.6

the movie is  模仿遊戲
the star is  8.7

the movie is  黑客帝國3：矩陣革命
the star is  8.8

the movie is  新世界
the star is  8.8

the movie is  頭腦特工隊
the star is  8.7

the movie is  荒蠻故事
the star is  8.8

the movie is  你的名字。
the star is  8.4

the movie is  真愛至上
the star is  8.6

the movie is  忠犬八公物語
the star is  9.2

the movie is  諜影重重2
the star is  8.7

the movie is  阿飛正傳
the star is  8.5

the movie is  地球上的星星
the star is  8.9

the movie is  彗星來的那一夜
the star is  8.5

the movie is  完美陌生人
the star is  8.5

the movie is  戰爭之王
the star is  8.7

the movie is  諜影重重
the star is  8.6

the movie is  香水
the star is  8.5

the movie is  東京教父
the star is  9.0

the movie is  東京物語
the star is  9.2

the movie is  朗讀者
the star is  8.6

the movie is  千鈞一髮
the star is  8.8

the movie is  再次出發之紐約遇見你
the star is  8.6

the movie is  驢得水
the star is  8.3

the movie is  猜火車
the star is  8.5

the movie is  黑客帝國2：重裝上陣
the star is  8.6

the movie is  無間道2
the star is  8.6

the movie is  我愛你
the star is  9.1

the movie is  浪潮
the star is  8.7

the movie is  崖上的波妞
the star is  8.5

the movie is  聚焦
the star is  8.8

the movie is  小蘿莉的猴神大叔
the star is  8.4

the movie is  追隨
the star is  8.9

the movie is  黑鷹墜落
the star is  8.7

the movie is  網路謎蹤
the star is  8.6

the movie is  虎口脫險
the star is  8.9

the movie is  人工智慧
the star is  8.7

the movie is  九品芝麻官
the star is  8.6

the movie is  2001太空漫遊
the star is  8.8

the movie is  可可西里
the star is  8.8

the movie is  羅生門
the star is  8.8

the movie is  色，戒
the star is  8.5

the movie is  終結者2：審判日
the star is  8.7

the movie is  城市之光
the star is  9.3

the movie is  初戀這件小事
the star is  8.4

the movie is  魂斷藍橋
the star is  8.8

the movie is  牯嶺街少年殺人事件
the star is  8.9

the movie is  遺願清單
the star is  8.7

the movie is  大佛普拉斯
the star is  8.7

the movie is  新龍門客棧
the star is  8.6

the movie is  波西米亞狂想曲
the star is  8.7

the movie is  源程式碼
the star is  8.5

the movie is  青蛇
the star is  8.6

the movie is  海洋
the star is  9.1

the movie is  燃情歲月
the star is  8.8

the movie is  無恥混蛋
the star is  8.6

the movie is  瘋狂的麥克斯4：狂暴之路
the star is  8.6

the movie is  血鑽
the star is  8.7

the movie is  穿越時空的少女
the star is  8.6

the movie is  步履不停
the star is  8.8

Tags: Python爬蟲