Scrapy框架下第一个爬虫
- 2019 年 10 月 6 日
- 筆記
1. 安装scrapy
pip install scrapy
2. 第一个爬虫代码myspider.py
import scrapy class BlogSpider(scrapy.Spider): name = 'blogspider' start_urls = ['https://blog.scrapinghub.com'] def parse(self, response): for title in response.css('h2.entry-title'): yield {'title': title.css('a ::text').extract_first()} for next_page in response.css('div.prev-post > a'): yield response.follow(next_page, self.parse)
3. 运行
scrapy runspider myspider.py -o result.json
4. 运行结果
017-08-06 17:44:56 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: scrapybot) 2017-08-06 17:44:56 [scrapy.utils.log] INFO: Overridden settings: {'SPIDER_LOADER_WARN_ONLY': True} 2017-08-06 17:44:56 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.memusage.MemoryUsage', ... 2017-08-06 17:44:56 [scrapy.core.engine] INFO: Spider opened 2017-08-06 17:44:56 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2017-08-06 17:44:56 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023 2017-08-06 17:45:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://blog.scrapinghub.com> (referer: None) 2017-08-06 17:45:01 [scrapy.core.scraper] DEBUG: Scraped from <200 https://blog.scrapinghub.com> {'title': u'Scraping the Steam Game Store with Scrapy'} 2017-08-06 17:45:01 [scrapy.core.scraper] DEBUG: Scraped from <200 https://blog.scrapinghub.com> {'title': u'Do Androids Dream of Electric Sheep?'} ... 2017-08-06 17:45:12 [scrapy.core.scraper] DEBUG: Scraped from <200 https://blog.scrapinghub.com/page/11/> {'title': u'Hello, world'} 2017-08-06 17:45:12 [scrapy.core.engine] INFO: Closing spider (finished) 2017-08-06 17:45:12 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 2933, 'downloader/request_count': 11, 'downloader/request_method_count/GET': 11, 'downloader/response_bytes': 123915, 'downloader/response_count': 11, 'downloader/response_status_count/200': 11, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2017, 8, 6, 9, 45, 12, 168728), 'item_scraped_count': 105, 'log_count/DEBUG': 117, 'log_count/INFO': 7, 'memusage/max': 43909120, 'memusage/startup': 43909120, 'request_depth_max': 10, 'response_received_count': 11, 'scheduler/dequeued': 11, 'scheduler/dequeued/memory': 11, 'scheduler/enqueued': 11, 'scheduler/enqueued/memory': 11, 'start_time': datetime.datetime(2017, 8, 6, 9, 44, 56, 752503)} 2017-08-06 17:45:12 [scrapy.core.engine] INFO: Spider closed (finished)
生成结果的文件result.json
[ {"title": "Scraping the Steam Game Store with Scrapy"}, {"title": "Do Androids Dream of Electric Sheep?"}, {"title": "Deploy your Scrapy Spiders from GitHub"}, {"title": "Looking Back at 2016"}, {"title": "How to Increase Sales with Online Reputation Management"}, {"title": "How to Build your own Price Monitoring Tool"}, ... {"title": "Spoofing your Scrapy bot IP using tsocks"}, {"title": "Hello, world"} ]
解析运行过程
当你运行下面命令时，Scrapy框架会启动爬虫引擎，根据myspider.py中的逻辑进行抓取网页，然后把结果存到result.json中。
scrapy runspider myspider.py -o result.json
第一步：爬虫先请求start_urls中定义到URLs。本例中，只有一个URL。请求该URL返回内容，如下所示。
... <h2 class="entry-title"> <a href="https://blog.scrapinghub.com/2017/07/07/scraping-the-steam-game-store-with-scrapy/" rel="bookmark">Scraping the Steam Game Store with Scrapy</a> </h2> ... <h2 class="entry-title"> <a href="https://blog.scrapinghub.com/2016/10/06/interview-how-up-hail-uses-scrapy-to-increase-transparency/" rel="bookmark">Interview: How Up Hail uses Scrapy to Increase Transparency</a> </h2> ... <div class="col-md-6 prev-post"> <a href="https://blog.scrapinghub.com/page/2/">OLDER POST<i class="fa fa-angle-double-right"></i></a> </div> ...
第二步：调用callback方法parse，解析返回到内容。在回调函数parse中，爬虫循环使用CSS选择器 h2.entry-title，找出网页中的<h2 class="entry-title">元素，并这些元素中的text找出来，生成Python的dict格式数据。
当前页中所有的h2.entry-title元素找完以后，爬虫会执行下面的代码
for next_page in response.css('div.prev-post > a'): yield response.follow(next_page, self.parse)
它会找div.prev-post元素，然后再次递归执行parse方法。从网页上可以看到，div.prev-post元素是翻页链接，所以，爬虫会不断翻页，知道抓取所有的网页。
第三步： 生成的字典数据存到result.json文件中
scrapy的任务是异步执行的，也就是说，它不用等一个请求返回以后才发送另一个请求，而是可以同时进行的。这可以加快运行速度。我们也可以对Scrapy进行设置，比如每一个请求延迟一段时间，等等。
