一、教程指南

注意这不一定是最新版本。
中文版：版本比较滞后。
https://www.osgeo.cn/scrapy/index.html#
英文版：
https://docs.scrapy.org/en/latest/

二、项目实践

（一）安装Scrapy

1、安装python，要3.4以上，3.7比较好。
2、设置python的Path
python\
python\Scripts
3、安装Scrapy：

  1. pip install Scrapy

（二）项目架构

Scrapy学习 - 图1
上面是Scrapy的架构图，下面简单介绍一下各个组件

  • Scrapy Engine：引擎用来处理整个系统的数据流，触发各个事件，是整个系统的核心部分。
  • Scheduler：调度器用来接受引擎发过来的Request请求, 压入队列中, 并在引擎再次请求的时候返回。
  • Downloader：下载器用于引擎发过来的Request请求对应的网页内容, 并将获取到的Responses返回给Spider。
  • Spiders：爬虫对Responses进行处理，从中获取所需的字段（即Item）,也可以从Responses获取所需的链接,让Scrapy继续爬取。
  • Item Pipeline:管道负责处理Spider中获取的实体，对数据进行清洗，保存所需的数据。
  • Downloader Middlewares:下载器中间件主要用于处理Scrapy引擎与下载器之间的请求及响应。

  • Spider Middlewares：爬虫中间件主要用于处理Spider的Responses和Requests。

    （三）创建项目

    Scrapy.exe就在python\scripts下，所以可以全局执行scrapy命令。

    1. Scrapy startproject tutorial

    将在当前目录下创建名为tutorial的爬虫项目

    （四）爬虫逻辑

    文件结构

  • 项目名/

[settings] default = projectX.settings

[deploy]

url = http://localhost:6800/

project = projectX

  1. <a name="WJzrQ"></a>
  2. ### settings.py
  3. 配置文件。配置项有很多，在文件头部有链接查看配置的完整说明。
  4. ```python
  5. # -*- coding: utf-8 -*-
  7. # Scrapy settings for projectX project
  8. #
  9. # For simplicity, this file contains only settings considered important or
  10. # commonly used. You can find more settings consulting the documentation:
  11. #
  12. #     https://docs.scrapy.org/en/latest/topics/settings.html
  13. #     https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
  14. #     https://docs.scrapy.org/en/latest/topics/spider-middleware.html
  16. import random
  18. BOT_NAME = 'projectX'
  20. SPIDER_MODULES = ['projectX.spiders']
  21. NEWSPIDER_MODULE = 'projectX.spiders'
  24. # Crawl responsibly by identifying yourself (and your website) on the user-agent
  25. #USER_AGENT = 'projectX (+http://www.yourdomain.com)'
  26. user_agent_list = [
  28.     "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50",
  30.     "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50",
  32.     "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0",
  34.     "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729; InfoPath.3; rv:11.0) like Gecko",
  36.     "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)",
  38.     "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0)",
  40.     "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)",
  42.     "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)",
  44.     "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1",
  46.     "Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1",
  48.     "Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; en) Presto/2.8.131 Version/11.11",
  50.     "Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11",
  52.     "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11"
  54. ]
  56. USER_AGENT = random.choice(user_agent_list)
  58. # Obey robots.txt rules
  59. ROBOTSTXT_OBEY = True
  61. # Configure maximum concurrent requests performed by Scrapy (default: 16)
  62. #CONCURRENT_REQUESTS = 32
  64. # Configure a delay for requests for the same website (default: 0)
  65. # See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
  66. # See also autothrottle settings and docs
  67. #DOWNLOAD_DELAY = 3
  68. # The download delay setting will honor only one of:
  69. #CONCURRENT_REQUESTS_PER_DOMAIN = 16
  70. #CONCURRENT_REQUESTS_PER_IP = 16
  72. # Disable cookies (enabled by default)
  73. #COOKIES_ENABLED = False
  75. # Disable Telnet Console (enabled by default)
  76. #TELNETCONSOLE_ENABLED = False
  78. # Override the default request headers:
  79. # DEFAULT_REQUEST_HEADERS = {
  80. #   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
  81. #   'Accept-Language': 'en',
  82. # }
  84. # Enable or disable spider middlewares
  85. # See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
  86. # SPIDER_MIDDLEWARES = {
  87. #    'projectX.middlewares.ProjectxSpiderMiddleware': 543,
  88. # }
  90. # Enable or disable downloader middlewares
  91. # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
  92. # DOWNLOADER_MIDDLEWARES = {
  93. #    'projectX.middlewares.ProjectxDownloaderMiddleware': 543,
  94. # }
  96. # Enable or disable extensions
  97. # See https://docs.scrapy.org/en/latest/topics/extensions.html
  98. # EXTENSIONS = {
  99. #    'scrapy.extensions.telnet.TelnetConsole': None,
  100. # }
  102. # Configure item pipelines
  103. # See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
  104. ITEM_PIPELINES = {
  105.     'projectX.pipelines.ProjectxPipeline': 300,
  106. }
  109. # Enable and configure the AutoThrottle extension (disabled by default)
  110. # See https://docs.scrapy.org/en/latest/topics/autothrottle.html
  111. #AUTOTHROTTLE_ENABLED = True
  112. # The initial download delay
  113. #AUTOTHROTTLE_START_DELAY = 5
  114. # The maximum download delay to be set in case of high latencies
  115. #AUTOTHROTTLE_MAX_DELAY = 60
  116. # The average number of requests Scrapy should be sending in parallel to
  117. # each remote server
  118. #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
  119. # Enable showing throttling stats for every response received:
  120. #AUTOTHROTTLE_DEBUG = False
  122. # Enable and configure HTTP caching (disabled by default)
  123. # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
  124. #HTTPCACHE_ENABLED = True
  125. #HTTPCACHE_EXPIRATION_SECS = 0
  126. #HTTPCACHE_DIR = 'httpcache'
  127. #HTTPCACHE_IGNORE_HTTP_CODES = []
  128. #HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

pipelines.py

提取非爬虫直接爬取数据时用到，就是复杂数据，比如获得下载链接还需要下载。
记得在settings中注册。

  1. # -*- coding: utf-8 -*-
  3. # Define your item pipelines here
  4. #
  5. # Don't forget to add your pipeline to the ITEM_PIPELINES setting
  6. # See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html
  9. class ProjectxPipeline:
  10.     def process_item(self, item, spider):
  11.         return item

middlewares.py

中间件，看上面架构图，理解其作用。

  1. # -*- coding: utf-8 -*-
  3. # Define here the models for your spider middleware
  4. #
  5. # See documentation in:
  6. # https://docs.scrapy.org/en/latest/topics/spider-middleware.html
  8. from scrapy import signals
  11. class ProjectxSpiderMiddleware:
  12.     # Not all methods need to be defined. If a method is not defined,
  13.     # scrapy acts as if the spider middleware does not modify the
  14.     # passed objects.
  16.     @classmethod
  17.     def from_crawler(cls, crawler):
  18.         # This method is used by Scrapy to create your spiders.
  19.         s = cls()
  20.         crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
  21.         return s
  23.     def process_spider_input(self, response, spider):
  24.         # Called for each response that goes through the spider
  25.         # middleware and into the spider.
  27.         # Should return None or raise an exception.
  28.         return None
  30.     def process_spider_output(self, response, result, spider):
  31.         # Called with the results returned from the Spider, after
  32.         # it has processed the response.
  34.         # Must return an iterable of Request, dict or Item objects.
  35.         for i in result:
  36.             yield i
  38.     def process_spider_exception(self, response, exception, spider):
  39.         # Called when a spider or process_spider_input() method
  40.         # (from other spider middleware) raises an exception.
  42.         # Should return either None or an iterable of Request, dict
  43.         # or Item objects.
  44.         pass
  46.     def process_start_requests(self, start_requests, spider):
  47.         # Called with the start requests of the spider, and works
  48.         # similarly to the process_spider_output() method, except
  49.         # that it doesn’t have a response associated.
  51.         # Must return only requests (not items).
  52.         for r in start_requests:
  53.             yield r
  55.     def spider_opened(self, spider):
  56.         spider.logger.info('Spider opened: %s' % spider.name)
  59. class ProjectxDownloaderMiddleware:
  60.     # Not all methods need to be defined. If a method is not defined,
  61.     # scrapy acts as if the downloader middleware does not modify the
  62.     # passed objects.
  64.     @classmethod
  65.     def from_crawler(cls, crawler):
  66.         # This method is used by Scrapy to create your spiders.
  67.         s = cls()
  68.         crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
  69.         return s
  71.     def process_request(self, request, spider):
  72.         # Called for each request that goes through the downloader
  73.         # middleware.
  75.         # Must either:
  76.         # - return None: continue processing this request
  77.         # - or return a Response object
  78.         # - or return a Request object
  79.         # - or raise IgnoreRequest: process_exception() methods of
  80.         #   installed downloader middleware will be called
  81.         return None
  83.     def process_response(self, request, response, spider):
  84.         # Called with the response returned from the downloader.
  86.         # Must either;
  87.         # - return a Response object
  88.         # - return a Request object
  89.         # - or raise IgnoreRequest
  90.         return response
  92.     def process_exception(self, request, exception, spider):
  93.         # Called when a download handler or a process_request()
  94.         # (from other downloader middleware) raises an exception.
  96.         # Must either:
  97.         # - return None: continue processing this exception
  98.         # - return a Response object: stops process_exception() chain
  99.         # - return a Request object: stops process_exception() chain
  100.         pass
  102.     def spider_opened(self, spider):
  103.         spider.logger.info('Spider opened: %s' % spider.name)

items.py

定义你爬取的数据的结构。

  1. # -*- coding: utf-8 -*-
  3. # Define here the models for your scraped items
  4. #
  5. # See documentation in:
  6. # https://docs.scrapy.org/en/latest/topics/items.html
  8. import scrapy
  10. class ProjectxItem(scrapy.Item):
  11.     # define the fields for your item here like:
  12.     name = scrapy.Field()
  14.     # # 电影标题
  15.     # title = scrapy.Field()
  16.     # # 豆瓣评分
  17.     # star = scrapy.Field()
  18.     # # 主演信息
  19.     # Staring = scrapy.Field()
  20.     # # 豆瓣排名
  21.     # rank = scrapy.Field()
  22.     # # 描述
  23.     # quote = scrapy.Field()
  24.     # # 豆瓣详情页
  25.     # url = scrapy.Field()

spiders/项目名.py

这个文件名是你创建时命名的。不一定是这个名字。
这是爬虫的爬取逻辑，包括如何起始url，提取页面逻辑等。

  1. # -*- coding: utf-8 -*-
  3. import scrapy
  6. class ExampleSpider(scrapy.Spider):  # Scrapy.Spider类型爬虫
  8.     # 标识蜘蛛。它在一个项目中必须是唯一的，也就是说，不能为不同的蜘蛛设置相同的名称。
  9.     name = 'example'
  11.     allowed_domains = ['example.com']
  13.     # 启动请求方法的快捷方式
  14.     # 数组形式
  15.     # start_urls = ['http://example.com/']
  17.     # 函數形式
  18.     # 必须返回一个ITable of requests
  19.     # （您可以返回一个请求列表或编写一个生成器函数），
  20.     # 蜘蛛将从中开始爬行。随后的请求将从这些初始请求中依次生成。
  21.     def start_requests(self):
  23.         urls = [
  24.             'http://quotes.toscrape.com/page/1/',
  25.             'http://quotes.toscrape.com/page/2/',
  26.         ]
  27.         for url in urls:
  28.             yield scrapy.Request(url=url, callback=self.parse)
  30.         # 获取运行时命令行参数
  31.         # url = 'http://quotes.toscrape.com/'
  32.         # tag = getattr(self, 'tag', None)
  33.         # if tag is not None:
  34.         #     url = url + 'tag/' +tag
  35.         # yield scrapy.Request(url, self.parse)
  37.     # 被调用时，每个初始URL完成下载后生成的 Response对象将会作为唯一的参数传递给该函数。
  38.     # 该方法负责解析返回的数据(response data)，
  39.     # 提取数据(生成item)以及生成需要进一步处理的URL的 Request 对象。
  40.     def parse(self, response):
  41.         for quote in response.css('div.quote'):
  42.             # 日志中输出
  43.             yield {
  44.                 'text': quote.css('span.text::text').get(),
  45.                 'author': quote.xpath('span/small/text()').get(),
  46.             }
  48.         # 提取url方式一：发送Request
  49.         # 最麻烦
  50.         for next_page in response.css('li.next a::attr(href)'):
  51.             # 可能是相对路径，所以要加这步
  52.             next_page = response.urljoin(next_page)
  53.             # 回调parse，递归爬取
  54.             yield scrapy.Request(next_page, callback=self.parse)
  56.         # 提取url方式二：response.follow(strHref)
  57.         # for href in response.css('li.next a::attr(href)'):
  58.         #     yield response.follow(href, callback=self.parse)
  60.         # 提取url方式三：response.follow(元素)
  61.         # 最简洁
  62.         # for a in response.css('li.next a'):
  63.         #     yield response.follow(a, callback=self.parse)

（五）运行项目

  1. #运行quotes爬虫
  2. scrapy crawl quotes
  4. #存储爬取数据之json
  5. #当前目录下生成一个quotes.json文件，出于历史原因，scrapy会附加到给定的文件(append)，而不是覆盖其内容。如果在第二次运行此命令两次之前不删除该文件，则最终会得到一个损坏的JSON文件。
  6. scrapy crawl quotes -o quotes.json
  8. #命令行传参：-a tag=fuck
  9. #它将传到爬虫逻辑的__init__方法中，变成爬虫的默认参数。
  10. scrapy crawl quotes -o quotes-humor.json -a tag=humor
  12. #存储爬取数据之JSON Lines
  13. #JSON Lines非常有用，类似于流，因为输出文件可以直接append而不会解析失败，另外每一条记录记录都是独立行方便处理大量数据。
  14. scrapy crawl quotes -o quotes.jl
  16. #上面可以应付直接存储直接爬取到的数据。
  17. #存储更复杂的数据比如图片视频就要用到Item Pipeline，
  18. #在pipelines.py中

（六）具体发生了什么？

Scrapy安排了 scrapy.Request 返回的对象 start_requests 蜘蛛的方法。在接收到每个响应时，它实例化 Response 对象并调用与请求关联的回调方法（在本例中，为 parse 方法）将响应作为参数传递。

（七）递归爬取

不能就爬取start_urls或者start_requests里面的几条链接，很可能是要爬取网站下所有。那就需要在爬取的网页中提取链接。

  1. import scrapy
  3. class QuotesSpider(scrapy.Spider):
  4.     name = "quotes"
  5.     start_urls = [
  6.         'http://quotes.toscrape.com/page/1/',
  7.     ]
  9.     def parse(self, response):
  10.         for quote in response.css('div.quote'):
  11.             yield {
  12.                 'text': quote.css('span.text::text').get(),
  13.                 'author': quote.css('small.author::text').get(),
  14.                 'tags': quote.css('div.tags a.tag::text').getall(),
  15.             }
  16.                 #提取页面中的链接
  17.         for a in response.css('li.next a'):
  18.         yield response.follow(a, callback=self.parse)
  19.         #response.follow，既可以接收string参数也可以接收元素类型。
  21.         #注意，下面这样是错误的，因为css返回的是列表
  22.         response.follow(response.css('li.next a'), callback=self.parse) 
  23.         #但是下面这样，也是可以的。
  24.         response.follow(response.css('li.next a')[0], callback=self.parse)

（八）调试抓取代码

爬虫一运行就停不下来，肯定不能正常调试，Scrapy Shell就可以通过命令行方式执行一次抓取，这样让调试就变得非常简单了。

三、Scrapy Shell教程

可交互的shell，可以输入代码调试抓取代码，无需运行爬虫，即可调试抓取代码，非常有用。

（一）配置Shell

1、方法一

配置环境变量：SCRAPY_PYTHON_SHELL，路径

  1. Python\Lib\site-packages\scrapy
  2. Python\Lib\site-packages\scrapy\commands
  3. #两个下面都有shell.py不知道是哪个

2、方法二

在scrapy.cfg中定义

  1. [settings]
  2. shell = bpython

（二）抓取一次数据

  1. #抓取网页链接
  2. #注意要带上引号，windows上要用双引号，unix系统要用单引号。
  3. scrapy shell 'http://quotes.toscrape.com/page/1/'
  5. #抓取本地文件，如果是本地文件时，注意shell先识别URL再识别路径。
  6. # UNIX-style
  7. scrapy shell ./path/to/file.html
  8. scrapy shell ../other/path/to/file.html
  9. scrapy shell /absolute/path/to/file.html
  11. #File URI
  12. scrapy shell file:///absolute/path/to/file.html
  14. scrapy shell index.html
  15. #不能正常运行，因为index.html被认为是url，html是相当于com

结果如下：

  1. [ ... Scrapy log here ... ]
  2. 2016-09-19 12:09:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/1/> (referer: None)
  3. [s] Available Scrapy objects:
  4. [s]   scrapy     scrapy module (contains scrapy.Request, scrapy.Selector, etc)
  5. [s]   crawler    <scrapy.crawler.Crawler object at 0x7fa91d888c90>
  6. [s]   item       {}
  7. [s]   request    <GET http://quotes.toscrape.com/page/1/>
  8. [s]   response   <200 http://quotes.toscrape.com/page/1/>
  9. [s]   settings   <scrapy.settings.Settings object at 0x7fa91d888c10>
  10. [s]   spider     <DefaultSpider 'default' at 0x7fa91c8af990>
  11. [s] Useful shortcuts:
  12. [s]   shelp()           Shell help (print this help)
  13. [s]   fetch(req_or_url) Fetch request (or URL) and update local objects
  14. [s]   view(response)    View response in a browser
  15. >>>

（三）使用css抽取数据

  1. #返回的是[]形式，是列表，可能有多个
  2. >>> response.css('title')
  3. [<Selector xpath='descendant-or-self::title' data='<title>Quotes to Scrape</title>'>]
  6. #提取title的文本，注意返回的是列表
  7. >>> response.css('title::text').getall()
  8. ['Quotes to Scrape']
  10. #提取的是title整个元素，注意是列表
  11. >>> response.css('title').getall()
  12. ['<title>Quotes to Scrape</title>']
  14. #提取列表第一个title的文本
  15. >>> response.css('title::text').get()
  16. 'Quotes to Scrape'
  18. #同上，但是如果一个没有[0]就会报错，不建议
  19. >>> response.css('title::text')[0].get()
  20. 'Quotes to Scrape'
  22. >>> response.css('title::text').get()
  23. 'Quotes to Scrape'
  25. #正则表达是提取
  26. >>> response.css('title::text').re(r'Quotes.*')
  27. ['Quotes to Scrape']
  29. #正则表达是提取
  30. >>> response.css('title::text').re(r'Q\w+')
  31. ['Quotes']
  33. #正则表达是提取
  34. >>> response.css('title::text').re(r'(\w+) to (\w+)')
  35. ['Quotes', 'Scrape']

（四）Selector Gadget 获取CSS

很好的工具，可以快速找到视觉上选中的元素的CSS选择器，它可以在许多浏览器中使用。

（五）除了CSS，也可以XPath

css底层其实就是转换成xpath，但是css明显要更流行。

  1. >>> response.xpath('//title')
  2. [<Selector xpath='//title' data='<title>Quotes to Scrape</title>'>]
  4. >>> response.xpath('//title/text()').get()
  5. 'Quotes to Scrape'

四、Shell实践

Http://quotes.toscrape.com的HTML结构如下：

  1. <div class="quote">
  2.     <span class="text">“The world as we have created it is a process of our
  3.     thinking. It cannot be changed without changing our thinking.”</span>
  4.     <span>
  5.         by <small class="author">Albert Einstein</small>
  6.         <a href="/author/Albert-Einstein">(about)</a>
  7.     </span>
  8.     <div class="tags">
  9.         Tags:
  10.         <a class="tag" href="/tag/change/page/1/">change</a>
  11.         <a class="tag" href="/tag/deep-thoughts/page/1/">deep-thoughts</a>
  12.         <a class="tag" href="/tag/thinking/page/1/">thinking</a>
  13.         <a class="tag" href="/tag/world/page/1/">world</a>
  14.     </div>
  15. </div>
  1. scrapy shell 'http://quotes.toscrape.com'
  3. >>> response.css("div.quote")
  5. >>> quote = response.css("div.quote")[0]
  7. >>> text = quote.css("span.text::text").get()
  8. >>> text
  9. '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'
  11. >>> author = quote.css("small.author::text").get()
  12. >>> author
  13. 'Albert Einstein'
  15. >>> tags = quote.css("div.tags a.tag::text").getall()
  16. >>> tags
  17. ['change', 'deep-thoughts', 'thinking', 'world']
  19. >>> for quote in response.css("div.quote"):
  20. ...     text = quote.css("span.text::text").get()
  21. ...     author = quote.css("small.author::text").get()
  22. ...     tags = quote.css("div.tags a.tag::text").getall()
  23. ...     print(dict(text=text, author=author, tags=tags))
  24. {'tags': ['change', 'deep-thoughts', 'thinking', 'world'], 'author': 'Albert Einstein', 'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'}
  25. {'tags': ['abilities', 'choices'], 'author': 'J.K. Rowling', 'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”'}
  26.     ... a few more of these, omitted for brevity
  27. >>>