下载中间件

中间件信息

  1. "scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware": 100, 
  2. "scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware": 300, 
  3. "scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware": 350, 
  4. "scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware": 400, 
  5. "scrapy.downloadermiddlewares.useragent.UserAgentMiddleware": 500, 
  6. "scrapy.downloadermiddlewares.retry.RetryMiddleware": 550, 
  7. "scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware": 560, 
  8. "scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware": 580, 
  9. "scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware": 590, 
  10. "scrapy.downloadermiddlewares.redirect.RedirectMiddleware": 600, 
  11. "scrapy.downloadermiddlewares.cookies.CookiesMiddleware": 700, 
  12. "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 750, 
  13. "scrapy.downloadermiddlewares.stats.DownloaderStats": 850, 
  14. "scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware": 900

1 进入项目 命令查看下载中间件信息

image.png

2 在settings文件中找到下载中间件信息并且存放信息

image.png

信息解读

1 数字越小越靠近引擎，越早执行该信息

image.png

2 middlewares中下载中间件的定义

image.png

3 对中间件的功能—-简单解读

  1. """
  2. "scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware": 100,  # 机器人协议中间件
  3. "scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware": 300,  #http身份验证中间件
  4. "scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware": 350,  # 下载超时中间件
  5. "scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware": 400, # 默认请求头 
  6. "scrapy.downloadermiddlewares.useragent.UserAgentMiddleware": 500,  # 用户代理中间件
  7. "scrapy.downloadermiddlewares.retry.RetryMiddleware": 550,  # 重新尝试中间件
  8. "scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware": 560,  # ajax抓取中间件
  9. "scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware": 580,  # 始终使用字符串作为原因
  10. "scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware": 590,  # 允许网站数据压缩发送 
  11. "scrapy.downloadermiddlewares.redirect.RedirectMiddleware": 600,  # 重定向
  12. "scrapy.downloadermiddlewares.cookies.CookiesMiddleware": 700,  # cookie 中间件
  13. "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 750, # 代理中间件
  14. "scrapy.downloadermiddlewares.stats.DownloaderStats": 850,  # 通过此中间件存储通过他的所有请求 响应
  15. "scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware": 900  # 缓存中间件
  16. """

4 常用内置中间件

  1. CookieMiddleware           支持cookie，通过设置COOKIES_ENABLED 来开启和关闭
  3. HttpProxyMiddleware    HTTP代理，通过设置request.meta['proxy']的值来设置
  5. UserAgentMiddleware    与用户代理中间件。

5 四个重要的下载中间件解读
第一个 process_request

  1.     def process_request(self, request, spider):
  2.         # Called for each request that goes through the downloader
  3.         # 功能： 处理请求   参数： request & spider对象   返回值
  4.         # middleware.
  6.         # Must either:  以下必选其一
  7.         # - return None: continue processing this request  返回None request被继续交给下一个中间件进行处理
  8.         # - or return a Response object  返回一个response对象，不会交给下一个process_request,而是直接交给下载器，最后传递给引擎
  9.         # - or return a Request object  返回一个request对象  直接交给引擎处理
  10.         # - or raise IgnoreRequest: process_exception() methods of  # 抛出异常，由process_exception()处理
  11.         #   installed downloader middleware will be called
  12.         return None

image.png
第二个 process_response

  1.     def process_response(self, request, response, spider):
  2.         # Called with the response returned from the downloader.
  3.         # 功能 处理响应 类型 request, response, spider
  5.         # Must either;
  6.         # - return a Response object  # 继续交给下一个中间件处理
  7.         # - return a Request object  # 返回一个request对象  直接交给引擎处理
  8.         # - or raise IgnoreRequest  # 抛出异常 使用process_exception处理

image.png

第三个 process_exception

  1.     def process_exception(self, request, exception, spider):
  2.         # Called when a download handler or a process_request()
  3.         # (from other downloader middleware) raises an exception.
  4.         # 处理异常
  5.         # Must either:
  6.         # - return None: continue processing this exception  # 继续调用其他中间件的process_exception
  7.         # - return a Response object: stops process_exception() chain  # 停止调用其他中间件的process_exception
  8.         # - return a Request object: stops process_exception() chain  # 直接交给引擎处理
  9.         pass

image.png
三种函数横向比较
image.png

自定义User-Agent中间件

多种不同的用户代理用来模拟浏览器登录

image.png

在middlewares中间件中自定义代理中间件

  1. class User_AgentDownloaderMiddleware(object):
  3.     def process_request(self, request, spider):
  4.         # Called for each request that goes through the downloader
  5.         # middleware.
  7.         # Must either:
  8.         # - return None: continue processing this request
  9.         # - or return a Response object
  10.         # - or return a Request object
  11.         # - or raise IgnoreRequest: process_exception() methods of
  12.         #   installed downloader middleware will be called
  13.         return None
  15.     def process_response(self, request, response, spider):
  16.         # Called with the response returned from the downloader.
  18.         # Must either;
  19.         # - return a Response object
  20.         # - return a Request object
  21.         # - or raise IgnoreRequest
  22.         return response
  24.     def process_exception(self, request, exception, spider):
  25.         # Called when a download handler or a process_request()
  26.         # (from other downloader middleware) raises an exception.
  28.         # Must either:
  29.         # - return None: continue processing this exception
  30.         # - return a Response object: stops process_exception() chain
  31.         # - return a Request object: stops process_exception() chain
  32.         pass

image.png

将浏览器代理信息复制到settings文件中，以包的形式导入进middlewares

尤其应该注意路径名的书写，同级目录下导入文件，直接写文件名即可
image.png

与以上操作相同

将IP代理信息复制到settings,在middlewares中重新制作一个自定义代理下载中间件,同时从settings中导入到此文件

细节:注意IP地址

  1.     def process_request(self, request, spider):
  2.         # Called for each request that goes through the downloader
  3.         # middleware.
  5.         # Must either:
  6.         # - return None: continue processing this request
  7.         # - or return a Response object
  8.         # - or return a Request object
  9.         # - or raise IgnoreRequest: process_exception() methods of
  10.         #   installed downloader middleware will be called
  11.         return None
  13.     def process_response(self, request, response, spider):
  14.         # Called with the response returned from the downloader.
  16.         # Must either;
  17.         # - return a Response object
  18.         # - return a Request object
  19.         # - or raise IgnoreRequest
  20.         return response
  22.     def process_exception(self, request, exception, spider):
  23.         # Called when a download handler or a process_request()
  24.         # (from other downloader middleware) raises an exception.
  26.         # Must either:
  27.         # - return None: continue processing this exception
  28.         # - return a Response object: stops process_exception() chain
  29.         # - return a Request object: stops process_exception() chain
  30.         pass

image.png

开启下载中间件

image.png

关闭特定下载中间件

  1. DOWNLOADER_MIDDLEWARES = {
  2.    'github.middlewares.GithubDownloaderMiddleware': 543,
  3.    'scrapy.downloadermiddlewares.retry.RetryMiddleware' : None
  4. }

image.png

设定指定下载中间件的优先级

  1. DOWNLOADER_MIDDLEWARES = {
  2.    'github.middlewares.GithubDownloaderMiddleware': 543,
  3.    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware' : 1024,
  4.    'scrapy.downloadermiddlewares.retry.RetryMiddleware' : None
  5. }

image.png

常用Scrapy.settings

第一部分

  1. BOT_NAME = 'baidu'  # scrapy  项目名称
  3. SPIDER_MODULES = ['baidu.spiders']  # 爬虫模块
  4. NEWSPIDER_MODULE = 'baidu.spiders'  # 使用genspider 命令创建的爬虫模板

image.png

第二部分

  1. # Obey robots.txt rules
  2. ROBOTSTXT_OBEY = False
  4. # Configure maximum concurrent requests performed by Scrapy (default: 16)
  5. CONCURRENT_REQUESTS = 32  # 通过scrapy设置最大的并发请求
  7. # Configure a delay for requests for the same website (default: 0)
  8. # See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
  9. # See also autothrottle settings and docs
  10. DOWNLOAD_DELAY = 3

image.png

第三部分

  1. # Obey robots.txt rules
  2. ROBOTSTXT_OBEY = False
  4. # Configure maximum concurrent requests performed by Scrapy (default: 16)
  5. CONCURRENT_REQUESTS = 32  # 通过scrapy设置最大的并发请求
  7. # Configure a delay for requests for the same website (default: 0)
  8. # See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
  9. # See also autothrottle settings and docs
  10. DOWNLOAD_DELAY = 3
  12. # The download delay setting will honor only one of:
  13. # CONCURRENT_REQUESTS_PER_DOMAIN = 16  # 单个域名运行的并发请求
  14. CONCURRENT_REQUESTS_PER_IP = 16  # 单个IP允许运行的并发请求
  16. # Disable cookies (enabled by default)
  17. COOKIES_ENABLED = False
  19. # Disable Telnet Console (enabled by default)
  20. # TELNETCONSOLE_ENABLED = False  # 设置控制台监控

image.png

image.png

第四部分

image.png

第五部分

  1. # Enable or disable extensions
  2. # See https://docs.scrapy.org/en/latest/topics/extensions.html
  3. # 选择开启或者关闭监听控制台
  4. EXTENSIONS = {
  5.    'scrapy.extensions.telnet.TelnetConsole': None,
  6. }
  8. # Configure item pipelines
  9. # See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
  10. ITEM_PIPELINES = {
  11.    'baidu.pipelines.BaiduPipeline': 300,  # 数字代表优先级 数字越大优先级越高
  12. }
  14. # 启用或者配置扩展
  15. # Enable and configure the AutoThrottle extension (disabled by default)
  16. # See https://docs.scrapy.org/en/latest/topics/autothrottle.html
  17. # AUTOTHROTTLE_ENABLED = True
  18. # The initial download delay  # 初始下载延迟
  19. AUTOTHROTTLE_START_DELAY = 5
  20. # The maximum download delay to be set in case of high latencies  # 最大下载延迟
  21. AUTOTHROTTLE_MAX_DELAY = 60
  22. # The average number of requests Scrapy should be sending in parallel to
  23. # each remote server
  24. # AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
  25. # 启用显示每一个响应的调节信息
  26. # Enable showing throttling stats for every response received:
  27. AUTOTHROTTLE_DEBUG = False

image.png

项目总结

这一个项目没有具体的构造一个获取数据的过程，主要是学习三个主要的下载中间件，之后用户配置下载中间件，最后讲解一些主要的settings设置。

这一节是对前面项目的补充。之前对settings的了解主要停留在设置延时，关闭对爬虫协议的遵循，然后是设置请求头，最后是开启日志。而在这一个章节，我们不仅学习了自定义下载中间件，设置爬虫的IP代理和用户池，还监视了一个叫做控制台监听的功能，虽然不知道具体做什么，但开了一个头。