第二步，在settings文件中设置 “DOWNLOADER_MIDDLEWARES”

# Enable or disable downloader middlewares
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
DOWNLOADER_MIDDLEWARES = {
   #'borenzhuan.middlewares.BorenzhuanDownloaderMiddleware': 543,
    'borenzhuan.middlewares.RandomUserAgentMiddleware': 400,
}

第三步，在 middlewares.py 文件中导入 settings模块中的 USER_AGENT_LIST 方法

from borenzhuan.settings import USER_AGENT_LIST # 导入中间件
import random

class RandomUserAgentMiddleware(object):
    def process_request(self, request, spider):
        rand_use  = random.choice(USER_AGENT_LIST)
        if rand_use:
            request.headers.setdefault('User-Agent', rand_use)

好了，随机的UA已经实现，你可以在parse函数中编写如下代码进行测试

print(response.request.headers)

4. B站博人传评论数据----完善item

这个操作相对简单，这些数据就是我们要保存的数据了。！

   author = scrapy.Field()
    content = scrapy.Field()
    ctime = scrapy.Field()
    disliked = scrapy.Field()
    liked = scrapy.Field()
    likes = scrapy.Field()
    score = scrapy.Field()
    user_season = scrapy.Field()

5. B站博人传评论数据案例---提高爬取速度

在settings.py中设置如下参数：

# Configure maximum concurrent requests performed by Scrapy (default: 16)
CONCURRENT_REQUESTS = 32
# Configure a delay for requests for the same website (default: 0)
# See https://doc.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
DOWNLOAD_DELAY = 1
# The download delay setting will honor only one of:
CONCURRENT_REQUESTS_PER_DOMAIN = 16
CONCURRENT_REQUESTS_PER_IP = 16
# Disable cookies (enabled by default)
COOKIES_ENABLED = False

解释说明

一、降低下载延迟

DOWNLOAD_DELAY = 0

将下载延迟设为0，这时需要相应的防ban措施，一般使用user agent轮转，构建user agent池，轮流选择其中之一来作为user agent。

二、多线程

CONCURRENT_REQUESTS = 32
CONCURRENT_REQUESTS_PER_DOMAIN = 16
CONCURRENT_REQUESTS_PER_IP = 16

scrapy网络请求是基于Twisted，而Twisted默认支持多线程，而且scrapy默认也是通过多线程请求的，并且支持多核CPU的并发，我们通过一些设置提高scrapy的并发数可以提高爬取速度。

三、禁用cookies

COOKIES_ENABLED = False

6. B站博人传评论数据案例---保存数据

最后在pipelines.py 文件中，编写保存代码即可

import os
import csv

class BorenzhuanPipeline(object):


    def __init__(self):
        store_file = os.path.dirname(__file__)+'/spiders/bore.csv'
        self.file = open(store_file,"a+",newline="",encoding="utf-8")
        self.writer = csv.writer(self.file)

    def process_item(self, item, spider):
        try:

            self.writer.writerow((
                item["author"],
                item["content"],
                item["ctime"],
                item["disliked"],
                item["liked"],
                item["likes"],
                item["score"],
                item["user_season"]
            ))

        except Exception as e:
            print(e.args)

        def close_spider(self, spider):
            self.file.close()

运行代码之后，发现过了一会报错了