scrapy-fake-useragent包官网:https://pypi.org/project/scrapy-fake-useragent/ 安装: pip install scrapy-fake-useragent
使用:
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None, # 关闭默认方法
'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400, # 开启
}
这里选用 http://httpbin.org/get 作为测试网站 测试结果:
本文参考自高级农名工:https://www.makcyun.top/web_scraping_withpython14.html