我用了一些代理程序来爬行一些网站。下面是我在settings.py做的事情:
# Retry many times since proxies often fail
RETRY_TIMES = 10
# Retry on most error codes since proxies fail for different reasons
RETRY_HTTP_CODES = [500, 503, 504, 400, 403, 404, 408]
DOWNLOAD_DELAY = 3 # 5,000 ms of delay
DOWNLOADER_MIDDLEWARES = {