我在尝试抓取时ConnectionResetError：[Errno 54]连接被对等重置

基础概念

ConnectionResetError: [Errno 54] 是一个常见的网络错误，表示在尝试建立或维持网络连接时，连接被对端（服务器）重置。这通常发生在以下几种情况：

服务器端主动关闭连接：服务器可能因为各种原因（如负载过高、安全策略、请求超时等）主动关闭连接。
网络问题：中间网络设备（如路由器、防火墙）可能因为各种原因（如网络拥塞、配置错误等）重置连接。
客户端问题：客户端可能在短时间内发送大量请求，导致服务器或中间设备认为这是攻击行为而重置连接。

解决方法

1. 客户端重试机制

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

def requests_retry_session(
    retries=3,
    backoff_factor=0.3,
    status_forcelist=(500, 502, 503, 504, 408),
    session=None,
):
    session = session or requests.Session()
    retry = Retry(
        total=retries,
        read=retries,
        connect=retries,
        backoff_factor=backoff_factor,
        status_forcelist=status_forcelist,
    )
    adapter = HTTPAdapter(max_retries=retry)
    session.mount('http://', adapter)
    session.mount('https://', adapter)
    return session

try:
    response = requests_retry_session().get('https://example.com')
    response.raise_for_status()
except requests.exceptions.RequestException as e:
    print(f"Error: {e}")

2. 使用连接池

import requests
from requests.adapters import HTTPAdapter

session = requests.Session()
adapter = HTTPAdapter(pool_connections=100, pool_maxsize=100)
session.mount('http://', adapter)
session.mount('https://', adapter)

try:
    response = session.get('https://example.com')
    response.raise_for_status()
except requests.exceptions.RequestException as e:
    print(f"Error: {e}")

3. 设置合理的超时时间

import requests

try:
    response = requests.get('https://example.com', timeout=(3.05, 27))
    response.raise_for_status()
except requests.exceptions.RequestException as e:
    print(f"Error: {e}")