requests 获取网页 HTML。BeautifulSoup 解析 HTML 结构。对于静态页面,requests + BeautifulSoup 已经足够轻量、好用;如果页面内容主要由 JavaScript 动态生成,则需要考虑 Selenium、Playwright 等浏览器自动化工具。
pip install requests beautifulsoup4import csv
import time
import random
import requests
from bs4 import BeautifulSoup
BASE_URL = "https://wlt.ddjoyhome.com"
HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/120.0.0.0 Safari/537.36"
}
def fetch_page(url: str) -> str:
"""获取网页 HTML,失败时抛出异常"""
resp = requests.get(url, headers=HEADERS, timeout=10)
resp.raise_for_status()
return resp.text
def parse_quotes(html: str) -> list[dict]:
"""从页面 HTML 中提取名言、作者和标签"""
soup = BeautifulSoup(html, "html.parser")
quotes = []
for quote_box in soup.select(".quote"):
text = quote_box.select_one(".text")
author = quote_box.select_one(".author")
tags = quote_box.select(".tag")
quotes.append({
"text": text.get_text(strip=True) if text else "",
"author": author.get_text(strip=True) if author else "",
"tags": "|".join(t.get_text(strip=True) for t in tags),
})
return quotes
def save_to_csv(rows: list[dict], filename: str = "quotes.csv") -> None:
"""将结果保存为 CSV"""
if not rows:
print("没有抓取到数据")
return
fieldnames = ["text", "author", "tags"]
with open(filename, "w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=fieldnames)
writer.writeheader()
writer.writerows(rows)
print(f"已保存 {len(rows)} 条数据到 {filename}")
def main() -> None:
all_quotes: list[dict] = []
for page in range(1, 4): # 先抓取前 3 页
url = f"{BASE_URL}/page/{page}"
print(f"正在抓取: {url}")
try:
html = fetch_page(url)
except Exception as e:
print(f"请求失败: {e}")
break
quotes = parse_quotes(html)
all_quotes.extend(quotes)
# 礼貌延时,避免请求过快
time.sleep(1 + random.random() * 2)
save_to_csv(all_quotes)
if __name__ == "__main__":
main()部分 | 作用 |
|---|---|
requests.get | 发送 HTTP 请求,获取网页源码 |
timeout=10 | 防止请求长时间卡死 |
User-Agent | 模拟浏览器请求,降低被拦截概率 |
BeautifulSoup | 解析 HTML,方便提取数据 |
soup.select(".quote") | 使用 CSS 选择器定位每个名言块 |
time.sleep | 控制请求频率,避免给服务器造成压力 |
requests 可能拿不到完整数据,可改用 Playwright 或 Selenium。原创声明:本文系作者授权腾讯云开发者社区发表,未经许可,不得转载。
如有侵权,请联系 cloudcommunity@tencent.com 删除。