爬虫的本质就是模拟浏览器与网站服务器通信,核心流程只有三步:
打开终端,安装以下库:
pip install requests beautifulsoup4 lxml pandas库名 | 用途 |
|---|---|
requests | 发送 HTTP 请求,获取网页内容 |
beautifulsoup4 | 解析 HTML,提取数据 |
lxml | BeautifulSoup 的高性能解析器 |
pandas | 数据整理与导出 CSV/Excel |
import requests
from bs4 import BeautifulSoup
url = "https://wlt.dxbdqn.cn/"
response = requests.get(url)
soup = BeautifulSoup(response.text, "lxml")
print(soup.title.text)运行后会输出类似 All products | Books to Scrape 的网页标题。
下面是一个完整的爬虫示例,抓取图书的书名、价格、评分和库存状态,并支持翻页批量抓取。
import requests
import time
import random
import csv
from bs4 import BeautifulSoup
def fetch_page(url):
"""发送请求并返回 BeautifulSoup 对象"""
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/120.0.0.0 Safari/537.36"
}
try:
resp = requests.get(url, headers=headers, timeout=10)
resp.raise_for_status()
return BeautifulSoup(resp.text, "lxml")
except requests.RequestException as e:
print(f"请求失败: {url} - {e}")
return None
def parse_books(soup):
"""从页面中解析所有图书信息"""
books = []
articles = soup.find_all("article", class_="product_pod")
for article in articles:
# 书名在 h3 > a 的 title 属性中
title_tag = article.find("h3").find("a")
title = title_tag.get("title", "")
# 价格
price = article.find("p", class_="price_color").get_text(strip=True)
# 评分(class 中包含 One/Two/Three/Four/Five)
rating_map = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}
star_tag = article.find("p", class_="star-rating")
rating = rating_map.get(star_tag["class"][1], 0) if star_tag else 0
# 库存状态
stock = article.find("p", class_="instock availability").get_text(strip=True)
books.append({
"书名": title,
"价格": price,
"评分": rating,
"库存": stock
})
return books
def crawl_all_books(total_pages=5):
"""批量翻页爬取"""
all_books = []
base_url = "https://wlt.dxbdqn.cn/catalogue/page-{}.html"
for page in range(1, total_pages + 1):
if page == 1:
url = "https://wlt.dxbdqn.cn/"
else:
url = base_url.format(page)
print(f"正在爬取第 {page} 页: {url}")
soup = fetch_page(url)
if soup is None:
continue
books = parse_books(soup)
all_books.extend(books)
print(f" 本页获取 {len(books)} 本书")
# 礼貌延迟,避免给服务器造成压力
time.sleep(1 + random.random() * 2)
return all_books
def save_to_csv(books, filename="books.csv"):
"""保存为 CSV 文件"""
if not books:
print("没有数据可保存")
return
with open(filename, "w", newline="", encoding="utf-8-sig") as f:
writer = csv.DictWriter(f, fieldnames=["书名", "价格", "评分", "库存"])
writer.writeheader()
writer.writerows(books)
print(f"✅ 已保存 {len(books)} 条数据到 {filename}")
if __name__ == "__main__":
all_books = crawl_all_books(total_pages=5)
save_to_csv(all_books)User-Agent 请求头:模拟浏览器访问,避免被简单识别为爬虫。timeout=10:设置超时,防止请求卡死。time.sleep() 随机延迟:每次请求间隔 1~3 秒,体现礼貌抓取原则。raise_for_status():自动检测 4xx/5xx 错误。utf-8-sig 编码:确保 CSV 在 Excel 中打开时中文不乱码。BeautifulSoup 支持 CSS 选择器,语法更简洁:
from bs4 import BeautifulSoup
soup = BeautifulSoup(html_text, "lxml")
# 等价于 find_all("a", class_="title")
titles = soup.select("a.title")
# 提取文本
for t in titles:
print(t.get_text(strip=True))
# 提取属性
links = soup.select("article.product_pod h3 a")
for link in links:
print(link["title"], link["href"])如果网页内容由 JavaScript 动态加载(如滚动加载更多、点击后显示数据),requests 只能拿到空壳 HTML。此时可以考虑:
方案 | 适用场景 |
|---|---|
检查 API 接口 | 浏览器开发者工具 → Network 面板,看是否有 JSON 接口直接返回数据 |
Selenium / Playwright | 模拟真实浏览器,执行 JS 后获取渲染后的页面 |
Scrapy + Splash | 大规模抓取 + JS 渲染的场景 |
Selenium 简单示例:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from bs4 import BeautifulSoup
options = Options()
options.add_argument("--headless")
driver = webdriver.Chrome(options=options)
driver.get("https://quotes.toscrape.com/js")
soup = BeautifulSoup(driver.page_source, "lxml")
for q in soup.select(".quote"):
print(q.select_one(".text").get_text(strip=True))
driver.quit()场景 | 推荐方案 |
|---|---|
静态页面,快速上手 | requests + BeautifulSoup |
大规模批量抓取 | Scrapy 框架 |
动态 JS 页面 | Playwright / Selenium |
高并发异步抓取 | httpx / aiohttp |
网站提供 API | 优先用 API,最稳定合规 |
一个规范的爬虫项目应该做到:
技术是工具,合规是底线。用得好,爬虫是高效的数据采集利器;用不好,可能带来法律风险。祝你在合法合规的前提下,愉快地探索 Python 爬虫世界!
原创声明:本文系作者授权腾讯云开发者社区发表,未经许可,不得转载。
如有侵权,请联系 cloudcommunity@tencent.com 删除。