多线程与多进程爬取数据

爬取的网站为：https://materialsweb.org/

代码：

略去"Not FOund", 运行结果如下：

thread_pool 4, costTime: 311.579127s ret.size: 780

process_pool 4, costTime: 340.535949s ret.size: 780

thread_pool 5, costTime: 251.317034s ret.size: 780

process_pool 5, costTime: 481.923806s ret.size: 780

thread_pool 6, costTime: 259.729788s ret.size: 780

process_pool 6, costTime: 214.018717s ret.size: 780

thread_pool 7, costTime: 187.721970s ret.size: 780

process_pool 7, costTime: 184.847098s ret.size: 780

thread_pool 8, costTime: 162.505511s ret.size: 780

process_pool 8, costTime: 190.380221s ret.size: 780