案例：使用BeautifuSoup4的爬虫

我们以 亚马逊Kindle电子书销售排行榜 商品页面来做演示：https://www.amazon.cn/gp/bestsellers/digital-text/116169071
TIM截图20181025174133.png
使用BeautifuSoup4解析器，将每件商品的的ASIN、标题、价格、star、评价数量，以及每件商品的链接爬取下来并存储在.csv文件中。

  1. import csv
  2. import requests
  3. from bs4 import BeautifulSoup
  6. def amazon():
  7.     base_url = 'https://www.amazon.cn'
  8.     url = 'https://www.amazon.cn/gp/bestsellers/digital-text/116169071'
  9.     headers = {
  10.         'Connection': 'keep-alive',
  11.         'Cache-Control': 'max-age=0',
  12.         'Upgrade-Insecure-Requests': '1',
  13.         'DNT': '1',
  14.         'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36',
  15.         'Accept': 'image/webp,image/apng,image/*,*/*;q=0.8',
  16.         'Accept-Encoding': 'gzip, deflate, br',
  17.         'Accept-Language': 'zh-HK,zh-CN;q=0.9,zh;q=0.8',
  18.         'Referer': 'https://www.amazon.cn/gp/bestsellers/digital-text/116169071',
  19.     }
  20.     # 给亚马逊发送请求
  21.     session = requests.session()
  22.     session.get('https://www.amazon.com')
  23.     resHtml = session.get(url, headers=headers).content.decode('utf-8')
  24.     # 将获取的对象转化为bs4对象
  25.     html_soup = BeautifulSoup(resHtml, 'lxml')
  26.     # 获取所有商品的标签
  27.     all_goods_li = html_soup.find('ol', id='zg-ordered-list').find_all('li', 'zg-item-immersion')
  28.     for li in all_goods_li:
  29.         # 准备一个空列表，用于储存商品信息
  30.         goods_info_list = []
  31.         # 商品链接
  32.         link = base_url + li.find('a', target='_blank')['href']
  33.         # 商品 asin
  34.         asin = link.split('/dp/')[1].split('/')[0]
  35.         # 标题
  36.         title = li.select("div[data-rows='1']")[0].get_text().strip()
  37.         # 价格
  38.         price = li.find('span', 'p13n-sc-price').text
  39.         # 星级
  40.         star = li.find('span', 'a-icon-alt').text
  41.         # 评价数
  42.         reviews = li.select("a[class='a-size-small a-link-normal']")[0].get_text()
  44.         # 将爬到的数据添加到列表中
  45.         goods_info_list.append(asin)
  46.         goods_info_list.append(title)
  47.         goods_info_list.append(price)
  48.         goods_info_list.append(star)
  49.         goods_info_list.append(reviews)
  50.         goods_info_list.append(link)
  52.         # 将数据写入 名为amazon_book.csv的文件中
  53.         csvFile = open('./amazon_book.csv', 'a', newline='', encoding='gb18030')  # 设置newline，否则两行之间会空一行
  54.         writer = csv.writer(csvFile)
  55.         writer.writerow(goods_info_list)
  56.         csvFile.close()
  59. if __name__ == '__main__':
  60.     amazon()

打开当前目录下的 amazon_book.csv 可见如图效果：
TIM截图20181025174651.png

Copyright © 黑五电商学院 amzfriday.com all right reserved