python利用beautifulSoup实现爬虫

2019-11-25 18:10:20

字体：大中小

来源：转载

供稿：网友

以前讲过利用phantomjs做爬虫抓网页 //www.VeVB.COm/article/55789.htm 是配合选择器做的

利用 beautifulSoup(文档：http://www.crummy.com/software/BeautifulSoup/bs4/doc/)这个python模块，可以很轻松的抓取网页内容

# coding=utf-8import urllibfrom bs4 import BeautifulSoupurl ='http://www.baidu.com/s'values ={'wd':'网球'}encoded_param = urllib.urlencode(values)full_url = url +'?'+ encoded_paramresponse = urllib.urlopen(full_url)soup =BeautifulSoup(response)alinks = soup.find_all('a')

上面可以抓取百度搜出来结果是网球的记录。

beautifulSoup内置了很多非常有用的方法。

几个比较好用的特性：

构造一个node元素

复制代码代码如下:

soup = BeautifulSoup('<b class="boldest">Extremely bold</b>')
tag = soup.b
type(tag)
# <class 'bs4.element.Tag'>

属性可以使用attr拿到，结果是字典

复制代码代码如下:

tag.attrs
# {u'class': u'boldest'}

或者直接tag.class取属性也可。

也可以自由操作属性

tag['class'] = 'verybold'tag['id'] = 1tag# <blockquote class="verybold" id="1">Extremely bold</blockquote>del tag['class']del tag['id']tag# <blockquote>Extremely bold</blockquote>tag['class']# KeyError: 'class'print(tag.get('class'))# None

还可以随便操作，查找dom元素，比如下面的例子

1.构建一份文档

html_doc = """<html><head><title>The Dormouse's story</title></head><p><b>The Dormouse's story</b></p><p>Once upon a time there were three little sisters; and their names were<a href="http://example.com/elsie" id="link1">Elsie</a>,<a href="http://example.com/lacie" id="link2">Lacie</a> and<a href="http://example.com/tillie" id="link3">Tillie</a>;and they lived at the bottom of a well.</p><p>...</p>"""from bs4 import BeautifulSoupsoup = BeautifulSoup(html_doc)

2.各种搞

soup.head# <head><title>The Dormouse's story</title></head>soup.title# <title>The Dormouse's story</title>soup.body.b# <b>The Dormouse's story</b>soup.a# <a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>soup.find_all('a')# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,# <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,# <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]head_tag = soup.headhead_tag# <head><title>The Dormouse's story</title></head>head_tag.contents[<title>The Dormouse's story</title>]title_tag = head_tag.contents[0]title_tag# <title>The Dormouse's story</title>title_tag.contents# [u'The Dormouse's story']len(soup.contents)# 1soup.contents[0].name# u'html'text = title_tag.contents[0]text.contentsfor child in title_tag.children:  print(child)head_tag.contents# [<title>The Dormouse's story</title>]for child in head_tag.descendants:  print(child)# <title>The Dormouse's story</title># The Dormouse's storylen(list(soup.children))# 1len(list(soup.descendants))# 25title_tag.string# u'The Dormouse's story'

上一篇：python服务器端收发请求的实现代码

下一篇：Python中为feedparser设置超时时间避免堵塞