Beautiful Soup is a
Python package for parsing
HTML
The HyperText Markup Language or HTML is the standard markup language for documents designed to be displayed in a web browser. It can be assisted by technologies such as Cascading Style Sheets (CSS) and scripting languages such as JavaScri ...
and
XML
Extensible Markup Language (XML) is a markup language and file format for storing, transmitting, and reconstructing arbitrary data. It defines a set of rules for encoding documents in a format that is both human-readable and machine-readable. ...
documents, including those with malformed markup. It creates a parse tree for documents that can be used to extract data from HTML, which is useful for
web scraping
Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites. Web scraping software may directly access the World Wide Web using the Hypertext Transfer Protocol or a web browser. While web scrapin ...
.
Beautiful Soup was started by Leonard Richardson, who continues to contribute to the project, and is additionally supported by Tidelift, a paid subscription to open-source maintenance.
Code example
Beautiful Soup represents parsed data as a tree which can be searched and iterated over with ordinary Python
loops. The example below uses the Python
standard library's urllib to load
Wikipedia
Wikipedia is a multilingual free online encyclopedia written and maintained by a community of volunteers, known as Wikipedians, through open collaboration and using a wiki-based editing system. Wikipedia is the largest and most-read ref ...
's main page, then uses Beautiful Soup to parse the document and search for all links within.
#!/usr/bin/env python3
# Anchor extraction from HTML document
from bs4 import BeautifulSoup
from urllib.request import urlopen
with urlopen('https://en.wikipedia.org/wiki/Main_Page') as response:
soup = BeautifulSoup(response, 'html.parser')
for anchor in soup.find_all('a'):
print(anchor.get('href', '/'))
History
Beautiful Soup is named both after a poem in
Alice's Adventures in Wonderland
''Alice's Adventures in Wonderland'' (commonly ''Alice in Wonderland'') is an 1865 English novel by Lewis Carroll. It details the story of a young girl named Alice who falls through a rabbit hole into a fantasy world of anthropomorphic creatur ...
and
tag soup.
Beautiful Soup 3 was the official release line of Beautiful Soup from May 2006 to March 2012. The current release i
Beautiful Soup 4.x Beautiful Soup 4 can be installed with
pip install beautifulsoup4.
In 2021, Python 2.7 support was retired and the release 4.9.3 was the last to support Python 2.7.
See also
*
Comparison of HTML parsers
HTML parsers are software for automated Hypertext Markup Language (HTML) parsing. They have two main purposes:
* HTML traversal: offer an interface for programmers to easily access and modify the "HTML string code". Canonical example: DOM par ...
*
jsoup
jsoup is an open-source Java library designed to parse, extract, and manipulate data stored in HTML documents.
History
jsoup was created in 2009 by Jonathan Hedley. It is distributed it under the MIT License, a permissive free software license s ...
*
Nokogiri
References
{{Reflist
Python (programming language) libraries
Software using the MIT license
HTML parsers
XML parsers