# Python Web Scraping with BeautifulSoup

BeautifulSoup is a Python library for pulling data out of HTML and XML files. It works with your favorite parser to provide idiomatic ways of navigating, searching, and modifying the parse tree.

## Installation

To use BeautifulSoup, you need to install it along with a parser (like `lxml` or `html5lib`) and a library to make HTTP requests (like `requests`).

```bash
pip install beautifulsoup4 requests lxml
```

## Basic Usage

First, you need to fetch the HTML content of the page you want to scrape.

```python
import requests
from bs4 import BeautifulSoup

url = 'https://example.com'
response = requests.get(url)

# Create a BeautifulSoup object
soup = BeautifulSoup(response.text, 'lxml')

# Print the formatted HTML
print(soup.prettify())
```

## Navigating the Tree

You can navigate the parse tree using tag names.

```python
print(soup.title)
# <title>Example Domain</title>

print(soup.title.string)
# Example Domain

print(soup.h1)
# <h1>Example Domain</h1>
```

## Searching the Tree

### `find()` and `find_all()`

*   `find()`: Returns the first occurrence of a tag.
*   `find_all()`: Returns a list of all occurrences.

```python
# Find the first 'p' tag
p_tag = soup.find('p')
print(p_tag.get_text())

# Find all 'a' tags
links = soup.find_all('a')
for link in links:
    print(link.get('href'))
```

### Searching by Attributes

You can search for tags with specific attributes (id, class, etc.).

```python
# Find element by ID
element = soup.find(id="main-content")

# Find elements by Class (use class_ because class is a reserved keyword)
items = soup.find_all(class_="item")
```

## CSS Selectors

You can use the `.select()` method to find tags using CSS selectors.

```python
# Select all 'a' tags inside a 'div' with class 'content'
links = soup.select('div.content a')
```

[[programming/python/python]]