# Python Web Scraping Guide

Web scraping is the process of extracting data from websites. Python is the go-to language for this due to its powerful libraries like `requests` and `BeautifulSoup`.

## Prerequisites

*   [[programming/python/modules/requests-module|Requests Module]]
*   [[programming/python/modules/beautifulsoup-module|Beautiful Soup Module]]

## The Workflow

1.  **Inspect**: Use your browser's Developer Tools to understand the structure of the website.
2.  **Request**: Fetch the HTML content using `requests`.
3.  **Parse**: Parse the HTML using `BeautifulSoup`.
4.  **Extract**: Find the specific data you need.
5.  **Store**: Save the data (CSV, JSON, Database).

## Example: Scraping Quotes

We will scrape quotes from `http://quotes.toscrape.com`.

```python
import requests
from bs4 import BeautifulSoup
import csv

# 1. Request the page
url = "http://quotes.toscrape.com"
response = requests.get(url)

# Check if request was successful
if response.status_code == 200:
    # 2. Parse the HTML
    soup = BeautifulSoup(response.text, "html.parser")
    
    quotes_data = []

    # 3. Extract Data
    # Find all elements with class "quote"
    quotes = soup.find_all("div", class_="quote")

    for quote in quotes:
        text = quote.find("span", class_="text").get_text()
        author = quote.find("small", class_="author").get_text()
        tags = [tag.get_text() for tag in quote.find_all("a", class_="tag")]
        
        quotes_data.append({
            "text": text,
            "author": author,
            "tags": ", ".join(tags)
        })

    # 4. Store Data
    with open("quotes.csv", "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=["text", "author", "tags"])
        writer.writeheader()
        writer.writerows(quotes_data)
        
    print("Scraping complete. Saved to quotes.csv")
else:
    print("Failed to retrieve the page")
```

## Best Practices & Ethics

1.  **Check `robots.txt`**: Always check `website.com/robots.txt` to see if scraping is allowed.
2.  **Rate Limiting**: Do not hammer the server with requests. Use `time.sleep()` between requests.
3.  **User-Agent**: Some sites block default python user agents. Set a custom one in headers.

```python
headers = {'User-Agent': 'Mozilla/5.0 (MyScraperBot 1.0)'}
requests.get(url, headers=headers)
```

[[programming/python/python]]