Home » Programming » Python » Web Scraping
📌 Pin 🔗 Copy Link 📋 Copy Markdown ⬇️ Download
2 min read

Python Web Scraping Guide

Web scraping is the process of extracting data from websites. Python is the go-to language for this due to its powerful libraries like requests and BeautifulSoup.

Prerequisites

The Workflow

  1. Inspect: Use your browser's Developer Tools to understand the structure of the website.
  2. Request: Fetch the HTML content using requests.
  3. Parse: Parse the HTML using BeautifulSoup.
  4. Extract: Find the specific data you need.
  5. Store: Save the data (CSV, JSON, Database).

Example: Scraping Quotes

We will scrape quotes from http://quotes.toscrape.com.

import requests
from bs4 import BeautifulSoup
import csv

# 1. Request the page
url = "http://quotes.toscrape.com"
response = requests.get(url)

# Check if request was successful
if response.status_code == 200:
    # 2. Parse the HTML
    soup = BeautifulSoup(response.text, "html.parser")

    quotes_data = []

    # 3. Extract Data
    # Find all elements with class "quote"
    quotes = soup.find_all("div", class_="quote")

    for quote in quotes:
        text = quote.find("span", class_="text").get_text()
        author = quote.find("small", class_="author").get_text()
        tags = [tag.get_text() for tag in quote.find_all("a", class_="tag")]

        quotes_data.append({
            "text": text,
            "author": author,
            "tags": ", ".join(tags)
        })

    # 4. Store Data
    with open("quotes.csv", "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=["text", "author", "tags"])
        writer.writeheader()
        writer.writerows(quotes_data)

    print("Scraping complete. Saved to quotes.csv")
else:
    print("Failed to retrieve the page")

Best Practices & Ethics

  1. Check robots.txt: Always check website.com/robots.txt to see if scraping is allowed.
  2. Rate Limiting: Do not hammer the server with requests. Use time.sleep() between requests.
  3. User-Agent: Some sites block default python user agents. Set a custom one in headers.
headers = {'User-Agent': 'Mozilla/5.0 (MyScraperBot 1.0)'}
requests.get(url, headers=headers)

programming/python/python

Created: March 17, 2026 • Last Modified: January 28, 2026