# Python Web Crawler with Scrapy

This guide demonstrates how to build a web crawler using **Scrapy**. While Scrapy is typically used to generate full projects with multiple files, this guide shows how to run a spider from a single script for simplicity. It crawls a quote website, extracts data, and follows pagination links automatically.

**Modules Used:**
*   `scrapy`: A fast high-level web crawling and web scraping framework.

## Installation

```bash
pip install scrapy
```

## The Code

Save this as `crawler.py`.

```python
import scrapy
from scrapy.crawler import CrawlerProcess

class QuoteSpider(scrapy.Spider):
    """
    A simple Spider that navigates to a URL, extracts data, 
    and follows pagination links.
    """
    name = "quotes"
    
    # The starting URL(s)
    start_urls = [
        'http://quotes.toscrape.com/page/1/',
    ]

    def parse(self, response):
        """
        This method handles the response downloaded for each of the 
        requests made.
        """
        
        # 1. Extract Data using CSS Selectors
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').get(),
                'author': quote.css('small.author::text').get(),
                'tags': quote.css('div.tags a.tag::text').getall(),
            }

        # 2. Follow Pagination
        # Find the link to the next page
        next_page = response.css('li.next a::attr(href)').get()
        
        if next_page is not None:
            # Schedule the next request
            # response.follow automatically handles relative URLs
            yield response.follow(next_page, callback=self.parse)

if __name__ == "__main__":
    # 3. Configure and Run the Crawler
    process = CrawlerProcess(settings={
        # Save output to a JSON file
        'FEEDS': {
            'quotes_data.json': {'format': 'json', 'overwrite': True},
        },
        # Reduce console noise
        'LOG_LEVEL': 'INFO',
        # Identify yourself (good practice)
        'USER_AGENT': 'Mozilla/5.0 (compatible; MyScraper/1.0)',
    })

    process.crawl(QuoteSpider)
    process.start() # the script will block here until the crawling is finished
```

## Usage

```bash
python crawler.py
```

[[programming/python/python]]