Polite Crawler

A polite web crawler built with PHP that respects rate limits, robots.txt directives, and stores crawled data in MySQL. It uses Redis for rate limiting and rotates user agents to mimic human behavior.

Features

• Rate Limiting: Configurable requests per second per domain using Redis.
• Robots.txt Compliance: Automatically checks and respects robots.txt files.
• User Agent Rotation: Rotates through various user agents to avoid detection.
• MySQL Storage: Stores crawled URLs, content, and metadata.
• Symfony Console: Command-line interface for easy usage.
• Docker Support: Easy setup with Docker Compose.

Requirements

• PHP 8.1 or higher
• Composer
• Docker and Docker Compose
• MySQL 8.0
• Redis

Installation

1. Clone the repository:

   git clone https://github.com/franclin/polite-crawler.git
   cd polite-crawler

2. Install PHP dependencies:

   composer install

3. Set up environment variables. Copy the example .env file or create one:

   cp .env.example .env

   Edit .env with your database and Redis settings.

4. Start the services with Docker Compose:

   docker-compose up -d

   This will start MySQL and Redis containers. The database migration will run automatically.

Usage

Use the convenience script to run the crawler:

./crawl-docker.sh <starting-url> [options]

Options

• --depth, -d: Maximum crawl depth (default: 2)
• --max-urls, -m: Maximum URLs to crawl (default: 100)
• --rate-limit, -r: Requests per second per domain (default: 1)
• --verbose, -v: Enable verbose output

Example

./crawl-docker.sh https://example.com --depth 3 --max-urls 500 --rate-limit 2

This will crawl up to 500 URLs starting from https://example.com, with a maximum depth of 3, and a rate limit of 2 requests per second per domain.

Configuration

The crawler uses environment variables for configuration. Create a .env file in the project root with the following variables:

DB_HOST=mysql
DB_PORT=3306
DB_NAME=crawler_db
DB_USER=crawler
DB_PASSWORD=crawler123
REDIS_HOST=redis
REDIS_PORT=6379

Database Schema

The project includes a migration file (migrations/001_create_crawler_tables.sql) that creates the necessary tables for storing crawled data.

Development

To run the crawler without Docker:

1. Ensure MySQL and Redis are running locally.
2. Update the .env file with local host settings.
3. Run the command directly:

   php crawl <starting-url> [options]

License

This project is open source. Please check the license file for details.