Python Web Scraper Builder
Generate a robust Python web scraper with error handling, rate limiting, and data export to CSV or JSON.
The Prompt
Build a Python web scraper for this use case: Target website type: [e.g., e-commerce products, news articles, job listings, real estate] Data to extract: [list specific fields, e.g., title, price, URL, date, description] Pagination: [yes/no — if yes, describe how pagination works] JavaScript rendering required: [yes (use Playwright) / no (use requests+BeautifulSoup)] Output format: [CSV / JSON / both] Rate limiting: [gentle — 1–2 second delay between requests] Build a complete scraper with: 1. Main scraper class with: - Session management with rotating user agents - Configurable delay between requests - Retry logic with exponential backoff (3 attempts) - Proxy support (optional, commented out) 2. Data extraction functions for each field with: - Primary CSS selector - Fallback selector - Data cleaning/normalization 3. Storage class: - Save to CSV with proper encoding - Save to JSON with formatting - Duplicate detection 4. Main runner: - Command line arguments (URL, output file, max pages) - Progress bar with tqdm - Summary stats on completion 5. requirements.txt Include comments explaining each major section.
Tips for Best Results
- →Always check robots.txt before scraping any site
- →Add a --dry-run flag to test extraction logic without saving data
Tags
Related Prompts
Full-Stack API Generator
Generate a complete REST API with database schema, authentication, CRUD operations, and documentation.
Code GenerationDatabase Schema Designer
Design a complete database schema with tables, relationships, indexes, and migration scripts for your application.
Code GenerationREST API Builder with Authentication
Generate a complete, production-ready REST API with JWT authentication, rate limiting, and full CRUD endpoints.
Code GenerationReact Dashboard Component Builder
Generate a complete analytics dashboard with charts, KPI cards, data tables, and filters using React and Recharts.
Code GenerationSQL Query Optimizer and Explainer
Paste any SQL query and get an optimized version with explanation of what was improved and why.
Code GenerationChrome Extension Builder
Build a functional Chrome extension with popup UI, content scripts, and background service worker from a single prompt.