Who PyScrappy is for#
AI engineers building training datasets or RAG pipelines
PyScrappy converts any public web page to clean markdown or JSON in one API call, removing the HTML noise that degrades embedding quality. The batch crawl API lets you queue thousands of URLs and pull results asynchronously, which fits standard ETL pipelines. Built-in scrapers for Wikipedia, GitHub, news feeds, and documentation sites return pre-structured data. Use the MCP server mode to let AI agents scrape web data on demand during inference.
Skip if:
Your dataset sources are already in structured formats (RSS, sitemaps, public APIs). PyScrappy's value is extracting from unstructured HTML. If your sources do not require it, a lightweight HTTP client is simpler.
Data analysts replacing paid scraping services
Teams currently paying $300+/month for ScraperAPI or Diffbot can self-host PyScrappy and eliminate recurring API costs. The adaptive selectors reduce maintenance overhead compared to custom scripts that break on every site update. Built-in scrapers for stock data, e-commerce (Amazon, IKEA, Newegg), and news feeds handle common use cases without writing custom parsers. DataFrame output integrates directly into pandas-based analysis workflows.
Skip if:
You need entity extraction (people, companies, products) from unstructured text. Diffbot's structured entity API is more advanced than PyScrappy's text extraction. PyScrappy gives you clean markdown and JSON; you handle entity recognition separately.
Developer teams building AI agents with MCP
The MCP server exposes all 24 scrapers as tools for Claude, Cursor, or any MCP-compatible agent. An agent can call scrape_wikipedia, scrape_stock, or scrape_url naturally and receive structured data back without the developer writing scraper code. For local models (Ollama), the built-in pyscrappy chat agent lets models that support tool calling (Llama 3.1, Qwen 2.5, Mistral) use the same scrapers directly.
Skip if:
Your AI agent workflow does not need live web data at inference time. If you pre-fetch and embed all sources offline, the MCP server adds unnecessary runtime complexity. Use the Python library in your data pipeline instead.
Developers who need reliable scrapers with minimal maintenance
The adaptive selector system reduces the maintenance burden for long-lived scrapers. When a site redesigns and breaks a CSS selector, PyScrappy relocates the element by structural and textual similarity instead of returning empty. The confidence score tells you how certain the relocation was. For sites that change markup frequently, this is more reliable than hard-coded selectors that silently break.
Skip if:
You are scraping APIs or structured feeds (JSON, XML, RSS) where selectors are not needed. Adaptive selectors solve a problem that only exists when parsing unstructured HTML.
The problem it solves#
Web scraping at production scale faces three major pain points. First, brittle selectors: custom scrapers break silently every time a site updates its HTML structure, requiring constant maintenance and monitoring. Second, anti-bot blocking: many sites detect standard HTTP clients by their TLS fingerprints and serve CAPTCHA challenges or error pages before you see any content. Third, cost at scale: commercial scraping APIs like Diffbot, ScraperAPI, and ScrapeOps charge per-page fees that range from $299 to over $1,000 per month for anything beyond toy projects.
The output quality problem compounds these issues. Raw HTML from dynamic JavaScript sites requires extensive cleaning before it is usable for AI training or RAG pipelines. Most scraping libraries return unstructured markup that needs a separate parsing and normalization step. For teams building LLM-powered systems, this preprocessing overhead adds latency and fragility to the data pipeline.
How it solves it#
Adaptive self-healing selectors
CSS selectors that remember an element's structural fingerprint and relocate it by similarity when a site changes its markup, so scrapers do not silently break after a redesign. The system uses weighted signals (stable id/data attributes count more than sibling order) and volatility-aware text matching (prices and dates are down-weighted). Confidence scores (0-100) tell you how certain the relocation was. Fingerprints persist in a local JSON store namespaced by site.
TLS-fingerprint impersonation (stealth mode)
Bypass anti-bot systems that block plain HTTP clients by mimicking a real browser's TLS/JA3 fingerprint. Set impersonate="chrome" to pass TLS-level detection without running a headless browser. Supports Chrome, Firefox, and Safari fingerprints via the optional curl_cffi backend. Works on the synchronous path; async clients raise a clear error if you set this option.
LLM-ready output formats
Every scrape result converts to clean markdown (.to_markdown()), structured JSON (.to_json()), or pandas DataFrame (.to_dataframe()) with one method call. Markdown output strips HTML noise and preserves semantic structure (headings, lists, tables, links) for direct embedding or LLM context. JSON output is typed and validated. No separate parsing step needed.
24 built-in scrapers for common platforms
Pre-built scrapers for Wikipedia, GitHub, IMDB, stock quotes (Yahoo Finance), news feeds (RSS/Atom), YouTube, Amazon, IKEA, Newegg, Hacker News, LinkedIn jobs, weather, crypto, currency, dictionaries, and more. Each scraper returns structured data in the same ScrapeResult format. GenericScraper handles any URL with auto-extraction of text, links, images, tables, and metadata.
MCP server for AI agents
Run PyScrappy as a Model Context Protocol server that exposes all 24 scrapers as tools for AI agents (Claude, Cursor, local LLMs via Ollama). Agents call scrape_url, scrape_wikipedia, scrape_stock, or search_github as tools and receive structured markdown or JSON back. For local models that lack MCP support, the built-in pyscrappy chat agent talks to Ollama directly and exposes the same 22 tools.
JavaScript rendering with Playwright
Optional Playwright backend for sites that require a browser to render. Scrapes React, Vue, and Angular SPAs by running headless Chromium server-side, so you get the same DOM state a real visitor would see. No Puppeteer or Playwright setup needed on the caller side. Install with pip install 'pyscrappy[browser]' and playwright install chromium.
Strengths and trade-offs#
Strengths
- MIT license with full self-hostingThe entire toolkit is MIT licensed, so you can run it on your infrastructure, modify scrapers, and use it commercially without restrictions. Unlike Diffbot (API-only, per-page billing) or ScraperAPI (managed proxy service with usage tiers), you pay once for the server and own the pipeline. For teams scraping more than a few thousand pages per month, this eliminates recurring API costs entirely.
- Self-healing selectors survive redesignsAdaptive selectors are a genuine innovation over traditional CSS or XPath scrapers. Hard-coded selectors break silently the day a site changes its markup. PyScrappy's fingerprint-based relocation uses weighted structural signals (stable ids and data attributes count more than tag order) and volatility-aware text matching (prices and dates are down-weighted) to find the element again. This reduces maintenance overhead for long-lived scrapers.
- Zero external dependencies for basic scrapingThe core library works with Python 3.9+ and standard HTTP clients, with no required external services or databases. Optional extras (browser support, MCP server, stealth mode, DataFrame output) are truly optional. You can install the base package, write a scraper, and run it on any Python environment without configuring a headless browser or proxy service first.
- Concurrent scraping with built-in rate limitingscrape_many and scrape_all run scrapes in parallel with exponential backoff retries and per-domain rate limiting built in. You pass a list of URLs and get back results as they complete, with failed requests automatically retried up to max_retries times. This is faster than sequential scraping and safer than naive threading without rate limits.
Trade-offs
- -JavaScript rendering requires 2GB+ RAM per workerThe Playwright backend runs headless Chromium, which needs at least 2GB RAM per concurrent worker. Lightweight VPS deployments under 1GB will hit out-of-memory errors on JavaScript-heavy sites. You can run the core library (static HTML scraping) on minimal resources, but dynamic SPAs require a beefier server or cloud deployment. This is not unique to PyScrappy (all browser-based scrapers have this cost), but it is a real constraint.
- -No hosted option or managed servicePyScrappy is self-hosted only. There is no official managed API or cloud service, so you provision and maintain your own infrastructure. Commercial services like Diffbot or ScraperAPI handle proxy rotation, anti-bot challenges, and infrastructure scaling for you. With PyScrappy, you handle those yourself (or use a third-party proxy service, which the library supports). This tradeoff makes sense for cost-sensitive or privacy-focused teams, but adds operational overhead.
- -Some built-in scrapers need proxies for blocked sitesA few built-in scrapers (Instagram, Twitter, Spotify, eBay) are marked as needing a proxy because those sites aggressively block automated requests. PyScrappy supports proxy configuration and scraping-API services (ScraperAPI, ScrapeOps, ScrapingBee) to route through, but you pay for those separately. The README is honest about which scrapers work without a proxy and which do not.
PyScrappy vs alternatives#
PyScrappy vs Diffbot
Both tools convert web pages to structured data, but they serve different deployment models and use cases. Diffbot is a managed API with enterprise pricing and advanced entity extraction. PyScrappy is open source with a self-hosted path and LLM-optimized output.
| Feature | PyScrappy | Diffbot |
|---|---|---|
| License | MIT | Proprietary |
| Self-hosting | Yes | No |
| JavaScript rendering | Yes (Playwright) | Yes |
| Output formats | Markdown, JSON, DataFrame | JSON (entity-structured) |
| Pricing | Free self-hosted | $299+/month managed API |
| Entity extraction | Manual (via LLM or NLP) | Built-in (people, companies, products) |
PyScrappy is the better choice when you need self-hosting for data privacy, cost control on high-volume scraping (thousands of pages per month), or the ability to modify the extraction pipeline. The markdown and JSON output formats are optimized for LLM ingestion with minimal preprocessing. Diffbot is worth considering when you need its structured entity extraction (people, articles, products) out of the box with no infrastructure management. Diffbot's entity recognition is more advanced than PyScrappy's text extraction, but you pay per API call and cannot self-host.
PyScrappy vs ScraperAPI
ScraperAPI is a managed proxy service that handles anti-bot challenges and returns raw HTML. PyScrappy is a self-hosted toolkit that scrapes and structures data, with optional proxy support.
| Feature | PyScrappy | ScraperAPI |
|---|---|---|
| License | MIT | Proprietary |
| Self-hosting | Yes | No |
| Anti-bot handling | TLS impersonation + proxy support | Managed proxy rotation |
| Output | Structured (markdown, JSON) | Raw HTML |
| Pricing | Free self-hosted | $49-$249+/month (usage tiers) |
| JavaScript rendering | Yes (Playwright) | Yes |
PyScrappy is the better choice when you want structured output (not raw HTML) and the ability to run scrapers on your infrastructure without per-request billing. ScraperAPI solves the proxy and anti-bot problem for you, but you still need to parse the returned HTML yourself. With PyScrappy, you handle anti-bot challenges via TLS impersonation or your own proxy (or route through ScraperAPI if you want), but you get clean markdown or JSON back instead of raw markup. For teams building AI pipelines, PyScrappy's LLM-ready output eliminates the HTML-to-text preprocessing step.
PyScrappy vs BeautifulSoup or Scrapy
BeautifulSoup and Scrapy are established Python scraping libraries. PyScrappy builds on similar patterns but adds adaptive selectors, LLM-optimized output, and MCP server support.
| Feature | PyScrappy | BeautifulSoup / Scrapy |
|---|---|---|
| Adaptive selectors | Yes (self-healing) | No (hard-coded CSS/XPath) |
| LLM-ready output | Yes (markdown, JSON) | Manual (HTML parsing) |
| Built-in scrapers | 24 (Wikipedia, GitHub, stocks, etc.) | None (you build all scrapers) |
| MCP server for AI agents | Yes | No |
| TLS-fingerprint stealth | Yes (optional) | No (requires separate proxy) |
PyScrappy is the better choice when you are building AI workflows that need LLM-ready data, when you want pre-built scrapers for common platforms, or when you need adaptive selectors that survive site redesigns. BeautifulSoup and Scrapy are more mature and widely used, but they return raw parsed HTML and require you to write custom cleaning and structuring logic. For traditional scraping workflows where you want full control, Scrapy is still the standard. For AI-native workflows, PyScrappy's markdown output and MCP server integration are purpose-built advantages.
Install and self-host#
PyScrappy requires Python 3.9 or newer. The core library installs via pip with no external dependencies. Optional extras add browser support (Playwright), MCP server capabilities (Python 3.10+ required), stealth mode (TLS impersonation), and DataFrame output.
```bash
# Core library (static HTML scraping, no browser)
pip install pyscrappy
# With browser support for JavaScript-rendered pages
pip install 'pyscrappy[browser]'
playwright install chromium
# With MCP server for AI agents (Python 3.10+)
pip install 'pyscrappy[mcp]'
# With stealth mode (TLS-fingerprint impersonation)
pip install 'pyscrappy[stealth]'
# Everything
pip install 'pyscrappy[all]'
```
For self-hosting in production, run the core library on any Python 3.9+ environment. If you need JavaScript rendering, provision at least 2GB RAM per concurrent worker for headless Chromium. Deploy the MCP server with pyscrappy-mcp (stdio by default) or pyscrappy-mcp --http for remote HTTP deployments. Register with Claude Code using claude mcp add pyscrappy pyscrappy-mcp. For Claude Desktop, add the server to your claude_desktop_config.json under mcpServers.What it's built on#
- Languages
- Python
FAQ#
Is PyScrappy free to use?
Yes. PyScrappy is MIT licensed and free to run on your own infrastructure. There is no managed cloud API or paid tier. The entire toolkit, including the MCP server and all 24 built-in scrapers, is open source. You pay for your own server or cloud compute, but there are no per-request fees or usage limits beyond what your infrastructure can handle.
Can I use PyScrappy with AI agents like Claude or Cursor?
Yes. PyScrappy ships an optional MCP (Model Context Protocol) server that exposes all scrapers as tools for AI agents. Install with pip install 'pyscrappy[mcp]' (requires Python 3.10+), run pyscrappy-mcp or python -m pyscrappy.mcp, and register it with your agent's MCP config. For Claude Desktop, add the server to claude_desktop_config.json. For Claude Code, run claude mcp add pyscrappy pyscrappy-mcp. The agent can then call scrape_url, scrape_wikipedia, scrape_stock, and 20+ other tools naturally.
How does PyScrappy handle sites that block automated requests?
PyScrappy has three anti-bot strategies. First, TLS-fingerprint impersonation (stealth mode): set impersonate="chrome" to mimic a real browser's TLS/JA3 fingerprint and bypass detection at the TLS handshake layer, before the site serves a CAPTCHA. Requires the optional stealth extra (pip install 'pyscrappy[stealth]'). Second, proxy support: route requests through a proxy or rotating proxy list. Third, scraping-API integration: configure ScraperAPI, ScrapeOps, or ScrapingBee to handle proxies and anti-bot challenges for you. Some built-in scrapers (Instagram, Twitter, Spotify) are marked as needing a proxy because those sites block aggressively.
What is the difference between PyScrappy and Diffbot or ScraperAPI?
Diffbot and ScraperAPI are managed cloud APIs with per-page billing ($299+/month for production workloads). PyScrappy is self-hosted and MIT licensed, so you run it on your infrastructure with zero per-request costs after setup. Diffbot offers advanced entity extraction (people, companies, products) from unstructured text; PyScrappy focuses on clean text/JSON/markdown extraction and leaves entity recognition to your LLM or separate NLP pipeline. ScraperAPI is a proxy service that handles anti-bot challenges; PyScrappy includes TLS impersonation and proxy support, but you manage the infrastructure. PyScrappy is the better choice when you need cost control, data privacy, or the ability to modify the extraction pipeline.
Do I need to run a headless browser for PyScrappy to work?
No, not for most sites. The core library scrapes static HTML with standard HTTP clients and no browser required. You only need the optional Playwright backend (pip install 'pyscrappy[browser]' and playwright install chromium) if you are scraping JavaScript-rendered single-page apps (React, Vue, Angular). Sites that serve full HTML on the initial request work fine without a browser. The README's GenericScraper supports both modes: it auto-detects when JavaScript rendering is needed, or you can force it with render_js=True.
Similar open-source tools#
deer-flow
Build super agents with DeerFlow's powerful framework
agency-agents
Expert AI agent personalities for every workflow
t3code
Control your coding agents from one interface
weathernext
AI-powered weather forecasting from Google DeepMind
open-seo
Open source alternative to Semrush and Ahrefs
transformers
Model-definition framework for state-of-the-art ML