README

Effortless Web Scraping for the Modern Web



    
    
    
    
    
    
    
    
    
    
    
    
    
    
        
    
        
    
        
    
    
        
    
        
    
    
      
    
    
      
    
    
    
        



    Selection methods
    ·
    Fetchers
    ·
    Spiders
    ·
    Proxy Rotation
    ·
    CLI
    ·
    MCP


Scrapling is a Python Web Scraping framework that brings fast HTTP requests, stealth browsers, powerful parsing, and concurrent crawling into one library.

Scrape static pages, render JavaScript, interact with websites, and extract data from the API calls behind them. Build crawlers with reusable sessions, proxy rotation, automatic retries, pause/resume, and built-in data exports.

Use it from Python, the terminal, or your AI agent through its MCP server and Agent Skill. From extracting a few fields to collecting a whole website as clean Markdown, Scrapling covers the full scraping workflow.

 What you can do with Scrapling

- Fetch & interact: Fast HTTP requests, stealth browsers, JavaScript rendering, Cloudflare Turnstile/Interstitial handling, persistent sessions, and background API capture.
- Parse & extract: CSS, XPath, text and regex search, adaptive element tracking, similar-element matching, and clean Markdown output.
- Crawl & export: Concurrent spiders, proxy rotation, retries, AutoThrottle, pause/resume, streaming, ready-made templates, and JSON/JSONL/CSV/XML exports.
- Scrape with AI or from your terminal: An MCP server for AI agents to browse and extract data, an Agent Skill, CLI commands, and an interactive shell.

Start with a single page:

python
from scrapling.fetchers import Fetcher

page = Fetcher.get('https://quotes.toscrape.com/')
quotes = page.css('.quote .text::text').getall()
print(quotes)


Or scale up with a spider:

python
from scrapling.spiders import Spider, Response

class QuotesSpider(Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]

    async def parse(self, response: Response):
        for quote in response.css('.quote'):
            yield {
                "text": quote.css('.text::text').get(),
                "author": quote.css('.author::text').get(),
            }

result = QuotesSpider().start()
print(len(result.items))
result.items.to_json("quotes.json")


This prints 10 and saves the quotes and authors to quotes.json.


    
        
    


 Platinum Sponsors

  
    
      
        
      
    
    
    NodeMaven - The most efficient proxy provider for Web Scraping and Automation with the Highest Quality IP on the market. SCRAPLING2X - 2X traffic for Black Friday
    
  
  
    
      
        
      
    
     Scrapling handles Cloudflare Turnstile. For enterprise-grade protection, 
        Hyper Solutions
       provides API endpoints that generate valid antibot tokens for Akamai, DataDome, Kasada, and Incapsula. Simple API calls, no browser automation required. 
  
  
    
      
    

Top issues

Discussions

Latest release

Small version bump, big browser upgrade! This version lets AI agents follow motion and control drag and drop, adds dedicated remote browser sessions, and brings fixes across parsing, spiders, proxies, and the shell 🚀

With this update, we extend the limits of current AI models 🔥 

 [!NOTE]
 Follow us on X for daily tips and tricks

 [!WARNING]
 This release introduces breaking changes to remote browser connections. Connecting to remote browsers now requires a new browser session class. Check the remote-session migration guide before updating.

 🚀 New Stuff and quality of life changes