README

Crawl4AI

Crawl4AI Cloud is a hosted web scraping, extraction, and search API for AI agents and data pipelines. Turn pages into clean Markdown for RAG, extract structured JSON, or search the live web with one API key. Crawl4AI runs the browsers and manages the proxies.

Get a free API key · 1,000 free credits every month. No credit card required.

Try a URL in the playground · Set up Crawl4AI Cloud in Claude Code, Cursor, or Codex

This repository is home to the Apache-2.0 crawler that powers Crawl4AI Cloud. Run it yourself with Python or Docker.


  
    
    
  











 Cloud quickstart

Hosted API base URL: https://api.crawl4ai.com.

Cloud API reference · Agent setup and MCP

Set CRAWL4AI_KEY to your API key, then scrape a page as Markdown:

bash
curl -s https://api.crawl4ai.com/scrape \
  -H "Authorization: Bearer $CRAWL4AI_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://news.ycombinator.com"}' | jq -r .markdown



Python 3

python
import json
import os
from urllib.request import Request, urlopen

request = Request(
    "https://api.crawl4ai.com/scrape",
    data=json.dumps({"url": "https://news.ycombinator.com"}).encode("utf-8"),
    headers={
        "Authorization": f"Bearer {os.environ['CRAWL4AI_KEY']}",
        "Content-Type": "application/json",
    },
    method="POST",
)

with urlopen(request) as response:
    print(json.load(response)["markdown"])





Node.js (18+)

javascript
async function main() {
  const key = process.env.CRAWL4AI_KEY;
  if (!key) throw new Error("Set CRAWL4AI_KEY first.");

  const response = await fetch("https://api.crawl4ai.com/scrape", {
    method: "POST",
    headers: {
      Authorization: Bearer ${key},
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ url: "https://news.ycombinator.com" }),
  });

  if (!response.ok) {
    throw new Error(Scrape failed (HTTP ${response.status}): ${await response.text()});
  }

  const { markdown } = await response.json();
  console.log(markdown);
}

main().catch((error) = {
  console.error(error.message);
  process.exitCode = 1;
});




Example Markdown output (illustrative excerpt):

markdown
 Hacker News

1. Example story title
2. Another story title


The same key works for /extract, /search, /serp, /answer and many URLs at once (/scrape/batch, /scrape/jobs). Pay as you go: live prices.
For Claude Code, connect to the hosted MCP server:

bash
claude mcp add --transport http crawl4ai https://api.crawl4ai.com/mcp \
  --header "Authorization: Bearer $CRAWL4AI_KEY"


 Run Crawl4AI yourself

Latest: v0.9.5 (14 Oct 2026) · all releases →


bash
pip install -U crawl4ai
crawl4ai-setup         installs the browser, once


python
import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://news.ycombinator.com")
        print(result.markdown)

asyncio.run(main())


Docker server, CLI and every option: Installation · Self-ho

Top issues

Discussions

Latest release

🎉 Crawl4AI v0.9.5 Released!

 📦 Installation

PyPI:
bash
pip install crawl4ai==0.9.5


Docker:
bash
docker pull unclecode/crawl4ai:0.9.5
docker pull unclecode/crawl4ai:latest


Note: Docker images are being built and will be available shortly.
Check the Docker Release workflow for build status.

 📝 What's Changed
See CHANGELOG.md for details.