Crawl4AI is a browser-based Python web extraction toolkit. It can turn rendered pages into Markdown and structured data for downstream processing. Start with one authorized page and an explicit content selector. A successful HTTP status does not establish that you extracted the content you wanted.
Install and choose a working environment
Tool version 0.9.5. Documentation checked 2026-11-16. The upstream source snapshot and installed package are pinned separately; the snapshot is not claimed to be the package’s release commit.
Create a project virtual environment first (python -m venv .venv), activate it using your shell’s normal procedure, and run the install command in that environment. The recorded Python tests used Linux/Python 3.12; other operating systems were not locally validated.
python -m pip install crawl4ai==0.9.5
crawl4ai-setup
crawl4ai-doctor
This package is a CLI/Python library, not an MCP server configured by the install command above. Run the examples in the project’s environment and give the resulting files to your assistant.
- Claude Code: open the project, ask it to inspect the saved script/output, and confirm the interpreter path before execution.
- Codex: open the same workspace, provide the script/output filename, and ask for a review or a locally permitted run.
- Cursor: choose the project’s Python interpreter, run the command in its terminal, then attach the resulting file in chat.
These are file/terminal workflows, not a claim of a vendor-specific integration. The three client interfaces were not tested locally.
Three practical uses
1. Extract one rendered page
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
async def main():
async with AsyncWebCrawler() as crawler:
r = await crawler.arun("https://example.com/", config=CrawlerRunConfig(css_selector="main"))
print(r.markdown.raw_markdown)
asyncio.run(main())
Expected: Expect content from main; verify that the site actually has that selector.
2. Reject empty extraction
After extraction, reject whitespace-only Markdown and require one known page heading. If either check fails, record the URL, status and selector instead of summarizing.
Expected: a response that follows the stated scope and exposes the requested source or tool result. Verify those references before acting.
3. Plan a bounded documentation crawl
Plan extraction for the three public documentation URLs I provide. Keep source URLs, exclude navigation and set rate limits. Do not discover extra URLs or run the plan yet.
Expected: a response that follows the stated scope and exposes the requested source or tool result. Verify those references before acting.
What our tests established
The 0.9.5 run compared a whole page, main, and a missing selector on the same local page. main retained five rows without navigation. The missing selector returned 200/success with effectively blank Markdown. Browser dependency failures and a shutdown warning remain recorded; broad public-site crawling was not tested.
Tests were run by Codex on synthetic data; no claim of the owner personally testing or reviewing this guide. Examples added above are documentation-based unless included in that recorded scope.
Common errors and fixes
Chromium does not start
Run setup/doctor and install browser system dependencies before increasing concurrency.
HTTP 200 with empty Markdown
Verify css_selector against the rendered page. HTTP success and content selection are separate checks.
Navigation contaminates extracted text
Select the article/main container and verify a known heading before using the output.
Limits, costs and license
Apache-2.0 with the project’s attribution requirements. Self-hosting needs browser/compute resources; optional models and hosted services can add fees. It does not guarantee access to blocked sites.
When to choose an alternative
-
When static HTML extraction is enough: Trafilatura.
-
When an agent needs interactive browser controls: Playwright MCP.
FAQ
How does Crawl4AI differ from Firecrawl?
Compare self-hosted browser maintenance with a hosted extraction API. This page contains no speed or quality benchmark between them.
Can I run Crawl4AI in Docker?
Yes, the project documents containers. The recorded isolated test is not a production-service deployment test.
Does installing Crawl4AI configure MCP?
No. This guide uses Python; an MCP wrapper requires separate setup and testing.
Official sources
If page state needs to be checked through interaction, see Playwright MCP's browser workflow. For static article text, compare Trafilatura's extraction output.