Third-party toolUpdated 2026.11.16

Crawl4AI Setup Guide: Installation, Examples and Fixes

Crawl4AI installation and getting started: client setup or a local workflow, three practical examples, tested findings, common errors, costs and alternatives.

Aaron recommends Open official GitHub

Crawl4AI is a browser-based Python web extraction toolkit. It can turn rendered pages into Markdown and structured data for downstream processing. Start with one authorized page and an explicit content selector. A successful HTTP status does not establish that you extracted the content you wanted.

Install and choose a working environment

Tool version 0.9.5. Documentation checked 2026-11-16. The upstream source snapshot and installed package are pinned separately; the snapshot is not claimed to be the package’s release commit.

Create a project virtual environment first (python -m venv .venv), activate it using your shell’s normal procedure, and run the install command in that environment. The recorded Python tests used Linux/Python 3.12; other operating systems were not locally validated.

python -m pip install crawl4ai==0.9.5
crawl4ai-setup
crawl4ai-doctor

This package is a CLI/Python library, not an MCP server configured by the install command above. Run the examples in the project’s environment and give the resulting files to your assistant.

  • Claude Code: open the project, ask it to inspect the saved script/output, and confirm the interpreter path before execution.
  • Codex: open the same workspace, provide the script/output filename, and ask for a review or a locally permitted run.
  • Cursor: choose the project’s Python interpreter, run the command in its terminal, then attach the resulting file in chat.

These are file/terminal workflows, not a claim of a vendor-specific integration. The three client interfaces were not tested locally.

Three practical uses

1. Extract one rendered page

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
async def main():
    async with AsyncWebCrawler() as crawler:
        r = await crawler.arun("https://example.com/", config=CrawlerRunConfig(css_selector="main"))
        print(r.markdown.raw_markdown)
asyncio.run(main())

Expected: Expect content from main; verify that the site actually has that selector.

2. Reject empty extraction

After extraction, reject whitespace-only Markdown and require one known page heading. If either check fails, record the URL, status and selector instead of summarizing.

Expected: a response that follows the stated scope and exposes the requested source or tool result. Verify those references before acting.

3. Plan a bounded documentation crawl

Plan extraction for the three public documentation URLs I provide. Keep source URLs, exclude navigation and set rate limits. Do not discover extra URLs or run the plan yet.

Expected: a response that follows the stated scope and exposes the requested source or tool result. Verify those references before acting.

What our tests established

The 0.9.5 run compared a whole page, main, and a missing selector on the same local page. main retained five rows without navigation. The missing selector returned 200/success with effectively blank Markdown. Browser dependency failures and a shutdown warning remain recorded; broad public-site crawling was not tested.

Tests were run by Codex on synthetic data; no claim of the owner personally testing or reviewing this guide. Examples added above are documentation-based unless included in that recorded scope.

Common errors and fixes

Chromium does not start

Run setup/doctor and install browser system dependencies before increasing concurrency.

HTTP 200 with empty Markdown

Verify css_selector against the rendered page. HTTP success and content selection are separate checks.

Navigation contaminates extracted text

Select the article/main container and verify a known heading before using the output.

Limits, costs and license

Apache-2.0 with the project’s attribution requirements. Self-hosting needs browser/compute resources; optional models and hosted services can add fees. It does not guarantee access to blocked sites.

When to choose an alternative

FAQ

How does Crawl4AI differ from Firecrawl?

Compare self-hosted browser maintenance with a hosted extraction API. This page contains no speed or quality benchmark between them.

Can I run Crawl4AI in Docker?

Yes, the project documents containers. The recorded isolated test is not a production-service deployment test.

Does installing Crawl4AI configure MCP?

No. This guide uses Python; an MCP wrapper requires separate setup and testing.

Official sources

If page state needs to be checked through interaction, see Playwright MCP's browser workflow. For static article text, compare Trafilatura's extraction output.

Subscribe to AI tool picks and practical advice

Get worthwhile tool picks, test findings and important updates in your inbox.

Get a monthly-updated list of AI tools and Skills right away. We email when there are worthwhile tool picks, test findings or important updates. Unsubscribe anytime. Privacy

Almost done: check your inboxClick the button in the confirmation email and you will get the download link for the tool list. If you do not see it, check your spam or Promotions folder.

Or follow us on: