Reliable, large-scale web extraction, now built to be drastically more cost-effective than any of the existing solutions.
👉 Apply here for early access
We’ll be onboarding in phases and working closely with early users.
Limited slots.
Crawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines. Fast, controllable, battle tested by a 50k+ star community.
✨ Check out latest update v0.9.2
✨ New in v0.9.2: Maintenance patch release. Fixes a MemoryAdaptiveDispatcher
task/page leak when a streaming crawl is closed, Docker Playground
"Advanced Config" and Monitor WebSocket auth, Playwright headless-shell
packaging, and GPU (ENABLE_GPU=true) Docker builds. Release notes →
✨ Recent v0.9.0: Major secure-by-default release of the Docker API server. Auth is on by default, the server binds loopback unless given a token, and the request body is now an untrusted trust boundary. Release notes →
✨ Recent v0.8.7: Security-hardening release. Fixes critical Docker API vulnerabilities (RCE, SSRF, auth bypass, file write, XSS, hardcoded JWT secret), adds DomainMapper, and ships scraping, deep-crawl, and LLM fixes. Release notes →
✨ Previous v0.8.0: Crash Recovery & Prefetch Mode! Deep crawl crash recovery with resume_state and on_state_change callbacks for long-running crawls. New prefetch=True mode for 5-10x faster URL discovery. Release notes →
✨ Previous v0.7.8: Stability & Bug Fix Release! 11 bug fixes addressing Docker API issues, LLM extraction improvements, URL handling fixes, and dependency updates. Release notes →
🤓 My Personal Story
Why developers pick Crawl4AI
- Install Crawl4AI:
# Install the package
pip install -U crawl4ai
# For pre release versions
pip install crawl4ai --pre
# Run post-installation setup
crawl4ai-setup
# Verify your installation
crawl4ai-doctorIf you encounter any browser-related issues, you can install them manually:
python -m playwright install --with-deps chromium- Run a simple web crawl with Python:
import asyncio
from crawl4ai import *
async def main():
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
url="https://www.nbcnews.com/business",
)
print(result.markdown)
if __name__ == "__main__":
asyncio.run(main())- Or use the new command-line interface:
# Basic crawl with markdown output
crwl https://www.nbcnews.com/business -o markdown
# Deep crawl with BFS strategy, max 10 pages
crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10
# Use LLM extraction with a specific question
crwl https://www.example.com/products -q "Extract all product prices"🎉 Sponsorship Program Now Open! After powering 51K+ developers and 1 year of growth, Crawl4AI is launching dedicated support for startups and enterprises. Be among the first 50 Founding Sponsors for permanent recognition in our Hall of Fame.
Crawl4AI is the #1 trending open-source web crawler on GitHub. Your support keeps it independent, innovative, and free for the community — while giving you direct access to premium benefits.
- 🌱 Believer ($5/mo) — Join the movement for data democratization
- 🚀 Builder ($50/mo) — Priority support & early access to features
- 💼 Growing Team ($500/mo) — Bi-weekly syncs & optimization help
- 🏢 Data Infrastructure Partner ($2000/mo) — Full partnership with dedicated support
Custom arrangements available - see SPONSORS.md for details & contact
Why sponsor?
No rate-limited APIs. No lock-in. Build and own your data pipeline with direct guidance from the creator of Crawl4AI.
📝 Markdown Generation
📊 Structured Data Extraction
🌐 Browser Integration
🔎 Crawling & Scraping
🚀 Deployment
🎯 Additional Features
✨ Play around with this
✨ Visit our Documentation Website
Crawl4AI offers flexible installation options to suit various use cases. You can install it as a Python package or use Docker.
You can check the project structure in the directory docs/examples. Over there, you can find a variety of examples; here, some popular examples are shared.
📝 Heuristic Markdown Generation with Clean and Fit Markdown
🖥️ Executing JavaScript & Extract Structured Data without LLMs
📚 Extracting Structured Data with LLMs
🤖 Using Your own Browser with Custom User Profile
Version 0.9.2 Release Highlights - Maintenance Bug Fixes
A maintenance patch release with bug fixes across the dispatcher, Docker, and GPU builds. MemoryAdaptiveDispatcher
no longer leaks crawl tasks and browser pages when a streaming crawl is
closed. Docker fixes cover the Playground "Advanced Config" 400, the
Monitor WebSocket 500 under JWT auth, and Playwright headless-shell
packaging. ENABLE_GPU=true Docker builds no longer fail on the CUDA toolkit.
No new features, no breaking changes.
pip install -U crawl4aiVersion 0.8.6 - Security Hotfix: litellm Supply Chain Fix
Crawl4AI follows standard Python version numbering conventions (PEP 440) to help users understand the stability and features of each release.
🚨 Documentation Update Alert: We're undertaking a major documentation overhaul next week to reflect recent updates and improvements. Stay tuned for a more comprehensive and up-to-date guide!
For current documentation, including installation instructions, advanced features, and API reference, visit our Documentation Website.
To check our development plans and upcoming features, visit our Roadmap.
📈 Development TODOs
We welcome contributions from the open-source community. Check out our contribution guidelines for more information.
I'll help modify the license section with badges. For the halftone effect, here's a version with it:
Here's the updated license section:
This project is licensed under the Apache License 2.0, attribution is recommended via the badges below. See the Apache 2.0 License file for details.
When using Crawl4AI, you must include one of the following attribution methods:
📈 1. Badge Attribution (Recommended)
 |
|
 |
|
 |
|
 |
If you use Crawl4AI in your research or project, please cite:
@software{crawl4ai2024,
author = {UncleCode},
title = {Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper},
year = {2024},
publisher = {GitHub},
journal = {GitHub Repository},
howpublished = {\url{https://github.com/unclecode/crawl4ai}},
commit = {Please use the commit hash you're working with}
}Text citation format:
UncleCode. (2024). Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper [Computer software].
GitHub. https://github.com/unclecode/crawl4ai
For questions, suggestions, or feedback, feel free to reach out:
- GitHub: unclecode
- Twitter: @unclecode
- Website: crawl4ai.com
Happy Crawling! 🕸️🚀
Our mission is to unlock the value of personal and enterprise data by transforming digital footprints into structured, tradeable assets. Crawl4AI empowers individuals and organizations with open-source tools to extract and structure data, fostering a shared data economy.
We envision a future where AI is powered by real human knowledge, ensuring data creators directly benefit from their contributions. By democratizing data and enabling ethical sharing, we are laying the foundation for authentic AI advancement.
🔑 Key Opportunities
These companies provide core infrastructure and technology that power Crawl4AI’s capabilities — from web access and proxy networks to AI tooling and data pipelines.
| Company | About |
|---|---|
 |
Massive is a web access API backed by millions of volunteer devices in 195+ countries. AI agents, models, and data pipelines use it to reach any site on the internet, reliably, in real time, and at scale. |
No comments:
Post a Comment