PHPackages kreuzberg-dev/kreuzcrawl - PHPackages - PHPackages [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register) 1. [Directory](/) 2. / 3. [Utility & Helpers](/categories/utility) 4. / 5. kreuzberg-dev/kreuzcrawl ActivePhp-ext[Utility & Helpers](/categories/utility) kreuzberg-dev/kreuzcrawl ======================== High-performance web crawling engine v1.0.1(3w ago)1183915[1 issues](https://github.com/xberg-io/crawlberg/issues)[1 PRs](https://github.com/xberg-io/crawlberg/pulls)MITRustPHP >=8.2CI failing Since Apr 14Pushed 4w agoCompare [ Source](https://github.com/xberg-io/crawlberg)[ Packagist](https://packagist.org/packages/kreuzberg-dev/kreuzcrawl)[ RSS](/packages/kreuzberg-dev-kreuzcrawl/feed)WikiDiscussions main Synced 3w ago READMEChangelog (10)Dependencies (6)Versions (114)Used By (0) ![Xberg](https://camo.githubusercontent.com/634e91576caf8f36ad27ce2180030b6dcd54a71dc87f7be493855aebfc430d5b/68747470733a2f2f63646e2e6a7364656c6976722e6e65742f67682f78626572672d696f2f6173736574734076312f62616e6e65722f726561646d652d62616e6e65722d6c696768742e737667) Crawlberg ========= [](#crawlberg) [ ![Bindings](https://camo.githubusercontent.com/23b2c7873e51d39aafa797a5571f07783c505f083f976594e38024d10eb4e433/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f42696e64696e67732d616c65662532302544372539302d303037656336) ](https://github.com/xberg-io/alef) [ ![Rust](https://camo.githubusercontent.com/0d1e454c35900c60d4aa0d50f87a3127dde44a307a0088bc598f2d3dbebd7788/68747470733a2f2f696d672e736869656c64732e696f2f6372617465732f762f637261776c626572673f6c6162656c3d5275737426636f6c6f723d303037656336) ](https://crates.io/crates/crawlberg) [ ![Python](https://camo.githubusercontent.com/a11d2460cdf62e370cb49b7320809bb4fdcf81782fd265c83b2e80e65a1a4717/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f637261776c626572673f6c6162656c3d507974686f6e26636f6c6f723d303037656336) ](https://pypi.org/project/crawlberg/) [ ![Node.js](https://camo.githubusercontent.com/3e6708f38d6bf306e6ee79cc34da2de39afb085b81e8a29eea7a69a911d77bf4/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f762f4078626572672d696f2f637261776c626572673f6c6162656c3d4e6f64652e6a7326636f6c6f723d303037656336) ](https://www.npmjs.com/package/@xberg-io/crawlberg) [ ![WASM](https://camo.githubusercontent.com/8d685d57d11682a6b0494bdf288edc8036b64ae4745344ca94ed25a654d64003/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f762f4078626572672d696f2f637261776c626572672d7761736d3f6c6162656c3d5741534d26636f6c6f723d303037656336) ](https://www.npmjs.com/package/@xberg-io/crawlberg-wasm) [ ![Java](https://camo.githubusercontent.com/541cac0e1d38d4be2e3ff3680d1ed1dedb635b5aeb1944fa1771edaaf1b6b1e6/68747470733a2f2f696d672e736869656c64732e696f2f6d6176656e2d63656e7472616c2f762f696f2e78626572672e637261776c626572672f637261776c626572673f6c6162656c3d4a61766126636f6c6f723d303037656336) ](https://central.sonatype.com/artifact/io.xberg.crawlberg/crawlberg) [ ![Go](https://camo.githubusercontent.com/0d6e3519d3903bce392eca6d19831499f00878743bf706b8be30af689cf71320/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f7461672f78626572672d696f2f637261776c626572673f6c6162656c3d476f26636f6c6f723d303037656336) ](https://pkg.go.dev/github.com/xberg-io/crawlberg/packages/go) [ ![C#](https://camo.githubusercontent.com/3d816ea9514db1079f23e8d743bb46b73228284f251aa3a64e763790e443e16a/68747470733a2f2f696d672e736869656c64732e696f2f6e756765742f762f5862657267496f2e437261776c626572673f6c6162656c3d4325323326636f6c6f723d303037656336) ](https://www.nuget.org/packages/XbergIo.Crawlberg/) [ ![PHP](https://camo.githubusercontent.com/2598679aada4a20f1fd9c635cd67e28f3aa7ab0e1c95c92343915eb845fd3d06/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f762f78626572672d696f2f637261776c626572673f6c6162656c3d50485026636f6c6f723d303037656336) ](https://packagist.org/packages/xberg-io/crawlberg) [ ![Ruby](https://camo.githubusercontent.com/715e462c070a6424cb04b07c5342bcfce938f145212a4fb168a3064c8a84e8c8/68747470733a2f2f696d672e736869656c64732e696f2f67656d2f762f637261776c626572673f6c6162656c3d5275627926636f6c6f723d303037656336) ](https://rubygems.org/gems/crawlberg) [ ![Elixir](https://camo.githubusercontent.com/98f61d6b9f4131649588e07c0f608ded08c6ba81a912e323114de0bccf4f83ff/68747470733a2f2f696d672e736869656c64732e696f2f686578706d2f762f637261776c626572673f6c6162656c3d456c6978697226636f6c6f723d303037656336) ](https://hex.pm/packages/crawlberg) [ ![Dart](https://camo.githubusercontent.com/e17264e52e89c5e2870e0e4e3deb25e5d9bb624ee10aee1314fa36dcd5d84847/68747470733a2f2f696d672e736869656c64732e696f2f7075622f762f637261776c626572673f6c6162656c3d4461727426636f6c6f723d303037656336) ](https://pub.dev/packages/crawlberg) [ ![Kotlin](https://camo.githubusercontent.com/6443bf0ead7288f09188877348af97f3da7c50869631e41c681741bf55808c38/68747470733a2f2f696d672e736869656c64732e696f2f6d6176656e2d63656e7472616c2f762f696f2e78626572672e637261776c626572672e616e64726f69642f637261776c626572672d616e64726f69643f6c6162656c3d4b6f746c696e26636f6c6f723d303037656336) ](https://central.sonatype.com/artifact/io.xberg.crawlberg.android/crawlberg-android) [ ![Swift](https://camo.githubusercontent.com/b8e3c9990c66f8b878644a15e0ffd0fba03df291119bca044b94033c83d27246/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f53776966742d53504d2d303037656336) ](https://github.com/xberg-io/crawlberg/tree/main/packages/swift) [ ![Zig](https://camo.githubusercontent.com/376558d850c9af32b9d23b0211489a0b5c979af703256fa0b7c3e917bd591c53/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f5a69672d7061636b6167652d303037656336) ](https://github.com/xberg-io/crawlberg/tree/main/packages/zig) [ ![C FFI](https://camo.githubusercontent.com/41b9c58c3810775402965a3a7652da9832ea83391c1ff255081e5551900db8f2/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f432d4646492d303037656336) ](https://github.com/xberg-io/crawlberg/releases) [ ![Docker](https://camo.githubusercontent.com/7d83f05278efa79f40cf820ee19ebf22bd638c94cb23b09bb354e7d1e3282496/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f446f636b65722d676863722e696f2d3030376563363f6c6f676f3d646f636b6572266c6f676f436f6c6f723d7768697465) ](https://github.com/xberg-io/crawlberg/pkgs/container/crawlberg) [ ![License](https://camo.githubusercontent.com/0cd4d42d83d2124c29737dd1519425c87c4b465016ef0cee20cbcb8ef420c0e0/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4c6963656e73652d4d49542d303037656336) ](https://github.com/xberg-io/crawlberg/blob/main/LICENSE) [ ![Documentation](https://camo.githubusercontent.com/c02375f24f32e0fa113e890e618832065f4ee355c3a159526c99fc4659f54d47/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f446f63732d637261776c626572672d303037656336) ](https://docs.crawlberg.xberg.io) [ ![Join Discord](https://camo.githubusercontent.com/c3d59355bb5f7fc8224936ba897a06b87e5ba6f52bd08dfd2f7f78bf669c550d/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f446973636f72642d436861742d3030376563363f6c6f676f3d646973636f7264266c6f676f436f6c6f723d7768697465) ](https://discord.gg/xt9WY3GnKR) High-performance Rust web crawling engine for structured data extraction. Scrape, crawl, and map websites with native bindings for 14 languages — same engine, identical results across every runtime. What and Why? ------------- [](#what-and-why) Crawlberg is the crawling **substrate**: everything you need to scrape and crawl a site end-to-end from a single Rust core — HTML→Markdown, headless-Chrome fallback, robots/sitemap parsing, per-domain throttling, and an SSRF-safe policy — with identical results across 14 language bindings. Productization concerns (managed proxy pools, tuned WAF fingerprints, authenticated-session injection, scheduling, billing) live in [xberg-enterprise](https://github.com/xberg-io/xberg-enterprise), the reference operational implementation. Every extension point (`Frontier`, `RateLimiter`, `CrawlStore`, `EventEmitter`, `ContentFilter`, `WafClassifier`, …) is a trait you inject via `CrawlEngineBuilder::with_(...)`. ### Features [](#features) FeatureDescription**Structured extraction**Text, metadata, links, images, assets, JSON-LD, Open Graph, hreflang, favicons, headings, response headers**Markdown conversion**Clean Markdown with citations, document structure, and fit-content mode**Concurrent crawling**Depth-first, breadth-first, or best-first traversal with configurable depth, page limits, and concurrency**14 language bindings**Rust, Python, Node.js, Ruby, Go, Java, Kotlin (Android), C#, PHP, Elixir, Dart, Swift, Zig, and WebAssembly**Smart filtering**BM25 relevance scoring, URL include/exclude patterns, robots.txt compliance, sitemap discovery**Browser rendering**Optional headless browser for JavaScript-heavy SPAs with WAF detection and bypass**Batch & streaming**Scrape or crawl hundreds of URLs concurrently; real-time crawl events via async streams**SSRF-safe by default**Refuses loopback, private, link-local, and cloud-metadata addresses; opt out via env var or `CrawlConfig`**Auth & rate limiting**HTTP Basic, Bearer, and custom-header auth with cookie jars; per-domain request throttling**MCP server & REST API**Model Context Protocol integration for AI agents plus an HTTP server with OpenAPI spec### Supported Platforms [](#supported-platforms) Precompiled binaries for Linux (x86\_64/aarch64), macOS (ARM64), and Windows (x64) across every binding. See the [platform support reference](https://docs.crawlberg.xberg.io) for the full matrix. [ ![Star Crawlberg on GitHub](docs/assets/star.gif) ](https://github.com/xberg-io/crawlberg/stargazers) **⭐ Star this repo to show your support — it helps others discover Crawlberg.** Quick Start ----------- [](#quick-start) ### Language Packages [](#language-packages) **Python**``` pip install crawlberg ``` See [Python README](https://github.com/xberg-io/crawlberg/tree/main/packages/python) for full documentation. **Node.js**``` npm install @xberg-io/crawlberg ``` See [Node.js README](https://github.com/xberg-io/crawlberg/tree/main/crates/crawlberg-node) for full documentation. **Rust**``` cargo add crawlberg ``` See [Rust README](https://github.com/xberg-io/crawlberg/tree/main/crates/crawlberg) for full documentation. **Go**``` go get github.com/xberg-io/crawlberg/packages/go ``` See [Go README](https://github.com/xberg-io/crawlberg/tree/main/packages/go) for full documentation. **Java**Available on Maven Central as `io.xberg.crawlberg:crawlberg`. See [Java README](https://github.com/xberg-io/crawlberg/tree/main/packages/java) for the dependency snippet and current version. **C#**``` dotnet add package Crawlberg ``` See [C# README](https://github.com/xberg-io/crawlberg/tree/main/packages/csharp) for full documentation. **Ruby**``` gem install crawlberg ``` See [Ruby README](https://github.com/xberg-io/crawlberg/tree/main/packages/ruby) for full documentation. **PHP**``` composer require xberg-io/crawlberg ``` See [PHP README](https://github.com/xberg-io/crawlberg/tree/main/packages/php) for full documentation. **Elixir**Add `{:crawlberg, "~> 0.3"}` to your `mix.exs` dependencies. See [Elixir README](https://github.com/xberg-io/crawlberg/tree/main/packages/elixir) for full documentation. **Dart / Flutter**``` dart pub add crawlberg ``` See [Dart README](https://github.com/xberg-io/crawlberg/tree/main/packages/dart) for full documentation. **Kotlin (Android)**Available on Maven Central as `io.xberg.crawlberg.android:crawlberg-android`. See [Kotlin README](https://github.com/xberg-io/crawlberg/tree/main/packages/kotlin-android) for the dependency snippet and current version. **Swift**Add via Swift Package Manager. See [Swift README](https://github.com/xberg-io/crawlberg/tree/main/packages/swift) for full documentation. **Zig**See [Zig README](https://github.com/xberg-io/crawlberg/tree/main/packages/zig) for installation and usage. **WebAssembly**``` npm install @xberg-io/crawlberg-wasm ``` See [WebAssembly README](https://github.com/xberg-io/crawlberg/tree/main/crates/crawlberg-wasm) for full documentation. **C/C++ (FFI)**C header + shared library from [GitHub Releases](https://github.com/xberg-io/crawlberg/releases). See [FFI crate](https://github.com/xberg-io/crawlberg/tree/main/crates/crawlberg-ffi) for full documentation. **CLI**``` cargo install crawlberg-cli ``` ``` brew install xberg-io/tap/crawlberg ``` See [CLI README](https://github.com/xberg-io/crawlberg/tree/main/crates/crawlberg-cli) for full documentation. ### AI Coding Assistants [](#ai-coding-assistants) Install the Crawlberg plugin from the [`xberg-io/plugins`](https://github.com/xberg-io/plugins) marketplace. It ships the Crawlberg agent skills (site crawling, HTML→Markdown scraping, headless-Chrome fallback) plus the `crawlberg` MCP server, and works with every major coding agent — expand your harness below. **Claude Code**``` /plugin marketplace add xberg-io/plugins /plugin install crawlberg@xberg ``` **Codex CLI**``` /plugins add https://github.com/xberg-io/plugins ``` Then search for `crawlberg` and select **Install Plugin**. **Cursor**Settings → Plugins → Add from URL → `https://github.com/xberg-io/plugins`, then select **crawlberg**. **Gemini CLI**``` gemini extensions install https://github.com/xberg-io/plugins ``` **Factory Droid**``` droid plugin marketplace add https://github.com/xberg-io/plugins droid plugin install crawlberg@xberg ``` **GitHub Copilot CLI**``` copilot plugin marketplace add https://github.com/xberg-io/plugins copilot plugin install crawlberg@xberg ``` **opencode**Add the package to `opencode.json`: ``` { "$schema": "https://opencode.ai/config.json", "plugin": ["@xberg-io/opencode-crawlberg"] } ``` Documentation ------------- [](#documentation) Full guides, per-language API references, the substrate/operational model, antibot strategy, and observability live at **[docs.crawlberg.xberg.io](https://docs.crawlberg.xberg.io)**. Contributing ------------ [](#contributing) Contributions are welcome! See our [Contributing Guide](https://github.com/xberg-io/crawlberg/blob/main/CONTRIBUTING.md). Part of Xberg.dev ----------------- [](#part-of-xbergdev) - [Xberg](https://github.com/xberg-io/xberg) — document intelligence: text, tables, metadata from 91+ formats with optional OCR. - [Xberg Enterprise](https://github.com/xberg-io/xberg-enterprise) — managed extraction API with SDKs, dashboards, and observability. - [crawlberg](https://github.com/xberg-io/crawlberg) — web crawling and scraping with HTML→Markdown and headless-Chrome fallback. - [html-to-markdown](https://github.com/xberg-io/html-to-markdown) — fast, lossless HTML→Markdown engine. - [liter-llm](https://github.com/xberg-io/liter-llm) — universal LLM API client with native bindings for 14 languages and 143 providers. - [tree-sitter-language-pack](https://github.com/xberg-io/tree-sitter-language-pack) — tree-sitter grammars and code-intelligence primitives. - [alef](https://github.com/xberg-io/alef) — the polyglot binding generator that produces every per-language binding across the 5 polyglot repos. License ------- [](#license) [MIT License](https://github.com/xberg-io/crawlberg/blob/main/LICENSE) Links ----- [](#links) - [Documentation](https://docs.crawlberg.xberg.io) - [GitHub Repository](https://github.com/xberg-io/crawlberg) - [Issue Tracker](https://github.com/xberg-io/crawlberg/issues) - [Changelog](CHANGELOG.md) - [Discord](https://discord.gg/xt9WY3GnKR) — community, roadmap, announcements. ### Health Score 53 — FairBetter than 96% of packages Maintenance94 Actively maintained with recent releases Popularity28 Limited adoption so far Community17 Small or concentrated contributor base Maturity63 Established project with proven stability Bus Factor1 Top contributor holds 94.2% of commits — single point of failure How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window. **Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores. **Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement. **Maturity (30%)** — Project age, version count, PHP version support, and release stability. ### Release Activity Cadence Every ~0 days Total 99 Last Release 26d ago Major Versions v0.3.0 → v1.0.0-rc.12026-06-26 PHP version history (2 changes)0.1.0-rc.1PHP ^8.4 v0.3.0-rc.28PHP >=8.2 ### Community Maintainers ![](https://avatars.githubusercontent.com/u/30733348?v=4)[Na'aman Hirschfeld](/maintainers/Goldziher)[@Goldziher](https://github.com/Goldziher) --- Top Contributors [![Goldziher](https://avatars.githubusercontent.com/u/30733348?v=4)](https://github.com/Goldziher "Goldziher (959 commits)")[![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?v=4)](https://github.com/github-actions[bot] "github-actions[bot] (29 commits)")[![kreuzberg-dev-publisher[bot]](https://avatars.githubusercontent.com/u/241328462?v=4)](https://github.com/kreuzberg-dev-publisher[bot] "kreuzberg-dev-publisher[bot] (19 commits)")[![v-tan](https://avatars.githubusercontent.com/u/22367932?v=4)](https://github.com/v-tan "v-tan (5 commits)")[![pratik-mahalle](https://avatars.githubusercontent.com/u/124587957?v=4)](https://github.com/pratik-mahalle "pratik-mahalle (3 commits)")[![kh3rld](https://avatars.githubusercontent.com/u/171191586?v=4)](https://github.com/kh3rld "kh3rld (1 commits)")[![Haoxincode](https://avatars.githubusercontent.com/u/48117929?v=4)](https://github.com/Haoxincode "Haoxincode (1 commits)")[![stchris](https://avatars.githubusercontent.com/u/217554?v=4)](https://github.com/stchris "stchris (1 commits)") --- Tags crawlingcsharpelixirffigolangjavamcpphppythonrubyrusttypescriptwasmweb-crawlerweb-scrapingwebspidercrawlscrape ### Code Quality TestsPHPUnit Static AnalysisPHPStan Code StylePHP CS Fixer Type Coverage Yes ### Embed Badge ![Health badge](/badges/kreuzberg-dev-kreuzcrawl/health.svg) ``` [![Health](https://phpackages.com/badges/kreuzberg-dev-kreuzcrawl/health.svg)](https://phpackages.com/packages/kreuzberg-dev-kreuzcrawl) ``` ### Alternatives [crwlr/crawler Web crawling and scraping library. 36917.4k2](/packages/crwlr-crawler)[jaeger/querylist Simple, elegant, extensible PHP Web Scraper (crawler/spider),Use the css3 dom selector,Based on phpQuery! 简洁、优雅、可扩展的PHP采集工具(爬虫)，基于phpQuery。 2.8k657.7k89](/packages/jaeger-querylist)[onokumus/metismenu A jQuery menu plugin 2.0k271.0k5](/packages/onokumus-metismenu)[vdb/php-spider A configurable and extensible PHP web spider 1.3k185.8k7](/packages/vdb-php-spider)[jaybizzle/laravel-crawler-detect A Laravel package to detect web crawlers via the user agent 3302.7M18](/packages/jaybizzle-laravel-crawler-detect)[jakubkulhan/chrome-devtools-protocol Chrome Devtools Protocol client for PHP 185972.3k3](/packages/jakubkulhan-chrome-devtools-protocol) PHPackages © 2026 [Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)