PHPackages                             datahelm/crawler - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Parsing & Serialization](/categories/parsing)
4. /
5. datahelm/crawler

ActiveLibrary[Parsing & Serialization](/categories/parsing)

datahelm/crawler
================

Generic Scrapy-like web crawler for Laravel — auto-detects lists, pagination, fields, and supports API/SPA, infinite scroll, image downloading, dedup, resumable crawls, and pluggable output sinks.

v1.0.5(1mo ago)14MITPHPPHP ^8.3

Since Jul 4Pushed 1mo agoCompare

[ Source](https://github.com/data-helm/crawler)[ Packagist](https://packagist.org/packages/datahelm/crawler)[ RSS](/packages/datahelm-crawler/feed)WikiDiscussions main Synced 1w ago

READMEChangelogDependencies (8)Versions (6)Used By (0)

DataHelm Crawler
================

[](#datahelm-crawler)

Scrapy-style web crawler for Laravel — auto-detects lists, pagination, and fields; supports API/SPA sites, infinite scroll, image downloading, dedup, pluggable output sinks, and **LLM-ready Markdown output** (like Firecrawl / Crawl4AI).

[![Documentation](https://camo.githubusercontent.com/5b5230f28253389c5e804a2ed603e3b8d28657fe499caec65d592911586aad44/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f646f63732d6461746168656c6d2e6465762d626c75653f7374796c653d666f722d7468652d6261646765266c6f676f3d72656164746865646f6373266c6f676f436f6c6f723d7768697465)](https://datahelm.dev/guide/introduction)

📖 **Documentation:**

 [![Architecture overview](https://camo.githubusercontent.com/2599e95aa269783c53aedcfd976c92f92f25544603ba9ae9c6147da0b4ce413c/68747470733a2f2f6461746168656c6d2e6465762f696d616765732f6172636869746563747572652d6f766572766965772e706e67)](https://camo.githubusercontent.com/2599e95aa269783c53aedcfd976c92f92f25544603ba9ae9c6147da0b4ce413c/68747470733a2f2f6461746168656c6d2e6465762f696d616765732f6172636869746563747572652d6f766572766965772e706e67)

Installation
------------

[](#installation)

```
composer require datahelm/crawler
```

Publish config (optional):

```
php artisan vendor:publish --tag=crawler-config
```

Quick start
-----------

[](#quick-start)

```
php artisan datahelm:scrap:generate "https://example.com/listing" --get-detail=true --robot
php artisan datahelm:robot:example --limit=10
```

`datahelm:scrap:generate` auto-detects the repeating item block, pagination, and fields on a listing page like this:

 [![Example listing page auto-detection](https://camo.githubusercontent.com/e4bc6eb137e3568cae276a6073212ff79d3e6c2b6bba76afcf409979b15cd189/68747470733a2f2f6461746168656c6d2e6465762f696d616765732f6578616d706c652d6c697374696e672d706167652e706e67)](https://camo.githubusercontent.com/e4bc6eb137e3568cae276a6073212ff79d3e6c2b6bba76afcf409979b15cd189/68747470733a2f2f6461746168656c6d2e6465762f696d616765732f6578616d706c652d6c697374696e672d706167652e706e67)

Presets (field-detection heuristics)
------------------------------------

[](#presets-field-detection-heuristics)

A preset is a named bundle of heuristics used **only** by `datahelm:scrap:generate` — the auto-detection step that guesses selectors on a site it has never seen. It has no effect once a blueprint is saved; it only shapes what gets written into that blueprint the moment it's generated.

```
php artisan datahelm:scrap:generate "https://example.com/listing" --preset=ecommerce
```

Built-in presets (`config/crawler.php` → `presets`), selected with `--preset=` or the `CRAWLER_PRESET` env var (default: `generic`):

PresetUse for`generic`Unsure / mixed content — safe default for any country, any vertical`ecommerce`Online shops, marketplaces (adds `handle`, `product-image`, `qty`, …)`auctions`Auction/lot listings`properties`Real-estate listingsEach preset is an array of hints the detectors match against CSS classes, HTML attributes, and JSON field names:

KeyControls`price_patterns`Regexes for currency formats (`$`, `R$`, `€`, `£`, …) — **locale-specific**, add your own symbol if missing`image_field_hints`CSS class / JSON key fragments that mark an image field (`image`, `thumb`, `gallery`, …)`link_field_hints`Same, for the item's URL/link field (`url`, `href`, `slug`, `handle`, …)`rating_hints`CSS class fragments for star/score widgets`stock_hints`CSS class fragments for availability/inventory`sku_hints`CSS class / JSON key fragments for product codes (SKU, EAN, MPN, …)`image_path_prefix`A fixed URL path segment that identifies image URLs on a known platform (e.g. VTEX's `/arquivos/`); `null` = auto`list_core_fields`, `list_min_core_fields`, `list_min_success_rate`, `list_min_link_uniqueness`Thresholds the detector uses to decide "this repeating block is really a list of items"`item_schema`Suggested `item_schema` (type-coercion map) to carry into the generated blueprint`image_field_hints` / `link_field_hints` / `rating_hints` / `stock_hints` / `sku_hints`are CSS-class vocabulary and stay in English regardless of the page's display language (developers write `class="star-rating"` on French/Portuguese/Arabic sites alike). Only `price_patterns` and `image_path_prefix` are actually locale/platform specific.

**Adding your own preset** — extend an existing one by merging in local vocabulary, in `config/crawler.php`:

```
'presets' => [
    // ...
    'auctions_pt_BR' => [
        'price_patterns'    => ['/R\$\s*[\d.,]+/'],
        'image_field_hints' => array_merge(
            ['image', 'img', 'photo', 'thumb'],
            ['foto', 'fotos', 'imagem', 'imagens', 'galeria'],
        ),
        'link_field_hints'  => ['url', 'link', 'href', 'permalink', 'lote'],
        'image_path_prefix' => '/arquivos/',
    ],
],
```

```
php artisan datahelm:scrap:generate  --preset=auctions_pt_BR
```

Item pipeline
-------------

[](#item-pipeline)

Where a preset shapes *how fields are found*, the pipeline shapes *what happens to their values afterwards* — it runs on every crawl execution (`datahelm:scrap:run`), not just generation, transforming each already-extracted `ScrapedItem` before it's exported.

By default (`config/crawler.php` → `pipeline`) every item passes through:

1. **`TrimProcessor`** — collapses whitespace and trims every string field.
2. **`AbsoluteUrlProcessor`** — resolves relative `link`/`image`/`gallery_images`/… URLs against the page they were scraped from.

A blueprint can override this default for itself with `pipeline_names` — a list of short names resolved against `config('crawler.pipeline_registry')`:

```
'pipeline_registry' => [
    'trim'            => TrimProcessor::class,
    'absolute_url'    => AbsoluteUrlProcessor::class,
    'schema_coercion' => SchemaCoercionProcessor::class,
],
```

```
{ "pipeline_names": ["trim", "schema_coercion"] }
```

This is a **replacement, not an addition**: listing `["trim"]` runs only `TrimProcessor` for that blueprint — `AbsoluteUrlProcessor` no longer runs, so relative URLs are left as-is. Leaving `pipeline_names` empty (`[]`, the default) keeps the global pipeline untouched — most blueprints never need to set this.

**Adding a custom processor** — implement `ItemProcessor`, register it, then reference it by name:

```
final class StripEmojiProcessor implements \DataHelm\Crawler\Pipeline\ItemProcessor
{
    public function process(ScrapedItem $item, string $pageUrl): ScrapedItem
    {
        $title = $item->get('title');
        if (is_string($title)) {
            $item->set('title', preg_replace('/[\x{1F300}-\x{1FAFF}]/u', '', $title));
        }

        return $item;
    }
}
```

```
// config/crawler.php
'pipeline_registry' => [
    // ...
    'strip_emoji' => \App\Pipeline\StripEmojiProcessor::class,
],
```

```
{ "pipeline_names": ["trim", "absolute_url", "strip_emoji"] }
```

### Presets vs. pipeline

[](#presets-vs-pipeline)

They sound similar (both pick a named config by string) but act at opposite ends of the process:

PresetsPipelineRuns during`scrap:generate` only, once`scrap:run`, every executionActs onDetection heuristics — **finding** the right selectorsExtracted values — **transforming** them after extractionSelected via`--preset=ecommerce` on the CLI`"pipeline_names": [...]` in the blueprint JSONOutlives the run?No — only its effect on the generated blueprint persistsYes — read from the blueprint on every future runMarkdown / LLM-ready output
---------------------------

[](#markdown--llm-ready-output)

Turn any page into clean Markdown instead of a wall of HTML — the feature Firecrawl and Crawl4AI are known for, now in the Laravel world. Two ways to use it:

**1. As a field** — render one element's content (an article body, a product description) as Markdown by setting the field `type` to `markdown`. The `css` selector locates the element; its content is converted to Markdown (headings, lists, links, images, code, tables preserved; scripts, styles, and site chrome stripped):

```
{
  "name": "description",
  "css": ".product-description",
  "type": "markdown"
}
```

**2. As an output format** — export a whole crawl as a single Markdown document, one section per item, ready to drop into an LLM context window or a RAG index. Set the blueprint's `output.format`:

```
{ "output": { "format": "markdown" } }
```

```
php artisan datahelm:scrap:run example --output=storage/app/scrapes/example.md
```

The converter (`DataHelm\Crawler\Markdown\HtmlToMarkdown`) is dependency-free (ext-dom only) and can be used on its own:

```
$markdown = (new \DataHelm\Crawler\Markdown\HtmlToMarkdown())->convert($html);
```

Output formats
--------------

[](#output-formats)

Set `output.format` in the blueprint (or rely on the default). Every format writes to `--output=` or `storage/app/scrapes/.` when omitted; use `--output=-`to stream to STDOUT.

FormatExtBest for`json` (**default**)`.json`Pretty array — human-readable, small crawls`jsonl``.jsonl`One object per line — large crawls, streaming consumers`csv``.csv`Spreadsheets; array fields are JSON-encoded per cell`markdown``.md`**LLM ingestion / RAG** — one Markdown section per itemHTTP transports
---------------

[](#http-transports)

The package works out of the box with **plain HTTP** (`guzzle`). Heavier transports are **optional** — only needed for JS-heavy sites or bot protection.

TransportWhat it doesExtra infrastructure`guzzle`Plain HTTP (**default**)None`auto`Escalates on bot blocksbrowserless and/or FlareSolverr recommended`browser`Headless Chrome (JS / SPA)[browserless](https://github.com/browserless/chrome)`flaresolverr`Cloudflare challenge solver[FlareSolverr](https://github.com/FlareSolverr/FlareSolverr)`scraping_api`Managed anti-bot APIPaid API keyEscalation ladder when using `auto`:

```
guzzle ─► browser ─► flaresolverr ─► scraping_api

```

Environment variables
---------------------

[](#environment-variables)

VariableDefaultDescription`CRAWLER_TRANSPORT``guzzle``guzzle`, `browser`, `flaresolverr`, `scraping_api`, `auto``CRAWLER_COMMAND_PREFIX``datahelm`Artisan command prefix (`datahelm:scrap:generate`, …)`BROWSERLESS_URL``http://browserless:3000`browserless service URL`BROWSERLESS_TOKEN`*(empty)*Optional browserless auth token`FLARESOLVERR_URL``http://flaresolverr:8191`FlareSolverr service URL`FLARESOLVERR_MAX_TIMEOUT``60000`Challenge timeout (ms)`CRAWLER_PROXY_URL`*(empty)*Upstream proxy for browser / flaresolverr transports`SCRAPING_API_URL`*(empty)*Managed scraping API base URL`SCRAPING_API_KEY`*(empty)*API key for `scraping_api` transportWhen Laravel runs **inside Docker** on the same network as the services, use hostnames `browserless` and `flaresolverr`. When Laravel runs on the **host machine**, use `http://localhost:3010` and `http://localhost:8191`.

Optional: anti-bot services only
--------------------------------

[](#optional-anti-bot-services-only)

Start browserless and FlareSolverr without a full development stack:

```
docker compose -f docker/compose.services.yml up -d
```

Stop when done (each service runs a full Chromium and uses RAM/CPU):

```
docker compose -f docker/compose.services.yml stop
```

Full development environment
----------------------------

[](#full-development-environment)

For nginx, PHP, PostgreSQL, Redis, Supervisor, and all crawler services together, use the separate environment repository:

**[github.com/datahelm/environment](https://github.com/datahelm/environment)**

```
git clone https://github.com/datahelm/environment.git
cd environment
cp .env.example .env
export UID=$(id -u) GID=$(id -g)
docker compose up -d
```

Artisan commands
----------------

[](#artisan-commands)

CommandDescription`datahelm:scrap:generate`Auto-detect a site and generate a scrape blueprint`datahelm:scrap:run`Run a blueprint and export items (JSON / JSONL / CSV / Markdown)`datahelm:scrap:shell`Interactive CSS/XPath selector shell against a live URL`datahelm:scrap:validate`Validate a blueprint JSON file`datahelm:robot:{name}`Run a site-specific robot (scaffolded with `--robot`)What's new
----------

[](#whats-new)

**LLM-ready Markdown output** — the Firecrawl / Crawl4AI feature, now in Laravel.

- **`markdown` field type** — set a field's `type` to `markdown` to render the matched element's content as clean Markdown (headings, nested lists, ordered lists, links, images, fenced code with language, tables, blockquotes, ``). Scripts, styles, and site chrome (`nav`/`header`/`footer`/`aside`) are stripped. See [Markdown / LLM-ready output](#markdown--llm-ready-output).
- **`markdown` output format** — set `output.format` to `markdown` to export a whole crawl as a single Markdown document, one section per item, ready for an LLM context window or a RAG index. See [Output formats](#output-formats).
- **`HtmlToMarkdown` converter** (`DataHelm\Crawler\Markdown\HtmlToMarkdown`) — the engine behind both, dependency-free (ext-dom only) and usable on its own.
- **Fix:** `datahelm:scrap:run` now honours the blueprint's `output.format`(JSON / JSONL / CSV / Markdown); previously it always wrote JSON.

License
-------

[](#license)

MIT

###  Health Score

42

—

FairBetter than 88% of packages

Maintenance92

Actively maintained with recent releases

Popularity7

Limited adoption so far

Community6

Small or concentrated contributor base

Maturity52

Maturing project, gaining track record

 Bus Factor1

Top contributor holds 100% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Every ~4 days

Total

5

Last Release

32d ago

### Community

Maintainers

![](https://avatars.githubusercontent.com/u/19832238?v=4)[murilo livorato](/maintainers/murilolivorato)[@murilolivorato](https://github.com/murilolivorato)

---

Top Contributors

[![murilolivorato](https://avatars.githubusercontent.com/u/19832238?v=4)](https://github.com/murilolivorato "murilolivorato (4 commits)")

---

Tags

laravelcrawlerscraperweb-scraping

###  Code Quality

TestsPHPUnit

### Embed Badge

![Health badge](/badges/datahelm-crawler/health.svg)

```
[![Health](https://phpackages.com/badges/datahelm-crawler/health.svg)](https://phpackages.com/packages/datahelm-crawler)
```

###  Alternatives

[statamic/cms

The Statamic CMS Core Package

4.9k3.8M1.2k](/packages/statamic-cms)[craftcms/cms

Craft CMS

3.6k3.7M3.4k](/packages/craftcms-cms)[backpack/crud

Quickly build admin interfaces using Laravel, Bootstrap and JavaScript.

3.4k3.8M228](/packages/backpack-crud)[unopim/unopim

UnoPim Laravel PIM

10.8k2.5k](/packages/unopim-unopim)[spatie/crawler

Crawl all internal links found on a website

2.8k19.3M74](/packages/spatie-crawler)[bagisto/bagisto

Bagisto Laravel E-Commerce

28.0k175.2k9](/packages/bagisto-bagisto)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)