PHPackages monteduro/laravel-datafast-aicrawl - PHPackages - PHPackages [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register) 1. [Directory](/) 2. / 3. [Utility & Helpers](/categories/utility) 4. / 5. monteduro/laravel-datafast-aicrawl ActiveLibrary[Utility & Helpers](/categories/utility) monteduro/laravel-datafast-aicrawl ================================== Server-side AI crawler & bot traffic tracking for Laravel, reporting to DataFast. Unofficial PHP port of @datafast/ai-crawl. v0.1.1(yesterday)24↑2900%MITPHPPHP ^8.2CI passing Since Jun 18Pushed yesterdayCompare [ Source](https://github.com/monteduro/laravel-datafast-aicrawl)[ Packagist](https://packagist.org/packages/monteduro/laravel-datafast-aicrawl)[ RSS](/packages/monteduro-laravel-datafast-aicrawl/feed)WikiDiscussions main Synced today READMEChangelogDependencies (5)Versions (3)Used By (0) Laravel DataFast AI Crawl ========================= [](#laravel-datafast-ai-crawl) [![Laravel DataFast AI Crawl — bot traffic tracking](https://raw.githubusercontent.com/monteduro/laravel-datafast-aicrawl/main/art/banner.jpeg)](https://raw.githubusercontent.com/monteduro/laravel-datafast-aicrawl/main/art/banner.jpeg) [![Latest Version on Packagist](https://camo.githubusercontent.com/df3da22f45a97a78501e6224024d7c63b4929183803fee91e7eb8d832c57e82d/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f762f6d6f6e74656475726f2f6c61726176656c2d64617461666173742d6169637261776c2e737667)](https://packagist.org/packages/monteduro/laravel-datafast-aicrawl)[![Total Downloads](https://camo.githubusercontent.com/06b34c3d25f1462c254b8abab77d458c726c3851dea023104ec3ecda8bc558c9/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f64742f6d6f6e74656475726f2f6c61726176656c2d64617461666173742d6169637261776c2e737667)](https://packagist.org/packages/monteduro/laravel-datafast-aicrawl)[![License](https://camo.githubusercontent.com/20a21f6a87d6dc6507360474f7913dfae3a5d5b41f0b6a3e8db7e71fe28b4c28/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f6c2f6d6f6e74656475726f2f6c61726176656c2d64617461666173742d6169637261776c2e737667)](LICENSE) Server-side **AI crawler & bot traffic tracking** for Laravel, reporting to [DataFast](https://datafa.st). See when AI assistants (ChatGPT, Claude, Perplexity…), search crawlers (Googlebot, Bingbot…) and model-training bots (GPTBot, ClaudeBot, CCBot…) request pages on your site. These bots fetch raw HTML and skip frontend JavaScript, so DataFast's browser script never sees them — this package detects them in your backend. > **Unofficial** PHP port of [`@datafast/ai-crawl`](https://www.npmjs.com/package/@datafast/ai-crawl). Not affiliated with DataFast. The ingestion endpoint it posts to is undocumented and may change. Install ------- [](#install) ``` composer require monteduro/laravel-datafast-aicrawl ``` The service provider is auto-discovered and the middleware is auto-registered on the `web` group. Two things to set in `.env`: ``` # Required — your DataFast website id DATAFAST_WEBSITE_ID=dfid_xxxxxxxx # Strongly recommended — run the tracking HTTP call on a real queue # so it never adds latency to your pages (defaults to the sync driver). DATAFAST_AICRAWL_QUEUE=default DATAFAST_AICRAWL_CONNECTION=redis ``` That's the whole setup. Requires PHP 8.2+ and Laravel 11/12. ### About the queue [](#about-the-queue) Every detected crawler request triggers one outbound HTTP POST to DataFast. That call is dispatched as a **best-effort queued job** (no retries, errors swallowed) and the middleware runs *after* the response is sent (`terminate()`), so even on the `sync` driver the visitor is never blocked. Point `DATAFAST_AICRAWL_CONNECTION`/`QUEUE` at a real worker for best throughput. How it works ------------ [](#how-it-works) For each `GET`/`HEAD` request the middleware classifies the `User-Agent` against a built-in catalog of ~80 crawlers, skips non-page traffic (static extensions, `/api`, `/_next`, `Sec-Fetch-Dest` sub-resources, …), and if it's a real crawler page hit, queues the event to DataFast. Payload sent: `websiteId`, `domain`, `href`, `referrer`, and an `ai` object with `provider`, `agent`, `category` (`answer_fetch` / `search_index` / `training` / `ai_crawler`), `userAgent`, `ip`, `statusCode`. Configuration ------------- [](#configuration) Optional — publish only to customize defaults: ``` php artisan vendor:publish --tag=datafast-aicrawl-config ``` Env varDefaultPurpose`DATAFAST_WEBSITE_ID`—**Required.** Your DataFast website id.`DATAFAST_AICRAWL_QUEUE`defaultQueue name for the tracking job.`DATAFAST_AICRAWL_CONNECTION`defaultQueue connection.`DATAFAST_AICRAWL_ENABLED``true`Master on/off switch.`DATAFAST_AICRAWL_ENDPOINT``https://datafa.st/api/ai-crawls`Ingestion endpoint.`DATAFAST_AICRAWL_DOMAIN`request hostOverride reported domain.`DATAFAST_AICRAWL_TIMEOUT``5`HTTP timeout (seconds).`DATAFAST_AICRAWL_AUTO_MIDDLEWARE``true`Auto-append middleware to a group.`DATAFAST_AICRAWL_MIDDLEWARE_GROUP``web`Group to append the middleware to.Disable crawler categories or tweak ignored paths/extensions in [`config/datafast-aicrawl.php`](config/datafast-aicrawl.php). To register the middleware manually instead, set `DATAFAST_AICRAWL_AUTO_MIDDLEWARE=false` and attach `\Monteduro\DataFastAiCrawl\Http\Middleware\TrackAiCrawler::class` yourself. Keeping the crawler list fresh ------------------------------ [](#keeping-the-crawler-list-fresh) The catalog data lives in [`resources/crawlers.php`](resources/crawlers.php), generated from the upstream npm bundle (not fetched at runtime). You normally never touch it — a **GitHub Action runs weekly**, regenerates it from `@datafast/ai-crawl`, runs the tests, and opens a PR when something changed. Merging that PR auto-tags the next patch version, which Packagist publishes. To run the sync manually: ``` composer sync-catalog:check # show drift vs upstream (exits non-zero if outdated) composer sync-catalog # regenerate resources/crawlers.php ``` > The weekly workflow needs **Settings → Actions → General → "Allow GitHub Actions to create and approve pull requests"** enabled on the repo. Testing ------- [](#testing) ``` composer install && composer test ``` Credits & License --------------------- [](#credits--license) Crawler catalog and detection logic ported from [`@datafast/ai-crawl`](https://www.npmjs.com/package/@datafast/ai-crawl) (MIT). MIT licensed — see [LICENSE](LICENSE). ### Health Score 39 — LowBetter than 85% of packages Maintenance100 Actively maintained with recent releases Popularity8 Limited adoption so far Community6 Small or concentrated contributor base Maturity37 Early-stage or recently created project Bus Factor1 Top contributor holds 100% of commits — single point of failure How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window. **Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores. **Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement. **Maturity (30%)** — Project age, version count, PHP version support, and release stability. ### Release Activity Cadence Every ~0 days Total 2 Last Release 1d ago ### Community Maintainers ![](https://avatars.githubusercontent.com/u/5638815?v=4)[monteduro](/maintainers/monteduro)[@monteduro](https://github.com/monteduro) --- Top Contributors [![monteduro](https://avatars.githubusercontent.com/u/5638815?v=4)](https://github.com/monteduro "monteduro (4 commits)") --- Tags ai-crawlersanalyticsbot-trafficdatafastlaravelllmmiddlewarephpseomiddlewarelaravelaicrawleranalyticsbotseollmdatafastbot-traffic ### Code Quality TestsPest ### Embed Badge ![Health badge](/badges/monteduro-laravel-datafast-aicrawl/health.svg) ``` [![Health](https://phpackages.com/badges/monteduro-laravel-datafast-aicrawl/health.svg)](https://phpackages.com/packages/monteduro-laravel-datafast-aicrawl) ``` ### Alternatives [psalm/plugin-laravel Psalm plugin for Laravel 3325.1M337](/packages/psalm-plugin-laravel)[laravel/mcp Rapidly build MCP servers for your Laravel applications. 76518.2M115](/packages/laravel-mcp)[spatie/laravel-export Create a static site bundle from a Laravel app 672139.5k6](/packages/spatie-laravel-export)[simplestats-io/laravel-client Analytics for Laravel. Track visitors, registrations, and payments. Discover which channels actually drive revenue, not just traffic. Server-side, GDPR compliant, ad-blocker proof. 5019.3k](/packages/simplestats-io-laravel-client)[api-platform/laravel API Platform support for Laravel 59156.3k11](/packages/api-platform-laravel)[aedart/athenaeum Athenaeum is a mono repository; a collection of various PHP packages 245.2k](/packages/aedart-athenaeum) PHPackages © 2026 [Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)