PHPackages                             spatie/crawler - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [HTTP & Networking](/categories/http)
4. /
5. spatie/crawler

ActiveLibrary[HTTP & Networking](/categories/http)

spatie/crawler
==============

Crawl all internal links found on a website

9.2.1(2mo ago)2.8k16.3M↓11.7%36820MITPHPPHP ^8.4CI passing

Since Nov 2Pushed today63 watchersCompare

[ Source](https://github.com/spatie/crawler)[ Packagist](https://packagist.org/packages/spatie/crawler)[ Docs](https://github.com/spatie/crawler)[ Fund](https://spatie.be/open-source/support-us)[ GitHub Sponsors](https://github.com/spatie)[ RSS](/packages/spatie-crawler/feed)WikiDiscussions main Synced 1mo ago

READMEChangelog (10)Dependencies (21)Versions (132)Used By (20)

 [   ![Logo for crawler](https://camo.githubusercontent.com/0a640ba75773fdddb909b38b0b080709bbe98c1c90c72a13453cdacffcca20ae/68747470733a2f2f7370617469652e62652f7061636b616765732f6865616465722f637261776c65722f68746d6c2f6c696768742e77656270)  ](https://spatie.be/open-source?utm_source=github&utm_medium=banner&utm_campaign=crawler)Crawl the web using PHP
=======================

[](#crawl-the-web-using-php)

[![Latest Version on Packagist](https://camo.githubusercontent.com/5920a25c35e56ff898c9f879853569a34014e8733ecdcc9db76fc1d27db9f4fe/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f762f7370617469652f637261776c65722e7376673f7374796c653d666c61742d737175617265)](https://packagist.org/packages/spatie/crawler)[![MIT Licensed](https://camo.githubusercontent.com/55c0218c8f8009f06ad4ddae837ddd05301481fcf0dff8e0ed9dadda8780713e/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d4d49542d627269676874677265656e2e7376673f7374796c653d666c61742d737175617265)](LICENSE.md)[![Tests](https://github.com/spatie/crawler/workflows/Tests/badge.svg)](https://github.com/spatie/crawler/workflows/Tests/badge.svg)[![Total Downloads](https://camo.githubusercontent.com/f0de3723819e89d47e5df65d7374aacf237e7ac08f5de198ee54d8234b1e5d23/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f64742f7370617469652f637261776c65722e7376673f7374796c653d666c61742d737175617265)](https://packagist.org/packages/spatie/crawler)

This package provides a powerful, easy to use class to crawl links on a website. Under the hood, Guzzle promises are used to [crawl multiple URLs concurrently](http://docs.guzzlephp.org/en/latest/quickstart.html?highlight=pool#concurrent-requests).

Because the crawler can execute JavaScript, it can crawl JavaScript rendered sites. Under the hood, [Chrome and Puppeteer](https://github.com/spatie/browsershot) are used to power this feature.

Here's a quick example:

```
use Spatie\Crawler\Crawler;
use Spatie\Crawler\CrawlResponse;

Crawler::create('https://example.com')
    ->onCrawled(function (string $url, CrawlResponse $response) {
        echo "{$url}: {$response->status()}\n";
    })
    ->start();
```

Or collect all URLs on a site:

```
$urls = Crawler::create('https://example.com')
    ->internalOnly()
    ->depth(3)
    ->foundUrls();
```

You can also test your crawl logic without making real HTTP requests:

```
Crawler::create('https://example.com')
    ->fake([
        'https://example.com' => 'About',
        'https://example.com/about' => 'About page',
    ])
    ->foundUrls();
```

If you need to stop a crawl based on external state, you can register a callback that receives the current crawler instance and is checked before scheduling each next request:

```
use Spatie\Crawler\Crawler;

$shouldStop = false;

Crawler::create('https://example.com')
    ->shouldStopCallback(function (Crawler $crawler) use (&$shouldStop) {
        return $shouldStop;
    })
    ->onCrawled(function (string $url) use (&$shouldStop) {
        $shouldStop = true;
    })
    ->start();
```

Support us
----------

[](#support-us)

[![](https://camo.githubusercontent.com/e9f8370802d8dbac5aa0b08505aa19bfe0a521574a2bf993287250b6f3509adc/68747470733a2f2f6769746875622d6164732e73332e65752d63656e7472616c2d312e616d617a6f6e6177732e636f6d2f637261776c65722e6a70673f743d31)](https://spatie.be/github-ad-click/crawler)

We invest a lot of resources into creating [best in class open source packages](https://spatie.be/open-source). You can support us by [buying one of our paid products](https://spatie.be/open-source/support-us).

We highly appreciate you sending us a postcard from your hometown, mentioning which of our package(s) you are using. You'll find our address on [our contact page](https://spatie.be/about-us). We publish all received postcards on [our virtual postcard wall](https://spatie.be/open-source/postcards).

Documentation
-------------

[](#documentation)

All documentation is available [on our documentation site](https://spatie.be/docs/crawler).

Testing
-------

[](#testing)

```
composer test
```

Changelog
---------

[](#changelog)

Please see [CHANGELOG](CHANGELOG.md) for more information on what has changed recently.

Contributing
------------

[](#contributing)

Please see [CONTRIBUTING](https://github.com/spatie/.github/blob/main/CONTRIBUTING.md) for details.

Security Vulnerabilities
------------------------

[](#security-vulnerabilities)

Please review [our security policy](../../security/policy) on how to report security vulnerabilities.

Credits
-------

[](#credits)

- [Freek Van der Herten](https://github.com/freekmurze)
- [All Contributors](../../contributors)

License
-------

[](#license)

The MIT License (MIT). Please see [License File](LICENSE.md) for more information.

###  Health Score

84

—

ExcellentBetter than 100% of packages

Maintenance95

Actively maintained with recent releases

Popularity76

Solid adoption and visibility

Community51

Growing community involvement

Maturity98

Battle-tested with a long release history

 Bus Factor1

Top contributor holds 69.8% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Every ~29 days

Recently: every ~4 days

Total

129

Last Release

60d ago

Major Versions

v4.x-dev → v5.x-dev2020-12-20

v5.x-dev → 6.0.12021-02-26

v6.x-dev → 7.0.02021-04-27

v7.x-dev → 8.0.02023-06-04

8.5.0 → v9.x-dev2026-03-02

PHP version history (11 changes)0.0.1PHP >=5.6.0

1.0.2PHP >=5.5.0

2.0.0PHP ^7.0

3.0.0PHP ^7.1

5.0.0PHP ^7.4

5.0.2PHP ^7.4|^8.0

4.7.6PHP ^7.3|^8.0

7.0.0PHP ^8.0

8.0.0PHP ^8.1

8.4.2PHP ^8.2

v9.x-devPHP ^8.4

### Community

Maintainers

![](https://avatars.githubusercontent.com/u/7535935?v=4)[Spatie](/maintainers/spatie)[@spatie](https://github.com/spatie)

---

Top Contributors

[![freekmurze](https://avatars.githubusercontent.com/u/483853?v=4)](https://github.com/freekmurze "freekmurze (347 commits)")[![brendt](https://avatars.githubusercontent.com/u/6905297?v=4)](https://github.com/brendt "brendt (35 commits)")[![Redominus](https://avatars.githubusercontent.com/u/22024214?v=4)](https://github.com/Redominus "Redominus (15 commits)")[![AdrianMrn](https://avatars.githubusercontent.com/u/12762044?v=4)](https://github.com/AdrianMrn "AdrianMrn (10 commits)")[![denvers](https://avatars.githubusercontent.com/u/1016564?v=4)](https://github.com/denvers "denvers (9 commits)")[![mattiasgeniar](https://avatars.githubusercontent.com/u/407270?v=4)](https://github.com/mattiasgeniar "mattiasgeniar (8 commits)")[![BenMorel](https://avatars.githubusercontent.com/u/1952838?v=4)](https://github.com/BenMorel "BenMorel (7 commits)")[![rubenvanassche](https://avatars.githubusercontent.com/u/619804?v=4)](https://github.com/rubenvanassche "rubenvanassche (6 commits)")[![sebastiandedeyne](https://avatars.githubusercontent.com/u/1561079?v=4)](https://github.com/sebastiandedeyne "sebastiandedeyne (5 commits)")[![AlexVanderbist](https://avatars.githubusercontent.com/u/6287961?v=4)](https://github.com/AlexVanderbist "AlexVanderbist (5 commits)")[![spekulatius](https://avatars.githubusercontent.com/u/8433587?v=4)](https://github.com/spekulatius "spekulatius (4 commits)")[![pascalbaljet](https://avatars.githubusercontent.com/u/8403149?v=4)](https://github.com/pascalbaljet "pascalbaljet (4 commits)")[![localheinz](https://avatars.githubusercontent.com/u/605483?v=4)](https://github.com/localheinz "localheinz (3 commits)")[![superpenguin612](https://avatars.githubusercontent.com/u/74030080?v=4)](https://github.com/superpenguin612 "superpenguin612 (3 commits)")[![TVke](https://avatars.githubusercontent.com/u/15680337?v=4)](https://github.com/TVke "TVke (3 commits)")[![andrzejkupczyk](https://avatars.githubusercontent.com/u/11018286?v=4)](https://github.com/andrzejkupczyk "andrzejkupczyk (3 commits)")[![systream](https://avatars.githubusercontent.com/u/1583029?v=4)](https://github.com/systream "systream (3 commits)")[![mansoorkhan96](https://avatars.githubusercontent.com/u/51432274?v=4)](https://github.com/mansoorkhan96 "mansoorkhan96 (2 commits)")[![akalongman](https://avatars.githubusercontent.com/u/423050?v=4)](https://github.com/akalongman "akalongman (2 commits)")[![akoepcke](https://avatars.githubusercontent.com/u/5311185?v=4)](https://github.com/akoepcke "akoepcke (2 commits)")

---

Tags

concurrencycrawlerguzzlephpspatielinkcrawlerwebsite

###  Code Quality

TestsPest

Static AnalysisPHPStan

Type Coverage Yes

### Embed Badge

![Health badge](/badges/spatie-crawler/health.svg)

```
[![Health](https://phpackages.com/badges/spatie-crawler/health.svg)](https://phpackages.com/packages/spatie-crawler)
```

###  Alternatives

[craftcms/cms

Craft CMS

3.6k3.6M2.6k](/packages/craftcms-cms)[onesignal/onesignal-php-api

A powerful way to send personalized messages at scale and build effective customer engagement strategies. Learn more at onesignal.com

34170.2k2](/packages/onesignal-onesignal-php-api)[ory/hydra-client-php

Documentation for all of Ory Hydra's APIs.

1710.8k](/packages/ory-hydra-client-php)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)