PHPackages lobotomised/laravel-autocrawler - PHPackages - PHPackages [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register) 1. [Directory](/) 2. / 3. [HTTP & Networking](/categories/http) 4. / 5. lobotomised/laravel-autocrawler ActiveLibrary[HTTP & Networking](/categories/http) lobotomised/laravel-autocrawler =============================== A tool to crawl your own laravel installation checking your HTTP status codes 1.3.0(9mo ago)242.0k—2.5%MITPHPPHP ^8.2CI passing Since Aug 16Pushed 9mo ago1 watchersCompare [ Source](https://github.com/lobotomised/laravel-autocrawler)[ Packagist](https://packagist.org/packages/lobotomised/laravel-autocrawler)[ Docs](https://github.com/lobotomised/laravel-autocrawler)[ RSS](/packages/lobotomised-laravel-autocrawler/feed)WikiDiscussions master Synced 1mo ago READMEChangelog (5)Dependencies (8)Versions (10)Used By (0) Check your application for broken links ======================================= [](#check-your-application-for-broken-links) [![GitHub Tests Action Status](https://github.com/lobotomised/laravel-autocrawler/actions/workflows/run-test.yml/badge.svg)](https://github.com/lobotomised/laravel-autocrawler/actions/workflows/run-test.yml)[![Total Downloads](https://camo.githubusercontent.com/f0176f3ae459457b6c4eb568c684fc1a3e71b290705fb6e5e5c1a6d1bd69acc3/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f64742f6c6f626f746f6d697365642f6c61726176656c2d6175746f637261776c65722e7376673f7374796c653d666c61742d737175617265)](https://packagist.org/packages/lobotomised/laravel-autocrawler)[![Latest Stable Version](https://camo.githubusercontent.com/e8a34cb51667e646b1a6a66708d00cd34d10bc3341eaa86cb02c18bf98a9296a/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f762f6c6f626f746f6d697365642f6c61726176656c2d6175746f637261776c6572)](https://packagist.org/packages/lobotomised/laravel-autocrawler)[![License](https://camo.githubusercontent.com/112ab1f68568fbc47314b7fd3d71bb8b38b99572e8f7efec788498663c455bcd/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f6c2f6c6f626f746f6d697365642f6c61726176656c2d6175746f637261776c6572)](https://packagist.org/packages/lobotomised/laravel-autocrawler) Using this package you can check if your application have broken links. ``` php artisan crawl 200 OK - http://myapp.test/ 200 OK - http://myapp.test/login found on http://myapp.test/ 200 OK - http://myapp.test/register found on http://myapp.test/ 301 301 Moved Permanently - http://myapp.test/homepage found on http://myapp.test/register 404 Not Found - http://myapp.test/brokenlink found on http://myapp.test/register 200 OK - http://myapp.test/features found on http://myapp.test/ Crawl finished Results: Status 200: 4 founds Status 301: 1 found Status 404: 1 found ``` Installation ------------ [](#installation) This package can be installed via Composer: ``` composer require --dev lobotomised/laravel-autocrawler ``` When crawling your site, it will automatically detect the url your application is using. If instead it scan , check in your .env you properly configure the APP\_URL variable ``` APP_URL="http://myapp.test" ``` Usage ----- [](#usage) ### Crawl a specific url [](#crawl-a-specific-url) By default, the crawler will crawl the URL from your current laravel installation. You can force the url with the `--url` option: ``` php artisan crawl --url=http://myapp.test/my-page ``` ### Concurrent connection [](#concurrent-connection) The crawler run with 10 concurrent connections to speed up the crawling process. You can change that by passing the `--concurrency` option: ``` php artisan crawl --concurrency=5 ``` ### Timeout [](#timeout) The request timeout is by default 30 seconds. Use the `--timeout` to change this value ``` php artisan crawl --timeout=10 ``` ### Ignore robots.txt [](#ignore-robotstxt) By default, the crawler respect the robots.txt. These rules can be ignored with the `--ignore-robots` option: ``` php artisan crawl --ignore-robots ``` ### External link [](#external-link) When the crawler find an external link, it will check this link. It can be deactivated with the `--ignore-external-links` option: ``` php artisan crawl --ignore-external-links ``` ### Log non-2xx or non-3xx status code [](#log-non-2xx-or-non-3xx-status-code) By default, the crawler will only in your console. You can log all non-2xx or non 3xx status code to a file with the `--output` option. Result will be store in `storage/autocrawler/output.txt` ``` php artisan crawl --output ``` The output.txt will look like that: ``` 403 Forbidden - http://myapp.test/dashboard found on http://myapp.test/home 404 Not Found - http://myapp.test/brokenlink found on http://myapp.test/register ``` ### Fail when non-2xx or non-3xx are found [](#fail-when-non-2xx-or-non-3xx-are-found) By default, the command exit codes is 0. You can change it to 1 to indicate that the command has failed with the `--fail-on-error` ``` php artisan crawl --fail-on-error ``` ### Launch the robot interactively [](#launch-the-robot-interactively) Eventually, you may configure the crawler interactively by using the `--interactive` option: ``` php artisan crawl --interactive ``` Working with GitHub actions --------------------------- [](#working-with-github-actions) To execute the crawler you first need to start a web server. You can choose to install apache or nginx. Here is an example using the php build-in webserver If the crawl found some non-2xx or non-3xx response, the action will fail, and the result will be store as an artifacts of the Action. ``` steps: - uses: actions/checkout@v3 - name: Prepare The Environment run: cp .env.example .env - name: Install Composer Dependencies run: composer install - name: Generate Application Key run: php artisan key:generate - name: Install npm Dependencies run: npm ci - name: Compile assets run: npm run build - name: Start php build-in webserver run: (php artisan serve &) || /bin/true - name: Crawl website run: php artisan crawl --url=http://localhost:8000/ --fail-on-error --output - name: Upload artifacts if: failure() uses: actions/upload-artifact@master with: name: Autocrawler path: ./storage/autocrawler ``` Documentation ------------- [](#documentation) All commands and informations are available with the command: ``` php artisan crawl --help ``` Alternatives ------------ [](#alternatives) This package is heavily inspire by [spatie/http-status-check](https://github.com/spatie/http-status-check), but instead of being a project dependency, it is a global installation Testing ------- [](#testing) First we need to start the included node http server in a separate terminal. ``` make start ``` Then to run the tests: ``` make test ``` Changelog --------- [](#changelog) Please see [CHANGELOG](CHANGELOG.md) for more information on what has changed recently. License ------- [](#license) The MIT License (MIT). Please see [License File](LICENSE.md) for more information. ### Health Score 44 — FairBetter than 92% of packages Maintenance56 Moderate activity, may be stable Popularity31 Limited adoption so far Community9 Small or concentrated contributor base Maturity65 Established project with proven stability Bus Factor1 Top contributor holds 89.5% of commits — single point of failure How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window. **Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores. **Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement. **Maturity (30%)** — Project age, version count, PHP version support, and release stability. ### Release Activity Cadence Every ~134 days Recently: every ~269 days Total 9 Last Release 295d ago Major Versions 0.0.4 → 1.0.02022-08-17 PHP version history (3 changes)0.0.1PHP ^8.0 1.1.0PHP ^8.1 1.3.0PHP ^8.2 ### Community Maintainers ![](https://www.gravatar.com/avatar/fe4805f3bac92a9671c86a70ac57b88556008a45856752b187b29fe24cc36b9d?d=identicon)[lobotomised](/maintainers/lobotomised) --- Top Contributors [![lobotomised](https://avatars.githubusercontent.com/u/34192441?v=4)](https://github.com/lobotomised "lobotomised (34 commits)")[![dependabot[bot]](https://avatars.githubusercontent.com/in/29110?v=4)](https://github.com/dependabot[bot] "dependabot[bot] (4 commits)") --- Tags autocrawlcrawlerlaravellaravel-packagelaravelcrawlerautocrawler ### Code Quality TestsPest Static AnalysisPHPStan ### Embed Badge ![Health badge](/badges/lobotomised-laravel-autocrawler/health.svg) ``` [![Health](https://phpackages.com/badges/lobotomised-laravel-autocrawler/health.svg)](https://phpackages.com/packages/lobotomised-laravel-autocrawler) ``` ### Alternatives [barryvdh/laravel-ide-helper Laravel IDE Helper, generates correct PHPDocs for all Facade classes, to improve auto-completion. 14.9k123.0M687](/packages/barryvdh-laravel-ide-helper)[laravel/ui Laravel UI utilities and presets. 2.7k134.9M601](/packages/laravel-ui)[tucker-eric/eloquentfilter An Eloquent way to filter Eloquent Models 1.8k4.8M26](/packages/tucker-eric-eloquentfilter)[wn/lumen-generators A collection of generators for Lumen and Laravel 5. 350205.3k2](/packages/wn-lumen-generators)[erag/laravel-disposable-email A Laravel package to detect and block disposable email addresses. 226102.4k](/packages/erag-laravel-disposable-email)[cybercog/laravel-clickhouse ClickHouse migrations for Laravel 163166.8k](/packages/cybercog-laravel-clickhouse) PHPackages © 2026 [Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)