PHPackages spatie/robots-txt - PHPackages - PHPackages [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register) 1. [Directory](/) 2. / 3. [Utility & Helpers](/categories/utility) 4. / 5. spatie/robots-txt ActiveLibrary[Utility & Helpers](/categories/utility) spatie/robots-txt ================= Determine if a page may be crawled from robots.txt and robots meta tags 2.5.4(2mo ago)25616.4M—3.1%4420MITPHPPHP ^8.1CI passing Since Mar 9Pushed 4w ago9 watchersCompare [ Source](https://github.com/spatie/robots-txt)[ Packagist](https://packagist.org/packages/spatie/robots-txt)[ Docs](https://github.com/spatie/robots-txt)[ Fund](https://spatie.be/open-source/support-us)[ GitHub Sponsors](https://github.com/spatie)[ RSS](/packages/spatie-robots-txt/feed)WikiDiscussions main Synced 1mo ago READMEChangelog (10)Dependencies (1)Versions (33)Used By (20) Parse `robots.txt`, `robots` meta and headers ============================================= [](#parse-robotstxt-robots-meta-and-headers) [![Latest Version on Packagist](https://camo.githubusercontent.com/52a196ac03f7fdb6be42782d0629f3921de055cd99a30c7555c37d7987fd9dd7/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f762f7370617469652f726f626f74732d7478742e7376673f7374796c653d666c61742d737175617265)](https://packagist.org/packages/spatie/robots-txt)[![Tests](https://github.com/spatie/robots-txt/actions/workflows/run-tests.yml/badge.svg)](https://github.com/spatie/robots-txt/actions/workflows/run-tests.yml)[![Quality Score](https://camo.githubusercontent.com/73a6bc4afeb4f25e61541cfd0ef178cdf8468da1c5e23bb97ee413d95b4864f0/68747470733a2f2f696d672e736869656c64732e696f2f7363727574696e697a65722f672f7370617469652f726f626f74732d7478742e7376673f7374796c653d666c61742d737175617265)](https://scrutinizer-ci.com/g/spatie/robots-txt)[![Total Downloads](https://camo.githubusercontent.com/be20bb8bbf5d8199bd1b055e5fc190b70c9438eceb6d98e364e34e99f6fb3f40/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f64742f7370617469652f726f626f74732d7478742e7376673f7374796c653d666c61742d737175617265)](https://packagist.org/packages/spatie/robots-txt) Determine if a page may be crawled from robots.txt, robots meta tags and robot headers. Support us ---------- [](#support-us) [![](https://camo.githubusercontent.com/9faa461e0cfdb3912f1682f6674ecf689986d165a42dbc30a73520641b822602/68747470733a2f2f6769746875622d6164732e73332e65752d63656e7472616c2d312e616d617a6f6e6177732e636f6d2f726f626f74732d7478742e6a70673f743d31)](https://spatie.be/github-ad-click/robots-txt) We invest a lot of resources into creating [best in class open source packages](https://spatie.be/open-source). You can support us by [buying one of our paid products](https://spatie.be/open-source/support-us). We highly appreciate you sending us a postcard from your hometown, mentioning which of our package(s) you are using. You'll find our address on [our contact page](https://spatie.be/about-us). We publish all received postcards on [our virtual postcard wall](https://spatie.be/open-source/postcards). Installation ------------ [](#installation) You can install the package via composer: ``` composer require spatie/robots-txt ``` Usage ----- [](#usage) ``` $robots = Spatie\Robots\Robots::create(); $robots->mayIndex('https://www.spatie.be/nl/admin'); $robots->mayFollowOn('https://www.spatie.be/nl/admin'); $robotsTxt = new RobotsTxt(' User-agent: * Disallow: /admin Crawl-delay: 1.5 '); $robotsTxt->allows('/admin', 'google'); // false $robotsTxt->whyDisallows('/admin', 'google')[0]->userAgent; // '*' $robotsTxt->crawlDelay('/admin', '*'); // '1.5' ``` You can also specify a user agent: ``` $robots = Spatie\Robots\Robots::create('UserAgent007'); ``` By default, `Robots` will look for a `robots.txt` file on `https://host.com/robots.txt`. Another location can be specified like so: ``` $robots = Spatie\Robots\Robots::create() ->withTxt('https://www.spatie.be/robots-custom.txt'); $robots = Spatie\Robots\Robots::create() ->withTxt(__DIR__ . '/public/robots.txt'); ``` ### Testing [](#testing) ``` composer test ``` ### Changelog [](#changelog) Please see [CHANGELOG](CHANGELOG.md) for more information what has changed recently. Contributing ------------ [](#contributing) Please see [CONTRIBUTING](https://github.com/spatie/.github/blob/main/CONTRIBUTING.md) for details. Security Vulnerabilities ------------------------ [](#security-vulnerabilities) Please review [our security policy](../../security/policy) on how to report security vulnerabilities. Postcardware ------------ [](#postcardware) You're free to use this package, but if it makes it to your production environment we highly appreciate you sending us a postcard from your hometown, mentioning which of our package(s) you are using. Our address is: Spatie, Kruikstraat 22, 2018 Antwerp, Belgium. We publish all received postcards [on our company website](https://spatie.be/en/opensource/postcards). Credits ------- [](#credits) - [Brent Roose](https://github.com/brendt) - [All Contributors](../../contributors) License ------- [](#license) The MIT License (MIT). Please see [License File](LICENSE.md) for more information. ### Health Score 74 — ExcellentBetter than 100% of packages Maintenance90 Actively maintained with recent releases Popularity67 Solid adoption and visibility Community40 Growing community involvement Maturity83 Battle-tested with a long release history Bus Factor2 2 contributors hold 50%+ of commits How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window. **Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores. **Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement. **Maturity (30%)** — Project age, version count, PHP version support, and release stability. ### Release Activity Cadence Every ~97 days Recently: every ~69 days Total 31 Last Release 83d ago Major Versions 0.1.3 → 1.0.02018-05-07 1.0.10 → 2.0.02021-03-28 PHP version history (4 changes)0.1.0PHP ^7.1 1.0.9PHP ^7.3 || ^8.0 2.0.0PHP ^8.0 2.2.0PHP ^8.1 ### Community Maintainers ![](https://avatars.githubusercontent.com/u/7535935?v=4)[Spatie](/maintainers/spatie)[@spatie](https://github.com/spatie) --- Top Contributors [![freekmurze](https://avatars.githubusercontent.com/u/483853?v=4)](https://github.com/freekmurze "freekmurze (91 commits)")[![brendt](https://avatars.githubusercontent.com/u/6905297?v=4)](https://github.com/brendt "brendt (22 commits)")[![dependabot[bot]](https://avatars.githubusercontent.com/in/29110?v=4)](https://github.com/dependabot[bot] "dependabot[bot] (13 commits)")[![Nielsvanpach](https://avatars.githubusercontent.com/u/10651054?v=4)](https://github.com/Nielsvanpach "Nielsvanpach (8 commits)")[![AdrianMrn](https://avatars.githubusercontent.com/u/12762044?v=4)](https://github.com/AdrianMrn "AdrianMrn (8 commits)")[![BenMorel](https://avatars.githubusercontent.com/u/1952838?v=4)](https://github.com/BenMorel "BenMorel (6 commits)")[![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?v=4)](https://github.com/github-actions[bot] "github-actions[bot] (6 commits)")[![gvlasov](https://avatars.githubusercontent.com/u/4467948?v=4)](https://github.com/gvlasov "gvlasov (5 commits)")[![tsjason](https://avatars.githubusercontent.com/u/26229182?v=4)](https://github.com/tsjason "tsjason (3 commits)")[![Enaah](https://avatars.githubusercontent.com/u/10073766?v=4)](https://github.com/Enaah "Enaah (3 commits)")[![patinthehat](https://avatars.githubusercontent.com/u/5508707?v=4)](https://github.com/patinthehat "patinthehat (3 commits)")[![Redominus](https://avatars.githubusercontent.com/u/22024214?v=4)](https://github.com/Redominus "Redominus (3 commits)")[![angeljqv](https://avatars.githubusercontent.com/u/79208641?v=4)](https://github.com/angeljqv "angeljqv (2 commits)")[![pascalbaljet](https://avatars.githubusercontent.com/u/8403149?v=4)](https://github.com/pascalbaljet "pascalbaljet (2 commits)")[![riasvdv](https://avatars.githubusercontent.com/u/3626559?v=4)](https://github.com/riasvdv "riasvdv (2 commits)")[![muhci](https://avatars.githubusercontent.com/u/14060891?v=4)](https://github.com/muhci "muhci (2 commits)")[![ihah](https://avatars.githubusercontent.com/u/35690113?v=4)](https://github.com/ihah "ihah (2 commits)")[![AlexVanderbist](https://avatars.githubusercontent.com/u/6287961?v=4)](https://github.com/AlexVanderbist "AlexVanderbist (2 commits)")[![stephane-monnot](https://avatars.githubusercontent.com/u/6066368?v=4)](https://github.com/stephane-monnot "stephane-monnot (1 commits)")[![peter279k](https://avatars.githubusercontent.com/u/9021747?v=4)](https://github.com/peter279k "peter279k (1 commits)") --- Tags crawlerphprobots-txtspatierobots-txt ### Code Quality TestsPHPUnit ### Embed Badge ![Health badge](/badges/spatie-robots-txt/health.svg) ``` [![Health](https://phpackages.com/badges/spatie-robots-txt/health.svg)](https://phpackages.com/packages/spatie-robots-txt) ``` ### Alternatives [spatie/laravel-package-tools Tools for creating Laravel packages 945125.5M7.0k](/packages/spatie-laravel-package-tools)[spatie/laravel-data Create unified resources and data transfer objects 1.8k28.9M627](/packages/spatie-laravel-data)[spatie/laravel-analytics A Laravel package to retrieve Google Analytics data. 3.2k5.7M57](/packages/spatie-laravel-analytics)[spatie/macroable A trait to dynamically add methods to a class 72759.6M64](/packages/spatie-macroable)[spatie/regex A sane interface for php's built in preg\_\* functions 1.1k17.1M59](/packages/spatie-regex)[spatie/enum PHP Enums 84529.1M68](/packages/spatie-enum) PHPackages © 2026 [Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)