PHPackages                             sonhb99/robots-txt - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. sonhb99/robots-txt

ActiveLibrary

sonhb99/robots-txt
==================

Determine if a page may be crawled from robots.txt and robots meta tags

04PHP

Since Mar 5Pushed 5y ago1 watchersCompare

[ Source](https://github.com/sonhb1231/Robots)[ Packagist](https://packagist.org/packages/sonhb99/robots-txt)[ RSS](/packages/sonhb99-robots-txt/feed)WikiDiscussions master Synced 1mo ago

READMEChangelogDependenciesVersions (1)Used By (0)

Parse `robots.txt`, `robots` meta and headers
=============================================

[](#parse-robotstxt-robots-meta-and-headers)

[![Latest Version on Packagist](https://camo.githubusercontent.com/52a196ac03f7fdb6be42782d0629f3921de055cd99a30c7555c37d7987fd9dd7/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f762f7370617469652f726f626f74732d7478742e7376673f7374796c653d666c61742d737175617265)](https://packagist.org/packages/spatie/robots-txt)[![GitHub Tests Action Status](https://camo.githubusercontent.com/802a8b4d8ea99e327503a815113013c30cd45f9ead1759c9e44f8afe2eb8ac34/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f776f726b666c6f772f7374617475732f7370617469652f726f626f74732d7478742f72756e2d74657374733f6c6162656c3d7465737473)](https://camo.githubusercontent.com/802a8b4d8ea99e327503a815113013c30cd45f9ead1759c9e44f8afe2eb8ac34/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f776f726b666c6f772f7374617475732f7370617469652f726f626f74732d7478742f72756e2d74657374733f6c6162656c3d7465737473)[![Quality Score](https://camo.githubusercontent.com/73a6bc4afeb4f25e61541cfd0ef178cdf8468da1c5e23bb97ee413d95b4864f0/68747470733a2f2f696d672e736869656c64732e696f2f7363727574696e697a65722f672f7370617469652f726f626f74732d7478742e7376673f7374796c653d666c61742d737175617265)](https://scrutinizer-ci.com/g/spatie/robots-txt)[![Total Downloads](https://camo.githubusercontent.com/be20bb8bbf5d8199bd1b055e5fc190b70c9438eceb6d98e364e34e99f6fb3f40/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f64742f7370617469652f726f626f74732d7478742e7376673f7374796c653d666c61742d737175617265)](https://packagist.org/packages/spatie/robots-txt)[![StyleCI](https://camo.githubusercontent.com/9adaaecaedece23d9dede8969d0c0e56debc51ba377017a68d9a53aff57034a7/68747470733a2f2f7374796c6563692e696f2f7265706f732f3132323937393730372f736869656c643f6272616e63683d6d6173746572)](https://styleci.io/repos/122979707)

Determine if a page may be crawled from robots.txt, robots meta tags and robot headers.

Support us
----------

[](#support-us)

[![](https://camo.githubusercontent.com/9faa461e0cfdb3912f1682f6674ecf689986d165a42dbc30a73520641b822602/68747470733a2f2f6769746875622d6164732e73332e65752d63656e7472616c2d312e616d617a6f6e6177732e636f6d2f726f626f74732d7478742e6a70673f743d31)](https://spatie.be/github-ad-click/robots-txt)

We invest a lot of resources into creating [best in class open source packages](https://spatie.be/open-source). You can support us by [buying one of our paid products](https://spatie.be/open-source/support-us).

We highly appreciate you sending us a postcard from your hometown, mentioning which of our package(s) you are using. You'll find our address on [our contact page](https://spatie.be/about-us). We publish all received postcards on [our virtual postcard wall](https://spatie.be/open-source/postcards).

Installation
------------

[](#installation)

You can install the package via composer:

```
composer require spatie/robots-txt
```

Usage
-----

[](#usage)

```
$robots = Robots::create();
$robots->mayIndex('https://www.spatie.be/nl/admin');
$robots->mayFollowOn('https://www.spatie.be/nl/admin');
```

You can also specify a user agent:

```
$robots = Robots::create('UserAgent007');
```

By default, `Robots` will look for a `robots.txt` file on `https://host.com/robots.txt`. Another location can be specified like so:

```
$robots = Robots::create()
    ->withTxt('https://www.spatie.be/robots-custom.txt');
$robots = Robots::create()
    ->withTxt(__DIR__ . '/public/robots.txt');
```

### Testing

[](#testing)

```
composer test
```

### Changelog

[](#changelog)

Please see [CHANGELOG](CHANGELOG.md) for more information what has changed recently.

Contributing
------------

[](#contributing)

Please see [CONTRIBUTING](CONTRIBUTING.md) for details.

### Security

[](#security)

If you discover any security related issues, please email  instead of using the issue tracker.

Postcardware
------------

[](#postcardware)

You're free to use this package, but if it makes it to your production environment we highly appreciate you sending us a postcard from your hometown, mentioning which of our package(s) you are using.

Our address is: Spatie, Kruikstraat 22, 2018 Antwerp, Belgium.

We publish all received postcards [on our company website](https://spatie.be/en/opensource/postcards).

Credits
-------

[](#credits)

- [Brent Roose](https://github.com/brendt)
- [All Contributors](../../contributors)

License
-------

[](#license)

The MIT License (MIT). Please see [License File](LICENSE.md) for more information.

###  Health Score

16

—

LowBetter than 5% of packages

Maintenance20

Infrequent updates — may be unmaintained

Popularity3

Limited adoption so far

Community7

Small or concentrated contributor base

Maturity30

Early-stage or recently created project

 Bus Factor1

Top contributor holds 100% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

### Community

Maintainers

![](https://www.gravatar.com/avatar/72035d725e2fd70fea069899f7ef801510a2d31cc64f6a0b0b699d4b32e150a9?d=identicon)[sonhb](/maintainers/sonhb)

---

Top Contributors

[![sonhb1231](https://avatars.githubusercontent.com/u/41715305?v=4)](https://github.com/sonhb1231 "sonhb1231 (4 commits)")

### Embed Badge

![Health badge](/badges/sonhb99-robots-txt/health.svg)

```
[![Health](https://phpackages.com/badges/sonhb99-robots-txt/health.svg)](https://phpackages.com/packages/sonhb99-robots-txt)
```

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)