PHPackages                             crwlr/robots-txt - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Parsing & Serialization](/categories/parsing)
4. /
5. crwlr/robots-txt

ActiveLibrary[Parsing & Serialization](/categories/parsing)

crwlr/robots-txt
================

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

v1.1.2(1y ago)1125.3k↓38.5%21MITPHPPHP ^8.0

Since Nov 7Pushed 4mo ago1 watchersCompare

[ Source](https://github.com/crwlrsoft/robots-txt)[ Packagist](https://packagist.org/packages/crwlr/robots-txt)[ Docs](https://www.crwlr.software/packages/robots-txt)[ GitHub Sponsors](https://github.com/sponsors/otsch)[ RSS](/packages/crwlr-robots-txt/feed)WikiDiscussions main Synced 1mo ago

READMEChangelog (7)Dependencies (6)Versions (8)Used By (1)

[![crwlr.software logo](https://github.com/crwlrsoft/graphics/raw/eee6cf48ee491b538d11b9acd7ee71fbcdbe3a09/crwlr-logo.png)](https://www.crwlr.software)

Robots Exclusion Standard/Protocol Parser
=========================================

[](#robots-exclusion-standardprotocol-parser)

for Web Crawling/Scraping
-------------------------

[](#for-web-crawlingscraping)

Use this library within crawler/scraper programs to parse robots.txt files and check if your crawler user-agent is allowed to load certain paths.

Documentation
-------------

[](#documentation)

You can find the documentation at [crwlr.software](https://www.crwlr.software/packages/robots-txt/getting-started).

Contributing
------------

[](#contributing)

If you consider contributing something to this package, read the [contribution guide (CONTRIBUTING.md)](CONTRIBUTING.md).

###  Health Score

46

—

FairBetter than 93% of packages

Maintenance61

Regular maintenance activity

Popularity36

Limited adoption so far

Community13

Small or concentrated contributor base

Maturity60

Established project with proven stability

 Bus Factor1

Top contributor holds 93.8% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Every ~196 days

Recently: every ~216 days

Total

7

Last Release

476d ago

Major Versions

v0.1.2 → v1.0.02022-09-22

PHP version history (2 changes)v0.1.0PHP ^7.4|^8.0

v1.0.0PHP ^8.0

### Community

Maintainers

![](https://www.gravatar.com/avatar/3074cef6e2926ede6d4c9c39a0cf29e2e86e6927255a17c103114d0a5957e1a7?d=identicon)[crwlr](/maintainers/crwlr)

---

Top Contributors

[![otsch](https://avatars.githubusercontent.com/u/4062813?v=4)](https://github.com/otsch "otsch (30 commits)")[![enricodias](https://avatars.githubusercontent.com/u/32619307?v=4)](https://github.com/enricodias "enricodias (2 commits)")

---

Tags

hacktoberfestrobots-exclusion-protocolrobots-exclusion-standardrobots-txtrobots-txt-parserweb-crawlingweb-scrapingparserprotocolcrawlerstandardbotspiderscraperrobotsscrapingcrwlrrobots.txtbotsexclusion

###  Code Quality

TestsPHPUnit

Static AnalysisPHPStan

Code StylePHP CS Fixer

Type Coverage Yes

### Embed Badge

![Health badge](/badges/crwlr-robots-txt/health.svg)

```
[![Health](https://phpackages.com/badges/crwlr-robots-txt/health.svg)](https://phpackages.com/packages/crwlr-robots-txt)
```

###  Alternatives

[crwlr/crawler

Web crawling and scraping library.

37214.8k2](/packages/crwlr-crawler)[vipnytt/robotstxtparser

Robots.txt parsing library, with full support for every directive and specification.

27754.6k7](/packages/vipnytt-robotstxtparser)[laurentvw/scrapher

A web scraper for PHP to easily extract data from web pages

192.5k1](/packages/laurentvw-scrapher)[vipnytt/sitemapparser

XML Sitemap parser class compliant with the Sitemaps.org protocol.

772.2M10](/packages/vipnytt-sitemapparser)[t1gor/robots-txt-parser

PHP class to parse robots.txt rules according to Google, Yandex, W3C and The Web Robots Pages specifications.

85494.6k4](/packages/t1gor-robots-txt-parser)[bopoda/robots-txt-parser

PHP Class for parsing robots.txt files according to Google, Yandex specifications.

48266.4k1](/packages/bopoda-robots-txt-parser)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)