PHPackages                             wizardcompass/robots-txt-parser - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Utility & Helpers](/categories/utility)
4. /
5. wizardcompass/robots-txt-parser

ActiveLibrary[Utility & Helpers](/categories/utility)

wizardcompass/robots-txt-parser
===============================

A PHP library to parse and analyze robots.txt files with validation and URL fetching capabilities

v1.0.2(7mo ago)0934↓20.6%MITPHPPHP >=8.1CI failing

Since Sep 22Pushed 7mo agoCompare

[ Source](https://github.com/wizardcompass/robots-txt-parser)[ Packagist](https://packagist.org/packages/wizardcompass/robots-txt-parser)[ RSS](/packages/wizardcompass-robots-txt-parser/feed)WikiDiscussions main Synced 1mo ago

READMEChangelog (3)Dependencies (4)Versions (4)Used By (0)

Robots.txt Parser for PHP
=========================

[](#robotstxt-parser-for-php)

[![PHP Version](https://camo.githubusercontent.com/7663c9d53dc13cedaf0660a8745a7e77d2dd711257f36aa86ebce12a0600ef42/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f7068702d253345253344382e312d626c75652e737667)](https://php.net/)[![Latest Version on Packagist](https://camo.githubusercontent.com/64ba5faaa61b04d4dad6872ed0ce2ca6f20c41f655d6370cbe4acd9d72f05645/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f762f7370617469652f726f626f74732d7478742d7061727365722e7376673f7374796c653d666c61742d737175617265)](https://packagist.org/packages/wizardcompass/robots-txt-parser)[![License: MIT](https://camo.githubusercontent.com/fdf2982b9f5d7489dcf44570e714e3a15fce6253e0cc6b5aa61a075aac2ff71b/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4c6963656e73652d4d49542d79656c6c6f772e737667)](https://opensource.org/licenses/MIT)

A comprehensive PHP library for parsing and analyzing robots.txt files. This package provides functionality to fetch, parse, and validate robots.txt files with support for large files and streaming downloads.

Features
--------

[](#features)

- 🚀 **Fast parsing** of robots.txt content with detailed statistics
- 🌐 **Smart URL fetching** with automatic robots.txt path resolution
- 🔄 **HTTP streaming** with Guzzle for reliable downloads and size limits
- ✅ **Validation** with detailed error and warning reporting
- 📊 **Comprehensive analysis** including directive counts by type and user agent
- 🛡️ **Size protection** with Google's 500KB limit enforcement
- 🔧 **Timeout handling** and redirect support
- 📈 **Performance optimized** for large files (tested up to 5MB+)

Installation
------------

[](#installation)

Install via Composer:

```
composer require wizardcompass/robots-txt-parser
```

Requirements
------------

[](#requirements)

- PHP 8.1 or higher
- Guzzle HTTP 7.0+ (for URL fetching)

Quick Start
-----------

[](#quick-start)

```