PHPackages                             vipnytt/robotstxtparser - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Parsing & Serialization](/categories/parsing)
4. /
5. vipnytt/robotstxtparser

ActiveLibrary[Parsing & Serialization](/categories/parsing)

vipnytt/robotstxtparser
=======================

Robots.txt parsing library, with full support for every directive and specification.

v2.1.0(5y ago)27754.6k↓36.8%6[1 issues](https://github.com/VIPnytt/RobotsTxtParser/issues)7MITPHPPHP ^7.3 || ^8.0

Since Apr 24Pushed 5y ago2 watchersCompare

[ Source](https://github.com/VIPnytt/RobotsTxtParser)[ Packagist](https://packagist.org/packages/vipnytt/robotstxtparser)[ Docs](https://github.com/VIPnytt/RobotsTxtParser)[ RSS](/packages/vipnytt-robotstxtparser/feed)WikiDiscussions master Synced 1mo ago

READMEChangelog (10)Dependencies (3)Versions (13)Used By (7)

[![Build Status](https://camo.githubusercontent.com/6427aa8138d5cc813ba79baa2ff450241fbd2e350f58ccce7ede4306d554abc4/68747470733a2f2f7472617669732d63692e6f72672f5649506e7974742f526f626f74735478745061727365722e7376673f6272616e63683d6d6173746572)](https://travis-ci.org/VIPnytt/RobotsTxtParser)[![Scrutinizer Code Quality](https://camo.githubusercontent.com/9520f03ed7068e1a6fe3ec88ff4611f186d6958be636c1dfdcce37904f2dfdaf/68747470733a2f2f7363727574696e697a65722d63692e636f6d2f672f5649506e7974742f526f626f74735478745061727365722f6261646765732f7175616c6974792d73636f72652e706e673f623d6d6173746572)](https://scrutinizer-ci.com/g/VIPnytt/RobotsTxtParser/?branch=master)[![Maintainability](https://camo.githubusercontent.com/ebbd00c71dd1f562387546b62516576fda246be8e10975b0ddf1c6c7b7537ccd/68747470733a2f2f6170692e636f6465636c696d6174652e636f6d2f76312f6261646765732f66306565616438623431353030393531313264612f6d61696e7461696e6162696c697479)](https://codeclimate.com/github/VIPnytt/RobotsTxtParser/maintainability)[![Test Coverage](https://camo.githubusercontent.com/57d581015e94cf43335c6354f313c1820e685c2e936c25311a73d5cf7aca49d9/68747470733a2f2f6170692e636f6465636c696d6174652e636f6d2f76312f6261646765732f66306565616438623431353030393531313264612f746573745f636f766572616765)](https://codeclimate.com/github/VIPnytt/RobotsTxtParser/test_coverage)[![License](https://camo.githubusercontent.com/2b210868a56775b76101e0af32a715c0ab299e8fe3fc1f6e055474c100ed108d/68747470733a2f2f706f7365722e707567782e6f72672f5649506e7974742f526f626f74735478745061727365722f6c6963656e7365)](https://github.com/VIPnytt/RobotsTxtParser/blob/master/LICENSE)[![Packagist](https://camo.githubusercontent.com/fd95e74c35ec3cadfd0599cd67a02b7deaa6366a7ac037b13ba49b1e6bf60391/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f762f7669706e7974742f726f626f74737478747061727365722e737667)](https://packagist.org/packages/vipnytt/robotstxtparser)[![Gitter](https://camo.githubusercontent.com/f9eb3d6aa6566a16975e6c26abeacab936ee849b38ecc09772ff86d366270597/68747470733a2f2f6261646765732e6769747465722e696d2f5649506e7974742f526f626f74735478745061727365722e737667)](https://gitter.im/VIPnytt/RobotsTxtParser)

Robots.txt parser
=================

[](#robotstxt-parser)

An easy to use, extensible `robots.txt` parser library with *full* support for literally every [directive](#directives) and [specification](#specifications) *on the Internet*.

[![SensioLabsInsight](https://camo.githubusercontent.com/bf6ab958c029b5ebc3b6b1dc30659067a9951401541402965d3f27e6ba9f1936/68747470733a2f2f696e73696768742e73656e73696f6c6162732e636f6d2f70726f6a656374732f36666234373432372d313636622d343564302d626434312d3430663761363363326230632f6269672e706e67)](https://insight.sensiolabs.com/projects/6fb47427-166b-45d0-bd41-40f7a63c2b0c)

#### Usage cases:

[](#usage-cases)

- Permission checks
- Fetch crawler rules
- Sitemap discovery
- Host preference
- Dynamic URL parameter discovery
- `robots.txt` rendering

### Advantages

[](#advantages)

*(compared to most other robots.txt libraries)*

- Automatic `robots.txt` download. (optional)
- Integrated [Caching system](https://github.com/VIPnytt/RobotsTxtParser/blob/master/docs/sql/cache.md). (optional)
- Crawl [Delay handler](https://github.com/VIPnytt/RobotsTxtParser/blob/master/docs/sql/delay.md).
- [Documentation](https://github.com/VIPnytt/RobotsTxtParser/tree/master/docs) available.
- Support for literally [every single directive](#directives), from [every specification](#specifications).
- HTTP Status code handler, *according to [Google](https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt)'s spec.*
- Dedicated `User-Agent` parser and group determiner library, for maximum accuracy.
- Provides additional data like *preferred host*, dynamic *URL parameters*, *Sitemap* locations, etc.
- Protocols supported: `HTTP`, `HTTPS`, `FTP`, `SFTP` and `FTP/S`.

#### Requirements:

[](#requirements)

- PHP 7.3+ or 8.0+
- PHP extensions:
    - [cURL](http://php.net/manual/en/book.curl.php)
    - [mbstring](http://php.net/manual/en/book.mbstring.php)

Installation
------------

[](#installation)

The recommended way to install the robots.txt parser is through [Composer](http://getcomposer.org). Add this to your `composer.json` file:

```
{
  "require": {
    "vipnytt/robotstxtparser": "^2.1"
  }
}
```

Then run: `php composer update`

Getting started
---------------

[](#getting-started)

### Basic usage example

[](#basic-usage-example)

```