PHPackages                             tomverran/robots-txt-checker - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Utility & Helpers](/categories/utility)
4. /
5. tomverran/robots-txt-checker

ActiveLibrary[Utility & Helpers](/categories/utility)

tomverran/robots-txt-checker
============================

Given a robots.txt file, user agent and URL path will tell you whether you're allowed to access a page

1.17(5y ago)960.9k↓44.8%4[3 PRs](https://github.com/tomverran/robots/pulls)3MITPHPPHP >=5.4.0CI failing

Since Apr 15Pushed 2y ago4 watchersCompare

[ Source](https://github.com/tomverran/robots)[ Packagist](https://packagist.org/packages/tomverran/robots-txt-checker)[ Docs](https://carefulcoder.co.uk)[ RSS](/packages/tomverran-robots-txt-checker/feed)WikiDiscussions master Synced 1mo ago

READMEChangelog (10)Dependencies (2)Versions (16)Used By (3)

Robots.txt checker
==================

[](#robotstxt-checker)

[![Build Status](https://camo.githubusercontent.com/3499c87286e6269c4ffdd4b5fa2bda83020693f46119d54fc3cf608f2800e5d5/68747470733a2f2f7472617669732d63692e6f72672f746f6d76657272616e2f726f626f74732e7376673f6272616e63683d6d6173746572)](https://travis-ci.org/tomverran/robots)[![Packagist](https://camo.githubusercontent.com/fbc9bfc7e99eccbd8fd5f7bf2ce6481e7161d323327f4cc86cbe0418205e44f8/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f762f746f6d76657272616e2f726f626f74732d7478742d636865636b65722e737667)](https://packagist.org/packages/tomverran/robots-txt-checker)[![Test Coverage](https://camo.githubusercontent.com/38357d786773184ef127739565264beff7f3a586211c1cd1c1807abc44c87ee4/68747470733a2f2f636f6465636c696d6174652e636f6d2f6769746875622f746f6d76657272616e2f726f626f74732f6261646765732f636f7665726167652e737667)](https://codeclimate.com/github/tomverran/robots/coverage)[![Code Climate](https://camo.githubusercontent.com/cc01ddf50ae1920a8e707b6200e24a5e95a6fbeb7db0342cad31b7210ab4f650/68747470733a2f2f636f6465636c696d6174652e636f6d2f6769746875622f746f6d76657272616e2f726f626f74732f6261646765732f6770612e737667)](https://codeclimate.com/github/tomverran/robots)

Given a robots.txt file this library will give you a straight forward yes/no as to whether you're allowed to access a given resource with a given user agent.

This library has been built with reference to both [Google's robots.txt specification](https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt) and the draft [norobots RFC](http://www.robotstxt.org/norobots-rfc.txt) from 1996(!). As such all the following features are supported:

- Wildcards in paths, including across directory boundaries
- The '$' suffix to anchor a path match to the end of a URL
- Sorting of multiple matching user-agent/path blocks by user-agent length
- Decoding of URL encoded paths, with the exception of forward slashes
- A trailing slash on a path indicating that all files under that directory should match

If you find any bugs with this library please don't hesitate to let me know, either create an issue on GitHub or submit a pull request.

Installation
------------

[](#installation)

Either type the following into your terminal:

```
composer require tomverran/robots-txt-checker
```

or add the following to your composer.json:

```
"require": {
    "tomverran/robots-txt-checker": "^1.15"
}
```

Example Usage
-------------

[](#example-usage)

I personally use this library alongside an http client library such that all requests go through a class that checks the site's robots.txt first. Basic usage is as follows:

```