PHPackages                             webignition/robots-txt-file - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Parsing & Serialization](/categories/parsing)
4. /
5. webignition/robots-txt-file

ActiveLibrary[Parsing & Serialization](/categories/parsing)

webignition/robots-txt-file
===========================

Models a robots.txt file

3.0(6y ago)181.3M↓38.5%3[3 issues](https://github.com/webignition/robots-txt-file/issues)[1 PRs](https://github.com/webignition/robots-txt-file/pulls)9MITPHPPHP >=7.2.0CI passing

Since Sep 23Pushed 3mo ago1 watchersCompare

[ Source](https://github.com/webignition/robots-txt-file)[ Packagist](https://packagist.org/packages/webignition/robots-txt-file)[ Docs](https://github.com/webignition/robots-txt-file)[ RSS](/packages/webignition-robots-txt-file/feed)WikiDiscussions master Synced 1mo ago

READMEChangelog (5)Dependencies (4)Versions (9)Used By (9)

robots-txt-file [![Build Status](https://camo.githubusercontent.com/4f760811987f5b933138a3b4ba66d189645e4bce2daba7a970a0eb31f273c35f/68747470733a2f2f7365637572652e7472617669732d63692e6f72672f77656269676e6974696f6e2f726f626f74732d7478742d66696c652e706e673f6272616e63683d6d6173746572)](http://travis-ci.org/webignition/robots-txt-file)
=============================================================================================================================================================================================================================================================================================================================================

[](#robots-txt-file-)

- [Introduction](#introduction)
    - [Overview](#overview)
    - [Robots.txt file format refresher](#robots.txt-file-format-refresher)
- [Usage](#usage)
    - [Parsing a robots.txt file from a string into a model](#parsing-a-robots.txt-file-from-a-string-into-a-model)
    - [Inspecting a model to get directives for a user agent](#inspecting-a-model-to-get-directives-for-a-user-agent)
    - [Check if a user agent is allowed to access a url path](#check-if-a-user-agent-is-allowed-to-access-a-url-path)
    - [Extract sitemap URLs](#extract-sitemap-urls)
    - [Filtering directives for a user agent to a specific field type](#filtering-directives-for-a-user-agent-to-a-specific-field-type)
- [Building](#building)
    - [Using as a library in a project](#using-as-a-library-in-a-project)
    - [Developing](#developing)
    - [Testing](#testing)

Introduction
------------

[](#introduction)

### Overview

[](#overview)

Handles [robots.txt](http://www.robotstxt.org/) files:

- parse a robots.txt file into a model
- get directives for a user agent
- check if a user agent is allowed to access a url path
- extract sitemap URLs
- programmatically create a model and cast to a string

### Robots.txt file format refresher

[](#robotstxt-file-format-refresher)

Let's quickly go over the format of a robots.txt file so that you can understand what you can get out of a `\webignition\RobotsTxt\File\File` object.

A robots.txt file contains a collection of **records**. A record provides a set of **directives** to a specified user agent. A directive instructs a user agent to do something (or not do something). A blank line is used to separate records.

Here's an example with two records:

```
User-agent: Slurp
Disallow: /

User-Agent: *
Disallow: /private

```

This instructs the user agent 'Slurp' that it is not allowed access to '/' (i.e. the whole site), and this instructs all other user agents that they are not allowed access to '/private'.

A robots.txt file can optionally contain directives that apply to all user agents irrespective of the specified records. These are included as a set of a directives that are not part of a record. A common use is the `sitemap` directive.

Here's an example with directives that apply to everyone and everything:

```
User-agent: Slurp
Disallow: /

User-Agent: *
Disallow: /private

Sitemap: http://example.com/sitemap.xml

```

Usage
-----

[](#usage)

### Parsing a robots.txt file from a string into a model

[](#parsing-a-robotstxt-file-from-a-string-into-a-model)

```