PHPackages                             vipnytt/robotstagparser - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [HTTP & Networking](/categories/http)
4. /
5. vipnytt/robotstagparser

ActiveLibrary[HTTP & Networking](/categories/http)

vipnytt/robotstagparser
=======================

X-Robots-Tag HTTP header parser class

v0.2.1(9y ago)13682MITPHPPHP >=5.6

Since Feb 7Pushed 9y ago1 watchersCompare

[ Source](https://github.com/VIPnytt/RobotsTagParser)[ Packagist](https://packagist.org/packages/vipnytt/robotstagparser)[ Docs](https://github.com/VIPnytt/RobotsTagParser)[ RSS](/packages/vipnytt-robotstagparser/feed)WikiDiscussions master Synced 1mo ago

READMEChangelog (5)Dependencies (4)Versions (6)Used By (0)

[![Build Status](https://camo.githubusercontent.com/39d80f0a32fc5558d2c9158a09e3c29928bc72dca5a553413f84d1e25619b625/68747470733a2f2f7472617669732d63692e6f72672f5649506e7974742f526f626f74735461675061727365722e7376673f6272616e63683d6d6173746572)](https://travis-ci.org/VIPnytt/RobotsTagParser)[![Scrutinizer Code Quality](https://camo.githubusercontent.com/6ec0222488d074942c09c5750602b05abca709c975f0fa919230aa656375ea31/68747470733a2f2f7363727574696e697a65722d63692e636f6d2f672f5649506e7974742f526f626f74735461675061727365722f6261646765732f7175616c6974792d73636f72652e706e673f623d6d6173746572)](https://scrutinizer-ci.com/g/VIPnytt/RobotsTagParser/?branch=master)[![Code Climate](https://camo.githubusercontent.com/1daac6dddfaee4e3c8aa880170fa1a3b8f7d8ab04dde4eb7cf6f5fd792e1edf8/68747470733a2f2f636f6465636c696d6174652e636f6d2f6769746875622f5649506e7974742f526f626f74735461675061727365722f6261646765732f6770612e737667)](https://codeclimate.com/github/VIPnytt/RobotsTagParser)[![Test Coverage](https://camo.githubusercontent.com/414046baa2893a1b6de6d3414fcdd23ce2bb2730ebcb6f429f04eb4005d037a1/68747470733a2f2f636f6465636c696d6174652e636f6d2f6769746875622f5649506e7974742f526f626f74735461675061727365722f6261646765732f636f7665726167652e737667)](https://codeclimate.com/github/VIPnytt/RobotsTagParser/coverage)[![License](https://camo.githubusercontent.com/e2565ee1bc636c3c4225db14f9f5a89a2c44e9d61767a248f3e869fc9e83ba7d/68747470733a2f2f706f7365722e707567782e6f72672f5649506e7974742f526f626f74735461675061727365722f6c6963656e7365)](https://github.com/VIPnytt/RobotsTagParser/blob/master/LICENSE)[![Packagist](https://camo.githubusercontent.com/c815dd7bf7dea9c7191675993e2bff4cf2b6af4d1cc056305da009d64ffb132b/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f762f7669706e7974742f726f626f74737461677061727365722e737667)](https://packagist.org/packages/vipnytt/robotstagparser)[![Gitter](https://camo.githubusercontent.com/b4ce28040631f3d2ef0a2050f984bd4c18f8bfae09ddac46fbd09bc372db1d51/68747470733a2f2f6261646765732e6769747465722e696d2f5649506e7974742f526f626f74735461675061727365722e737667)](https://gitter.im/VIPnytt/RobotsTagParser)

X-Robots-Tag HTTP header parser
===============================

[](#x-robots-tag-http-header-parser)

PHP class to parse X-Robots-Tag HTTP headers according to [Google X-Robots-Tag HTTP header specifications](https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag#using-the-x-robots-tag-http-header).

[![SensioLabsInsight](https://camo.githubusercontent.com/955806faffd64c7fc6ac8f836b8b90bbab2e7e2b3be7ecddc5983cb03744ccc4/68747470733a2f2f696e73696768742e73656e73696f6c6162732e636f6d2f70726f6a656374732f62383961303730662d303764332d343930612d383431612d3061653939353537343135382f6269672e706e67)](https://insight.sensiolabs.com/projects/b89a070f-07d3-490a-841a-0ae995574158)

#### Requirements:

[](#requirements)

- PHP [>=5.6](http://php.net/supported-versions.php)
- PHP [mbstring](http://php.net/manual/en/book.mbstring.php) extension

Note: HHVM support is planned once [facebook/hhvm#4277](https://github.com/facebook/hhvm/issues/4277) is fixed.

Installation
------------

[](#installation)

The library is available via [Composer](https://getcomposer.org). Add this to your `composer.json` file:

```
{
    "require": {
        "vipnytt/robotstagparser": "~0.2"
    }
}
```

Then run `composer update`.

Getting Started
---------------

[](#getting-started)

### Basic example

[](#basic-example)

Get all rules affecting *you*, this includes the following:

- All generic rules
- Rules specific to *your* User-Agent (if there is any)

```
use vipnytt\XRobotsTagParser;

$headers = [
    'X-Robots-Tag: noindex, noodp',
    'X-Robots-Tag: googlebot: noindex, noarchive',
    'X-Robots-Tag: bingbot: noindex, noarchive, noimageindex'
];

$parser = new XRobotsTagParser('myUserAgent', $headers);
$rules = $parser->getRules(); // getRules();
```

#### Use your existing GuzzleHttp request

[](#use-your-existing-guzzlehttp-request)

```
use vipnytt\XRobotsTagParser;
use GuzzleHttp\Client;

$client = new GuzzleHttp\Client();
$response = $client->request('GET', 'http://example.com/');

$parser = new XRobotsTagParser\Adapters\GuzzleHttp($response, 'myUserAgent');
$array = $parser->getRules();
```

#### Provide HTTP headers as an string

[](#provide-http-headers-as-an-string)

```
use vipnytt\XRobotsTagParser;

$string = getRules();
```

### Export all rules

[](#export-all-rules)

Returns an array containing *all* rules for *any* User-Agent.

```
use vipnytt\XRobotsTagParser;

$parser = new XRobotsTagParser('myUserAgent', $headers);
$array = $parser->export();
```

Directives:
-----------

[](#directives)

- `all` - There are no restrictions for indexing or serving.
- `none` - Equivalent to `noindex` and `nofollow`.
- `noindex` - Do not show this page in search results and do not show a "Cached" link in search results.
- `nofollow` - Do not follow the links on this page.
- `noarchive` - Do not show a "Cached" link in search results.
- `nosnippet` - Do not show a snippet in the search results for this page.
- `noodp` - Do not use metadata from the [Open Directory project](http://dmoz.org/) for titles or snippets shown for this page.
- `notranslate` - Do not offer translation of this page in search results.
- `noimageindex` - Do not index images on this page.
- `unavailable_after` - Do not show this page in search results after the specified date/time.

Source: [https://developers.google.com/webmasters/control-crawl-index/docs/robots\_meta\_tag](https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag#valid-indexing--serving-directives)

###  Health Score

26

—

LowBetter than 43% of packages

Maintenance20

Infrequent updates — may be unmaintained

Popularity15

Limited adoption so far

Community9

Small or concentrated contributor base

Maturity52

Maturing project, gaining track record

 Bus Factor1

Top contributor holds 100% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Every ~39 days

Total

5

Last Release

3585d ago

PHP version history (3 changes)0.1-alphaPHP >=5.4.0

v0.1.0PHP >=5.5.0

v0.2.0PHP >=5.6

### Community

Maintainers

![](https://www.gravatar.com/avatar/26994e517317bb178af65b50893ef46ad3a1e6a54fe1fdd8056aece47620a703?d=identicon)[JanPetterMG](/maintainers/JanPetterMG)

---

Top Contributors

[![JanPetterMG](https://avatars.githubusercontent.com/u/11933090?v=4)](https://github.com/JanPetterMG "JanPetterMG (65 commits)")

---

Tags

googlespiderrobots.txtrobotx-robots-tagRobots meta tagweb-crawlerHTTP-headerX-Robots

###  Code Quality

TestsPHPUnit

### Embed Badge

![Health badge](/badges/vipnytt-robotstagparser/health.svg)

```
[![Health](https://phpackages.com/badges/vipnytt-robotstagparser/health.svg)](https://phpackages.com/packages/vipnytt-robotstagparser)
```

###  Alternatives

[vipnytt/robotstxtparser

Robots.txt parsing library, with full support for every directive and specification.

27754.6k7](/packages/vipnytt-robotstxtparser)[t1gor/robots-txt-parser

PHP class to parse robots.txt rules according to Google, Yandex, W3C and The Web Robots Pages specifications.

85494.6k4](/packages/t1gor-robots-txt-parser)[khr/php-mcurl-client

wrap curl client (http client) for PHP 5.3; using php multi curl, parallel request and write asynchronous code

71219.8k6](/packages/khr-php-mcurl-client)[shuber/curl

PHP Wrapper for Curl

311.1M2](/packages/shuber-curl)[serpapi/google-search-results-php

Get Google, Bing, Baidu, Ebay, Yahoo, Yandex, Home depot, Naver, Apple, Duckduckgo, Youtube search results via SerpApi.com

69114.3k](/packages/serpapi-google-search-results-php)[garf/laravel-pinger

Ping search engines about website updates

2422.2k](/packages/garf-laravel-pinger)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)