PHPackages                             diggin/diggin-robotrules - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Parsing & Serialization](/categories/parsing)
4. /
5. diggin/diggin-robotrules

ActiveLibrary[Parsing & Serialization](/categories/parsing)

diggin/diggin-robotrules
========================

parser/handler for Robots Exclusion Protocol (robots.txt and more)

0.10.0(10y ago)71872BSD-3-ClausePHPPHP >=5.3.4

Since Jun 15Pushed 7y ago3 watchersCompare

[ Source](https://github.com/diggin/Diggin_RobotRules)[ Packagist](https://packagist.org/packages/diggin/diggin-robotrules)[ RSS](/packages/diggin-diggin-robotrules/feed)WikiDiscussions master Synced 1w ago

READMEChangelog (2)Dependencies (3)Versions (8)Used By (0)

Diggin\_RobotRules
==================

[](#diggin_robotrules)

PHP parser/handler for Robots Exclusion Protocol (robots.txt and more..)

Master: [![Build Status](https://camo.githubusercontent.com/ee8b09f073387d8d739b0fc2f5243835c94ff2125b548f06701b2dc7c241f7b1/68747470733a2f2f7365637572652e7472617669732d63692e6f72672f64696767696e2f44696767696e5f526f626f7452756c65732e706e673f6272616e63683d6d6173746572)](http://travis-ci.org/diggin/Diggin_RobotRules)[![Coverage Status](https://camo.githubusercontent.com/a8963d5b4e1f0f530eeb694aba6dddce0a21cafd6716e8636a29d69b48145b2a/68747470733a2f2f636f766572616c6c732e696f2f7265706f732f64696767696e2f44696767696e5f526f626f7452756c65732f62616467652e706e67)](https://coveralls.io/r/diggin/Diggin_RobotRules)

Features
--------

[](#features)

- implements

    - \[DONE\] "3.2.2 The Allow and Disallow lines" - as test-case
    - \[DONE\] "4.Examples" as test-case
- passing Nutch's test code [ref](https://github.com/apache/nutch/blob/trunk/src/plugin/lib-http/src/test/org/apache/nutch/protocol/http/api/TestRobotRulesParser.java)

    - \[DONE\] @see tests/Diggin/RobotRules/Imported/NutchTest.php
- parsing & handling html-meta

ToDos
-----

[](#todos)

- handle Crawl-Delay
- sync or testing a little pattern w/ Google Test robots.txt tool
    - [https://www.google.com/webmasters/tools/robots-analysis-ac?hl=en&siteUrl=http://yourdomain](https://www.google.com/webmasters/tools/robots-analysis-ac?hl=en&siteUrl=http://yourdomain)
- rewrite with PHPPEG.(because current preg\* base parser makes difficulty.)
- more test, refactoring on and on..

USAGE
-----

[](#usage)

```