PHPackages bopoda/robots-txt-parser - PHPackages - PHPackages [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register) 1. [Directory](/) 2. / 3. [Parsing & Serialization](/categories/parsing) 4. / 5. bopoda/robots-txt-parser ActiveLibrary[Parsing & Serialization](/categories/parsing) bopoda/robots-txt-parser ======================== PHP Class for parsing robots.txt files according to Google, Yandex specifications. v2.5.0(1y ago)48266.4k—6.1%16[4 issues](https://github.com/bopoda/robots-txt-parser/issues)1MITDIGITAL Command LanguagePHP ^7.4 || ^8.0 || ^8.1 || ^8.2 || ^8.3 || ^8.4CI passing Since Jan 2Pushed 1y ago4 watchersCompare [ Source](https://github.com/bopoda/robots-txt-parser)[ Packagist](https://packagist.org/packages/bopoda/robots-txt-parser)[ Docs](https://github.com/bopoda/robots-txt-parser)[ RSS](/packages/bopoda-robots-txt-parser/feed)WikiDiscussions master Synced 1mo ago READMEChangelog (8)Dependencies (3)Versions (5)Used By (1) robots-txt-parser ================= [](#robots-txt-parser) [![Build Status](https://github.com/bopoda/robots-txt-parser/actions/workflows/php.yml/badge.svg)](https://github.com/bopoda/robots-txt-parser/actions/workflows/php.yml) [RobotsTxtParser](https://github.com/bopoda/robots-txt-parser/blob/master/src/RobotsTxtParser/RobotsTxtParser.php) — PHP class for parsing all the directives of the robots.txt files ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [](#robotstxtparser--php-class-for-parsing-all-the-directives-of-the-robotstxt-files) [RobotsTxtValidator](https://github.com/bopoda/robots-txt-parser/blob/master/src/RobotsTxtParser/RobotsTxtValidator.php) — PHP class for check is url allow or disallow according to robots.txt rules. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [](#robotstxtvalidator--php-class-for-check-is-url-allow-or-disallow-according-to-robotstxt-rules) Try [demo](http://robots.jeka.by/) of RobotsTxtParser on-line on live domains. Parsing is carried out according to the rules in accordance with Google & Yandex specifications: - [Google: Robots.txt Specifications](https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt) - [Yandex: Robots.txt Specifications](https://help.yandex.com/webmaster/controlling-robot/robots-txt.xml) ### Last improvements: [](#last-improvements) 1. Pars the Clean-param directive according to the clean-param syntax. 2. Deleting comments (everything following the '#' character, up to the first line break, is disregarded) 3. The improvement of the Parse of Host — the intersection directive, should refer to the user-agent '\*'; If there are multiple hosts, the search engines take the value of the first. 4. From the class, unused methods are removed, refactoring done, the scope of properties of the class is corrected. 5. Added more test cases, as well as test cases added to the whole new functionality. 6. [RobotsTxtValidator](https://github.com/bopoda/robots-txt-parser/blob/master/src/RobotsTxtParser/RobotsTxtValidator.php) class added to check if url is allowed to parsing. 7. With version 2.0, the speed of RobotsTxtParser was significantly improved. ### Supported Directives: [](#supported-directives) - DIRECTIVE\_ALLOW = 'allow'; - DIRECTIVE\_DISALLOW = 'disallow'; - DIRECTIVE\_HOST = 'host'; - DIRECTIVE\_SITEMAP = 'sitemap'; - DIRECTIVE\_USERAGENT = 'user-agent'; - DIRECTIVE\_CRAWL\_DELAY = 'crawl-delay'; - DIRECTIVE\_CLEAN\_PARAM = 'clean-param'; - DIRECTIVE\_NOINDEX = 'noindex'; Installation ------------ [](#installation) Install the latest version with ``` composer require bopoda/robots-txt-parser ``` Run tests --------- [](#run-tests) Run phpunit tests using command ``` php vendor/bin/phpunit ``` ### Usage example [](#usage-example) You can start the parser by getting the content of a robots.txt file from a website: ``` $parser = new RobotsTxtParser(file_get_contents('http://example.com/robots.txt')); var_dump($parser->getRules()); ``` Or simply using the contents of the file as input (ie: when the content is already cached): ``` $parser = new RobotsTxtParser(" User-Agent: * Disallow: /ajax Disallow: /search Clean-param: param1 /path/file.php User-agent: Yahoo Disallow: / Host: example.com Host: example2.com "); var_dump($parser->getRules()); ``` This will output: ``` array(2) { ["*"]=> array(3) { ["disallow"]=> array(2) { [0]=> string(5) "/ajax" [1]=> string(7) "/search" } ["clean-param"]=> array(1) { [0]=> string(21) "param1 /path/file.php" } ["host"]=> string(11) "example.com" } ["yahoo"]=> array(1) { ["disallow"]=> array(1) { [0]=> string(1) "/" } } } ``` In order to validate URL, use the RobotsTxtValidator class: ``` $parser = new RobotsTxtParser(file_get_contents('http://example.com/robots.txt')); $validator = new RobotsTxtValidator($parser->getRules()); $url = '/'; $userAgent = 'MyAwesomeBot'; if ($validator->isUrlAllow($url, $userAgent)) { // Crawl the site URL and do nice stuff } ``` ### Contribution [](#contribution) Feel free to create PR in this repository. Please, follow PSR style. See the list of [contributors](https://github.com/bopoda/robots-txt-parser/graphs/contributors) which participated in this project. ### Final Notes: [](#final-notes) Please use [v2.0](https://github.com/bopoda/robots-txt-parser/releases/tag/2.0)+ version which works by same rules but is more highly performance. ### Health Score 54 — FairBetter than 97% of packages Maintenance46 Moderate activity, may be stable Popularity48 Moderate usage in the ecosystem Community22 Small or concentrated contributor base Maturity81 Battle-tested with a long release history Bus Factor1 Top contributor holds 72% of commits — single point of failure How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window. **Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores. **Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement. **Maturity (30%)** — Project age, version count, PHP version support, and release stability. ### Release Activity Cadence Every ~770 days Total 4 Last Release 382d ago PHP version history (2 changes)v2.3.0PHP >=5.4.0 v2.5.0PHP ^7.4 || ^8.0 || ^8.1 || ^8.2 || ^8.3 || ^8.4 ### Community Maintainers ![](https://www.gravatar.com/avatar/787dfca6375d25b588239257750ff39e83d5e618ffb0aa2932a8d68361b46bf2?d=identicon)[jekaby](/maintainers/jekaby) --- Top Contributors [![bopoda](https://avatars.githubusercontent.com/u/881283?v=4)](https://github.com/bopoda "bopoda (72 commits)")[![LeMoussel](https://avatars.githubusercontent.com/u/1816573?v=4)](https://github.com/LeMoussel "LeMoussel (18 commits)")[![kennylajara](https://avatars.githubusercontent.com/u/10185829?v=4)](https://github.com/kennylajara "kennylajara (3 commits)")[![lyrixx](https://avatars.githubusercontent.com/u/408368?v=4)](https://github.com/lyrixx "lyrixx (3 commits)")[![eugene-yurkevich](https://avatars.githubusercontent.com/u/198213176?v=4)](https://github.com/eugene-yurkevich "eugene-yurkevich (2 commits)")[![tacman](https://avatars.githubusercontent.com/u/619585?v=4)](https://github.com/tacman "tacman (2 commits)") --- Tags googlephprobots-txt-parseryandexgoogleparseryandexrobots.txt ### Code Quality TestsPHPUnit Static AnalysisPsalm Code StylePHP CS Fixer Type Coverage Yes ### Embed Badge ![Health badge](/badges/bopoda-robots-txt-parser/health.svg) ``` [![Health](https://phpackages.com/badges/bopoda-robots-txt-parser/health.svg)](https://phpackages.com/packages/bopoda-robots-txt-parser) ``` ### Alternatives [nikic/php-parser A PHP parser written in PHP 17.4k902.6M1.8k](/packages/nikic-php-parser)[doctrine/lexer PHP Doctrine Lexer parser library that can be used in Top-Down, Recursive Descent Parsers. 11.2k910.8M118](/packages/doctrine-lexer)[erusev/parsedown Parser for Markdown. 15.0k151.8M732](/packages/erusev-parsedown)[league/commonmark Highly-extensible PHP Markdown parser which fully supports the CommonMark spec and GitHub-Flavored Markdown (GFM) 3.0k404.0M702](/packages/league-commonmark)[masterminds/html5 An HTML5 parser and serializer. 1.8k242.8M229](/packages/masterminds-html5)[sabberworm/php-css-parser Parser for CSS Files written in PHP 1.8k191.2M65](/packages/sabberworm-php-css-parser) PHPackages © 2026 [Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)