PHPackages fkrzski/robots-txt - PHPackages - PHPackages [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register) 1. [Directory](/) 2. / 3. [Utility & Helpers](/categories/utility) 4. / 5. fkrzski/robots-txt ActiveLibrary[Utility & Helpers](/categories/utility) fkrzski/robots-txt ================== A modern, fluent PHP package for managing robots.txt rules with type safety and great developer experience 2.1.0(6mo ago)51.6k↓27%1MITPHPPHP ^8.4 | ^8.5CI passing Since Dec 16Pushed 3mo ago1 watchersCompare [ Source](https://github.com/fkrzski/robots-txt)[ Packagist](https://packagist.org/packages/fkrzski/robots-txt)[ GitHub Sponsors](https://github.com/sponsors/fkrzski)[ RSS](/packages/fkrzski-robots-txt/feed)WikiDiscussions master Synced 3w ago READMEChangelog (6)Dependencies (5)Versions (7)Used By (0) [![Fkrzski PHP Package Skeleton](./art/banner.png)](./art/banner.png)[![GitHub branch check runs](https://camo.githubusercontent.com/bf4afd3c4e2d9033bc2ee1108f618481c9a0d1959643893eeeb373d493e924f5/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f636865636b2d72756e732f666b727a736b692f726f626f74732d7478742f6d61737465723f7374796c653d666f722d7468652d6261646765)](https://camo.githubusercontent.com/bf4afd3c4e2d9033bc2ee1108f618481c9a0d1959643893eeeb373d493e924f5/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f636865636b2d72756e732f666b727a736b692f726f626f74732d7478742f6d61737465723f7374796c653d666f722d7468652d6261646765)[![Packagist Downloads](https://camo.githubusercontent.com/312af3e15f1ce2e09706b2bced67b484eda0e0ecfde4ab5e28f6a8c1f216fff3/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f64742f666b727a736b692f726f626f74732d7478743f7374796c653d666f722d7468652d6261646765)](https://camo.githubusercontent.com/312af3e15f1ce2e09706b2bced67b484eda0e0ecfde4ab5e28f6a8c1f216fff3/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f64742f666b727a736b692f726f626f74732d7478743f7374796c653d666f722d7468652d6261646765)[![Packagist Version](https://camo.githubusercontent.com/525d866a39621360c0dc4945899f8482032c51366c95cbc5db7286f1e1224e9d/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f762f666b727a736b692f726f626f74732d7478743f7374796c653d666f722d7468652d6261646765)](https://camo.githubusercontent.com/525d866a39621360c0dc4945899f8482032c51366c95cbc5db7286f1e1224e9d/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f762f666b727a736b692f726f626f74732d7478743f7374796c653d666f722d7468652d6261646765)[![Packagist License](https://camo.githubusercontent.com/e7d26946160cbc0a05c3915b5e381e9476805da36c5f6d6bea8a903109c63ce8/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f6c2f666b727a736b692f726f626f74732d7478743f7374796c653d666f722d7468652d6261646765)](https://camo.githubusercontent.com/e7d26946160cbc0a05c3915b5e381e9476805da36c5f6d6bea8a903109c63ce8/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f6c2f666b727a736b692f726f626f74732d7478743f7374796c653d666f722d7468652d6261646765) --- PHP Robots.txt ============== [](#php-robotstxt) A modern, fluent PHP package for managing robots.txt rules with type safety and great developer experience. Requirements ------------ [](#requirements) - PHP 8.4 or higher - Code coverage driver Installation ------------ [](#installation) You can install the package via composer: ``` composer require fkrzski/robots-txt ``` Documentation ------------- [](#documentation) The `RobotsTxt` class provides a fluent interface for creating and managing robots.txt rules with type safety and immutability. ### Basic Methods [](#basic-methods) #### Constructor [](#constructor) Creates a new instance of the RobotsTxt class. ``` $robots = new RobotsTxt(); ``` Output: ``` // Empty output - no rules defined yet ``` #### allow(string $path) [](#allowstring-path) Adds an Allow rule for the specified path. The path must: - Start with a forward slash (/) - Not contain query parameters - Not contain fragments - Not be empty ``` $robots = new RobotsTxt(); $robots->allow('/public'); ``` Output: ``` User-agent: * Allow: /public ``` #### disallow(string $path) [](#disallowstring-path) Adds a Disallow rule for the specified path. Has the same path requirements as `allow()`. ``` $robots = new RobotsTxt(); $robots->disallow('/private'); ``` Output: ``` User-agent: * Disallow: /private ``` #### crawlDelay(int $seconds) [](#crawldelayint-seconds) Sets the crawl delay in seconds. The delay value must be non-negative. ``` $robots = new RobotsTxt(); $robots->crawlDelay(10); ``` Output: ``` User-agent: * Crawl-delay: 10 ``` #### sitemap(string $url) [](#sitemapstring-url) Adds a Sitemap URL. The URL must: - Be a valid URL - Use HTTP or HTTPS protocol - Have an .xml extension ``` $robots = new RobotsTxt(); $robots->sitemap('https://example.com/sitemap.xml'); ``` Output: ``` Sitemap: https://example.com/sitemap.xml ``` #### disallowAll(bool $disallow = true) [](#disallowallbool-disallow--true) A convenience method for quickly blocking access to the entire site. When `$disallow` is true (default): - Clears all existing rules in the current context (global or user-agent specific) - Adds a single "Disallow: /\*" rule - Preserves sitemap entries and rules for other user agents ``` // Block everything globally $robots = new RobotsTxt(); $robots ->allow('/public') // This will be cleared ->disallow('/admin') // This will be cleared ->disallowAll(); // Only Disallow: /* remains ``` Output: ``` User-agent: * Disallow: /* ``` Block access only for specific crawler: ``` $robots = new RobotsTxt(); $robots ->disallow('/admin') // Global rule - keeps ->userAgent(CrawlerEnum::GOOGLE) ->allow('/public') // Google rule - cleared ->disallow('/private') // Google rule - cleared ->disallowAll() // Only Disallow: /* for Google ->userAgent(CrawlerEnum::BING) ->disallow('/secret'); // Bing rule - keeps ``` Output: ``` User-agent: * Disallow: /admin User-agent: Googlebot Disallow: /* User-agent: Bingbot Disallow: /secret ``` When `$disallow` is false, the method does nothing. #### userAgent(CrawlerEnum $crawler) [](#useragentcrawlerenum-crawler) Sets the context for subsequent rules to apply to a specific crawler. ``` $robots = new RobotsTxt(); $robots->userAgent(CrawlerEnum::GOOGLE); ``` Output: ``` User-agent: Googlebot ``` ### Combining Methods [](#combining-methods) #### Basic Rule Combinations [](#basic-rule-combinations) You can chain multiple rules together: ``` $robots = new RobotsTxt(); $robots ->disallow('/admin') ->allow('/public') ->crawlDelay(5); ``` Output: ``` User-agent: * Disallow: /admin Allow: /public Crawl-delay: 5 ``` #### Crawler-Specific Rules [](#crawler-specific-rules) You can set rules for specific crawlers: ``` $robots = new RobotsTxt(); $robots ->userAgent(CrawlerEnum::GOOGLE) ->disallow('/private') ->allow('/public') ->crawlDelay(10); ``` Output: ``` User-agent: Googlebot Disallow: /private Allow: /public Crawl-delay: 10 ``` #### Multiple Crawlers [](#multiple-crawlers) You can define rules for multiple crawlers: ``` $robots = new RobotsTxt(); $robots ->userAgent(CrawlerEnum::GOOGLE) ->disallow('/google-private') ->userAgent(CrawlerEnum::BING) ->disallow('/bing-private'); ``` Output: ``` User-agent: Googlebot Disallow: /google-private User-agent: Bingbot Disallow: /bing-private ``` #### Using forUserAgent() [](#using-foruseragent) The `forUserAgent()` method provides a closure-based syntax for grouping crawler-specific rules: ``` $robots = new RobotsTxt(); $robots->forUserAgent(CrawlerEnum::GOOGLE, function (RobotsTxt $robots): void { $robots ->disallow('/private') ->allow('/public') ->crawlDelay(10); }); ``` Output: ``` User-agent: Googlebot Disallow: /private Allow: /public Crawl-delay: 10 ``` #### Complex Example [](#complex-example) Combining global rules, multiple crawlers, and sitemaps: ``` $robots = new RobotsTxt(); $robots ->disallow('/admin') // Global rule ->sitemap('https://example.com/sitemap1.xml') ->forUserAgent(CrawlerEnum::GOOGLE, function (RobotsTxt $robots): void { $robots ->disallow('/google-private') ->allow('/public/*'); }) ->forUserAgent(CrawlerEnum::BING, function (RobotsTxt $robots): void { $robots ->disallow('/bing-private') ->crawlDelay(5); }) ->sitemap('https://example.com/sitemap2.xml'); ``` Output: ``` User-agent: * Disallow: /admin User-agent: Googlebot Disallow: /google-private Allow: /public/* User-agent: Bingbot Disallow: /bing-private Crawl-delay: 5 Sitemap: https://example.com/sitemap1.xml Sitemap: https://example.com/sitemap2.xml ``` ### Working with Wildcards [](#working-with-wildcards) The library supports wildcards in paths: ``` $robots = new RobotsTxt(); $robots ->disallow('/*.php') // Block all PHP files ->allow('/public/*') // Allow all under public ->disallow('/private/$'); // Exact match for /private/ ``` Output: ``` User-agent: * Disallow: /*.php Allow: /public/* Disallow: /private/$ ``` #### toFile(?string $path = null) [](#tofilestring-path--null) Saves the robots.txt content to a file. If no path is provided, saves to `robots.txt` in the project root directory. ``` $robots = new RobotsTxt(); $robots ->disallow('/admin') ->allow('/public'); // Save to default location (project root) $robots->toFile(); // Save to custom location $robots->toFile('/var/www/html/robots.txt'); ``` The method will throw a `RuntimeException` if: - The target directory doesn't exist or isn't writable - The existing robots.txt file isn't writable Returns `true` if the file was successfully written. ### Best Practices [](#best-practices) 1. **Start with Global Rules**: Define global rules before crawler-specific rules for better organization. 2. **Group Related Rules**: Use the `forUserAgent()` method to group rules for the same crawler. 3. **Use Wildcards Carefully**: Be precise with wildcard patterns to avoid unintended matches. 4. **Order Matters**: More specific rules should come before more general ones. 5. **Validate Paths**: Always ensure paths start with a forward slash and don't contain query parameters or fragments. ### Error Handling [](#error-handling) The class will throw `InvalidArgumentException` in the following cases: - Path doesn't start with forward slash - Path contains query parameters or fragments - Path is empty - Sitemap URL is invalid or not HTTP/HTTPS - Sitemap URL doesn't end with .xml - Crawl delay is negative These validations ensure that the generated robots.txt file is always valid and follows the standard format. Testing and Development ----------------------- [](#testing-and-development) The project includes several command groups for testing and code quality: ### Code Quality & Formatting [](#code-quality--formatting) ``` # Run profanity checks composer test:profanity # Run static analysis with PHPStan composer analyse # Format code with Laravel Pint composer lint # Check code formatting (without fixing) composer test:lint # Run automated refactoring with Rector composer refactor # Check refactor suggestions (dry-run) composer test:refactor # Check type coverage (100% required) composer test:type-coverage ``` ### Testing [](#testing) ``` # Check PHP syntax composer test:syntax # Run unit tests with coverage composer test:unit # Run mutation testing composer test:unit:mutation ``` ### Complete Test Suite [](#complete-test-suite) ``` # Run all tests and quality checks composer test ``` 🀝 Contributing -------------- [](#-contributing) We welcome contributions! Please see our [Contributing Guide](.github/CONTRIBUTING.md) for details on: - Setting up the development environment - Running tests - Submitting pull requests - Code style guidelines - Reporting issues ### Quick Contribution Setup [](#quick-contribution-setup) 1. Fork this repository 2. Clone your fork: `git clone https://github.com/yourusername/php-package-skeleton.git` 3. Install dependencies: `composer install` 4. Create a feature branch: `git checkout -b feature/amazing-feature` 5. Make your changes and run tests: `composer test` 6. Submit a pull request πŸ“œ License --------- [](#-license) This project is open-sourced software licensed under the [MIT License](LICENSE.md). πŸ‘¨β€πŸ’» Author ---------- [](#‍-author) **PHP Package Skeleton** was created by [Filip KrzyΕΌanowski](https://linkedin.com/in/fkrzski). πŸ™ Acknowledgments ----------------- [](#-acknowledgments) Special thanks to the amazing PHP community and the maintainers of: - [Laravel Pint](https://laravel.com/docs/pint) - [PEST PHP](https://pestphp.com/) - [PHPStan](https://phpstan.org/) - [Rector](https://getrector.org/) --- **Happy coding! πŸš€** ### Health Score 46 β€” FairBetter than 92% of packages Maintenance75 Regular maintenance activity Popularity26 Limited adoption so far Community10 Small or concentrated contributor base Maturity60 Established project with proven stability Bus Factor1 Top contributor holds 98.6% of commits β€” single point of failure How is this calculated?**Maintenance (25%)** β€” Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window. **Popularity (30%)** β€” Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores. **Community (15%)** β€” Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement. **Maturity (30%)** β€” Project age, version count, PHP version support, and release stability. ### Release Activity Cadence Every ~90 days Total 5 Last Release 193d ago Major Versions 1.0.2 β†’ 2.0.02025-01-28 PHP version history (3 changes)1.0.1PHP ^8.2 2.0.0PHP ^8.3 2.1.0PHP ^8.4 | ^8.5 ### Community Maintainers ![](https://www.gravatar.com/avatar/accf44fd0ffa5349a96edefe2daf51dc4475af31a01611acade1d44b35341d61?d=identicon)[fkrzski](/maintainers/fkrzski) --- Top Contributors [![fkrzski](https://avatars.githubusercontent.com/u/75097934?v=4)](https://github.com/fkrzski "fkrzski (68 commits)")[![phpcon](https://avatars.githubusercontent.com/u/116893932?v=4)](https://github.com/phpcon "phpcon (1 commits)") --- Tags phprobots-txtseoseotoolscrawlerseorobotsphp8robots.txtweb-crawler ### Code Quality TestsPest Static AnalysisPHPStan, Rector Code StyleLaravel Pint Type Coverage Yes ### Embed Badge ![Health badge](/badges/fkrzski-robots-txt/health.svg) ``` [![Health](https://phpackages.com/badges/fkrzski-robots-txt/health.svg)](https://phpackages.com/packages/fkrzski-robots-txt) ``` ### Alternatives [spatie/laravel-robots-middleware Add an `all` or `none` robots header to your requests via a middleware in Laravel 3332.2M5](/packages/spatie-laravel-robots-middleware)[melbahja/seo SEO library for PHP is a simple PHP library to help developers 🍻 do better on-page SEO optimizations. 358153.6k7](/packages/melbahja-seo)[novactive/ezseobundle Novactive eZ SEO Bundle is an Ibexa Platform bundle for SEO simplications. metas, sitemaps, robots.txt, etc. 26256.6k3](/packages/novactive-ezseobundle)[mad-web/laravel-robots Robots.txt generator service. 33126.9k](/packages/mad-web-laravel-robots)[calotype/seo A package containing SEO helpers. 732.6k](/packages/calotype-seo) PHPackages Β© 2026 [Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)