PHPackages angeo/module-robots-txt-aeo - PHPackages - PHPackages [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register) 1. [Directory](/) 2. / 3. [Utility & Helpers](/categories/utility) 4. / 5. angeo/module-robots-txt-aeo ActiveMagento2-module[Utility & Helpers](/categories/utility) angeo/module-robots-txt-aeo =========================== Magento 2 module for AI Engine Optimization (AEO). Injects AI crawler rules (OAI-SearchBot, GPTBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, ClaudeBot, anthropic-ai, Claude-User, Applebot, cohere-ai, Amazonbot, Meta-ExternalAgent) into robots.txt — without overwriting your existing configuration. Supports per-bot Allow/Disallow lists, Crawl-delay, Sitemap directives, multi-store, and a public Api\\RobotsStatusInterface for cross-module integration with angeo/module-aeo-audit. 2.0.0(1w ago)15↑650%MITPHPPHP ~8.1.0||~8.2.0||~8.3.0||~8.4.0 Since Apr 25Pushed 1w agoCompare [ Source](https://github.com/angeo-dev/module-robots-txt-aeo)[ Packagist](https://packagist.org/packages/angeo/module-robots-txt-aeo)[ Docs](https://angeo.dev)[ RSS](/packages/angeo-module-robots-txt-aeo/feed)WikiDiscussions main Synced 1w ago READMEChangelogDependencies (7)Versions (4)Used By (0) Angeo Robots.txt AEO — AI Crawler Rules for Magento 2 ===================================================== [](#angeo-robotstxt-aeo--ai-crawler-rules-for-magento-2) [![Packagist](https://camo.githubusercontent.com/bb3d67e8c8226e1d3bb817d40fb45e96b56fa60ee1f0fa76cc79de3f55fb1381/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f762f616e67656f2f6d6f64756c652d726f626f74732d7478742d61656f2e737667)](https://packagist.org/packages/angeo/module-robots-txt-aeo)[![License](https://camo.githubusercontent.com/7013272bd27ece47364536a221edb554cd69683b68a46fc0ee96881174c4214c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d4d49542d626c75652e737667)](LICENSE)[![PHP](https://camo.githubusercontent.com/ff80bec985a3d59c06f94022276b5c38bfd4419ee49efdcfd662bf698169b453/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f7068702d382e31253230253743253230382e32253230253743253230382e33253230253743253230382e342d3838393242462e737667)](https://php.net)[![Magento](https://camo.githubusercontent.com/7230ded7e061dba1a4eec8e1d348e93edbdc708e43487a2e0a8e548dd8c541c6/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6d6167656e746f2d322e342e36253230253743253230322e342e37253230253743253230322e342e382d4545363732462e737667)](https://magento.com) Injects AI crawler rules into your Magento 2 `robots.txt` — **without overwriting your existing configuration**. Bots managed out-of-the-box: `OAI-SearchBot`, `GPTBot`, `ChatGPT-User`, `PerplexityBot`, `Perplexity-User`, `Google-Extended`, `ClaudeBot`, `anthropic-ai`, `Claude-User`, `Applebot`, `cohere-ai`, `Amazonbot`, `Meta-ExternalAgent`. Fixes the **"robots.txt — AI Bot Access"** signal in [`angeo/module-aeo-audit`](https://packagist.org/packages/angeo/module-aeo-audit). --- What's new in 2.0 ----------------- [](#whats-new-in-20) - **5 new built-in bots** aligned with the AEO Audit v3 catalogue: `Claude-User`, `Applebot`, `cohere-ai`, `Amazonbot`, `Meta-ExternalAgent`. An out-of-the-box install now passes the AEO Audit's `robots_txt` check. - **Audit-clean output** — emitted robots.txt no longer triggers syntax warnings: - `Crawl-delay` suppressed on bots that ignore it (GPTBot, ClaudeBot, Google-Extended). - No `Allow: /` + `Disallow: /` conflict on the same agent. - Versioned UAs sanitised at the catalogue layer. - Sitemap URLs upgraded to `https://` when the store base URL is HTTPS. - **`Api\RobotsStatusInterface`** — public read-only API for cross-module integration. Consumers like `angeo/module-aeo-audit` can wire to it and skip the HTTP round-trip. - **Dedicated cache type** `angeo_robots_txt_aeo` — flush in isolation from System → Cache Management. - **Backend validation** — `PathList` and `CrawlDelay` backend models normalise admin input on save. - **CSP-clean admin UI** — no inline styles, no inline scripts. - **i18n/en\_US.csv** — admin labels are translatable. - **Removed runtime remote-registry feature** — bot catalogue is now release-managed only. Dynamic catalogue injection from an external endpoint was a security trade-off (anyone with the endpoint could inject UA strings into every install's robots.txt) and a half-implemented UX one (added bots had no admin checkbox). New bots ship via module releases. - **Removed orphan code** — the unused `RemoteRegistryUpdater` triplet from 1.x is gone. See [CHANGELOG.md](CHANGELOG.md) for the full list. --- How it works ------------ [](#how-it-works) The module intercepts the robots.txt response at render time via a plugin on `Magento\Robots\Model\Robots::getData()` and prepends a managed block of AI bot rules. **No database writes. No filesystem changes.** Your existing admin config is untouched. ### Inject mode (default — recommended) [](#inject-mode-default--recommended) ``` # Angeo AEO — AI Crawler Rules # https://angeo.dev | module-robots-txt-aeo # Do not edit this block manually — manage via Stores > Config > Angeo > Robots.txt AEO User-agent: OAI-SearchBot Allow: / User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / Disallow: /admin/ User-agent: Claude-User Allow: / User-agent: Applebot Allow: / # End Angeo AEO block User-agent: * Disallow: /checkout/ ... (your existing rules follow unchanged) # Angeo AEO — Sitemaps Sitemap: https://example-store.com/sitemap.xml # End Angeo AEO sitemaps ``` ### Replace mode [](#replace-mode) Regenerates the full robots.txt. Preserves your custom `Disallow` rules from the existing wildcard block. Use only if you want this module to own the entire file. --- Installation ------------ [](#installation) ``` composer require angeo/module-robots-txt-aeo bin/magento module:enable Angeo_RobotsTxtAeo bin/magento setup:upgrade bin/magento setup:di:compile bin/magento cache:flush ``` That's it. The module is enabled with sensible defaults — all 10 mainstream AI bots are allowed; the 3 lower-traffic bots (cohere-ai, Amazonbot, Meta-ExternalAgent) are catalogued but disabled by default. --- Configuration ------------- [](#configuration) `Stores → Configuration → Angeo → Robots.txt AEO` SectionPurpose**General**Enable/disable, choose Inject or Replace mode**AI Crawlers**Tick which bots to allow. Bots marked ★ are critical for AEO Audit pass**AI Crawler Path Overrides**Per-bot `Allow:`, `Disallow:`, `Crawl-delay:`**Sitemap Directive**Auto-detect from `Magento_Sitemap`, manual list, or none**Live Preview**Renders the AEO block that will be injectedAll settings respect store scope — multi-store installs can configure each store independently. --- CLI --- [](#cli) ``` # Render what would be emitted, without applying it bin/magento angeo:robots:preview [--store=N] # Fetch the live robots.txt and check enabled bot rules are present bin/magento angeo:robots:validate [--store=N] [--insecure] ``` `validate` exits non-zero when expected bot rules are missing from the live file — useful in post-deploy smoke tests: ``` # .github/workflows/post-deploy.yml - run: bin/magento angeo:robots:validate ``` For a full AEO scoring of robots.txt (critical-bot checks, syntax warnings, sitemap quality) install [`angeo/module-aeo-audit`](https://packagist.org/packages/angeo/module-aeo-audit). It reads the effective output of this module via `Api\RobotsStatusInterface` — no HTTP round-trip when both modules are installed. --- Cross-module integration (Api\\RobotsStatusInterface) ----------------------------------------------------- [](#cross-module-integration-apirobotsstatusinterface) The module exposes a public read-only API that consumer modules can wire to via DI. Soft-coupling pattern — consumers `interface_exists()`-check before declaring the dependency, so they keep working when this module is not installed. ``` use Angeo\RobotsTxtAeo\Api\RobotsStatusInterface; class MyChecker { public function __construct( private readonly ?RobotsStatusInterface $robotsStatus = null, ) {} public function check(int $storeId): void { if ($this->robotsStatus !== null) { // Zero-overhead — pure in-process call $effective = $this->robotsStatus->getEffectiveRobotsTxt($storeId); $bots = $this->robotsStatus->getEnabledBotUserAgents($storeId); // ... } else { // Fall back to HTTP fetch } } } ``` Used by `angeo/module-aeo-audit` v3+ when both modules are installed. --- How robots.txt manual content interacts --------------------------------------- [](#how-robotstxt-manual-content-interacts) The module's admin form (Inject mode) **does not** modify the existing Magento admin robots.txt textarea (Content → Design → Configuration → Edit Custom instruction of robots.txt). Both sources coexist: - Your custom block is preserved untouched. - The AEO block is prepended at render time. - Re-running the plugin is idempotent — the AEO block is replaced, not stacked. If you'd rather manage AI bot rules yourself, either disable the module (`bin/magento module:disable Angeo_RobotsTxtAeo`) or untick individual bots in admin. --- Compatibility ------------- [](#compatibility) StatusMagento 2.4.6 (PHP 8.1)✅Magento 2.4.7 (PHP 8.2 / 8.3)✅Magento 2.4.8 (PHP 8.3 / 8.4)✅Magento Open Source / Commerce / Cloud✅Hyvä / PWA Studio✅ (robots.txt is server-side)Multi-store / multi-website✅`Magento_Sitemap` not installed✅ (soft dependency, no-op resolver)Varnish / Fastly⚠️ purge CDN cache after config changes--- License ------- [](#license) MIT. See [LICENSE](LICENSE). Security -------- [](#security) See [SECURITY.md](SECURITY.md) for the disclosure policy. Contributing ------------ [](#contributing) See [CONTRIBUTING.md](CONTRIBUTING.md). ### Health Score 43 — FairBetter than 89% of packages Maintenance98 Actively maintained with recent releases Popularity7 Limited adoption so far Community2 Small or concentrated contributor base Maturity53 Maturing project, gaining track record How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window. **Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores. **Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement. **Maturity (30%)** — Project age, version count, PHP version support, and release stability. ### Release Activity Cadence Every ~17 days Total 3 Last Release 11d ago Major Versions 1.0.1 → 2.0.02026-05-29 PHP version history (2 changes)1.0.0PHP >=8.2 2.0.0PHP ~8.1.0||~8.2.0||~8.3.0||~8.4.0 ### Community Maintainers ![](https://avatars.githubusercontent.com/u/557748?v=4)[angeo](/maintainers/angeo)[@angeo](https://github.com/angeo) --- Tags aeochatgptmagentomagento2oai-searchbotrobots-txtmagento2ChatGptrobots.txtmagento 2 moduleangeoaeoai-searchai engine optimizationai visibilitygptbotclaudebotperplexitybotoai-searchbotgoogle-extendedapplebotamazonbotmeta-externalagentai crawlers ### Code Quality TestsPHPUnit ### Embed Badge ![Health badge](/badges/angeo-module-robots-txt-aeo/health.svg) ``` [![Health](https://phpackages.com/badges/angeo-module-robots-txt-aeo/health.svg)](https://phpackages.com/packages/angeo-module-robots-txt-aeo) ``` ### Alternatives [run-as-root/magento2-prometheus-exporter Magento2 Prometheus Exporter 68353.9k](/packages/run-as-root-magento2-prometheus-exporter)[loki/magento2-components Core module for defining Alpine.js components with advanced AJAX features 1010.0k22](/packages/loki-magento2-components)[mollie/magento2 Mollie Payment Module for Magento 2 1131.8M12](/packages/mollie-magento2)[mage-os/module-inventory-reservations-grid Add a grid with the list of inventory reservations. 1512.6k](/packages/mage-os-module-inventory-reservations-grid)[mage-os/module-admin-activity-log The Admin Activity extension makes it easy to track all admin activity with comprehensive audit logging. 293.3k](/packages/mage-os-module-admin-activity-log)[opengento/module-category-import-export This module add the capability to import and export the categories from the back-office. 1310.2k1](/packages/opengento-module-category-import-export) PHPackages © 2026 [Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)