PHPackages mage2kishan/module-robots-seo - PHPackages - PHPackages [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register) 1. [Directory](/) 2. / 3. [Utility & Helpers](/categories/utility) 4. / 5. mage2kishan/module-robots-seo ActiveMagento2-module[Utility & Helpers](/categories/utility) mage2kishan/module-robots-seo ============================= Panth Robots SEO — dedicated robots.txt, X-Robots-Tag, and LLM-bot (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.) policy control for Magento 2. Extracted from Panth\_AdvancedSEO. Self-contained: emits a per-store robots.txt, adds X-Robots-Tag response headers, validates robots meta directives, and exposes admin CRUD for user-agent / path policies. Hyva and Luma compatible. 1.0.8(1mo ago)024↓66.7%1proprietaryPHPPHP ~8.1.0||~8.2.0||~8.3.0||~8.4.0 Since Apr 20Pushed 1mo agoCompare [ Source](https://github.com/mage2sk/module-robots-seo)[ Packagist](https://packagist.org/packages/mage2kishan/module-robots-seo)[ Docs](https://kishansavaliya.com)[ RSS](/packages/mage2kishan-module-robots-seo/feed)WikiDiscussions main Synced 3w ago READMEChangelogDependencies (16)Versions (10)Used By (0) Magento 2 Robots SEO Extension: robots.txt, X-Robots-Tag and LLM Bot Policy Control (Hyva + Luma) ================================================================================================= [](#magento-2-robots-seo-extension-robotstxt-x-robots-tag-and-llm-bot-policy-control-hyva--luma) [![Magento 2.4.4 - 2.4.8](https://camo.githubusercontent.com/079c832211eed4f9451ebe264e3865f825b0f9f31b041cbf03676c6e254535d4/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4d6167656e746f2d322e342e342532302d2d253230322e342e382d6f72616e67653f6c6f676f3d6d6167656e746f266c6f676f436f6c6f723d7768697465)](https://magento.com)[![PHP 8.1 - 8.4](https://camo.githubusercontent.com/56b3cce18841623e2cbed2ebf09b06be1be8807e99e6e054a89d304ab4790b8e/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f5048502d382e312532302d2d253230382e342d626c75653f6c6f676f3d706870266c6f676f436f6c6f723d7768697465)](https://php.net)[![Hyva + Luma](https://camo.githubusercontent.com/d4d6a05401973034a5aae392f030440eefa40afc9191ffb06ed897af5d25adb1/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f5468656d65732d487976612532302532422532304c756d612d313462386136)](https://www.hyva.io)[![Live Demo & Details](https://camo.githubusercontent.com/f6700c184624c920741e1092b3e6ebc34c5fa9fddd8f195960a5af13c08a133b/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4c69766525323044656d6f25323025323625323044657461696c732d6d6167656e746f2d2d322d2d726f626f74732d2d73656f2d3044393438383f7374796c653d666c6174)](https://kishansavaliya.com/magento-2-robots-seo.html)[![Packagist](https://camo.githubusercontent.com/321e0bb7d1e1991b286db1e682b569c43cc27d33675717fa1f33ea67c4f124d6/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f5061636b61676973742d6d616765326b697368616e2532466d6f64756c652d2d726f626f74732d2d73656f2d6f72616e67653f6c6f676f3d7061636b6167697374266c6f676f436f6c6f723d7768697465)](https://packagist.org/packages/mage2kishan/module-robots-seo)[![Upwork Top Rated Plus](https://camo.githubusercontent.com/6f72584179420c41ed90432fd2579a4ed36199d4229e8181d20f353c1c4ee4eb/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f5570776f726b2d546f702532305261746564253230506c75732d3134613830303f6c6f676f3d7570776f726b266c6f676f436f6c6f723d7768697465)](https://www.upwork.com/freelancers/~016dd1767321100e21)[![Website](https://camo.githubusercontent.com/f1ae86d28e2b505aee60f240d3e5508e390b0a8dc7a9b7ecf1b450fad862053f/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f576562736974652d6b697368616e736176616c6979612e636f6d2d304439343838)](https://kishansavaliya.com) > **Full robots and crawler policy control for Magento 2.** Panth Robots SEO takes over `/robots.txt` at the router layer, emits an `X-Robots-Tag` HTTP header on every frontend response, and gives you one-click toggles for 14 AI and LLM crawlers including GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. Every directive passes a CRLF-safe validator before it reaches the wire. Works identically on **Hyva** and **Luma**. **Product page:** [kishansavaliya.com/magento-2-robots-seo.html](https://kishansavaliya.com/magento-2-robots-seo.html) --- Quick Answer ------------ [](#quick-answer) **What is Panth Robots SEO?** It is a Magento 2 extension that replaces Magento's limited robots handling with a dedicated controller for `/robots.txt`, a per-response `X-Robots-Tag` HTTP header, and an admin UI for managing per-user-agent path policies and AI crawler access. **What does it add to my store?** - A **dynamic `/robots.txt` per store view** built from LLM bot toggles, admin policy rows, crawl-delay, and sitemap references. No static file is ever read from disk. - **14 LLM and AI crawler toggles** for GPTBot, ClaudeBot, ChatGPT-User, PerplexityBot, Google-Extended, Cohere-AI, Bytespider, and more. One click blocks or allows each bot. - An **`X-Robots-Tag` HTTP response header** on every frontend HTML page, with automatic noindex for error pages, private paths, layered-nav filters, and search result pages. - An **admin CRUD grid** for per-user-agent, per-path, per-store-view allow/disallow rows, plus a live **robots.txt Preview** page so you can verify output before it goes public. **Which themes are supported?** Both **Hyva** and **Luma**. The module works at the controller and plugin layer, so no theme-specific template is needed. **What does it need?** Magento 2.4.4 to 2.4.8, PHP 8.1 to 8.4, and the free `mage2kishan/module-core` package. --- Need Custom Magento 2 Development? ---------------------------------- [](#need-custom-magento-2-development) > **Get a free quote for your project in 24 hours** for custom modules, Hyva themes, performance work, M1 to M2 migrations, and Adobe Commerce Cloud. [ ![Get a Free Quote](https://camo.githubusercontent.com/eac8c45d21cff8b139ddc392325f3bd6c8266a6f3d7b23f15131c958f3d3c8d0/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f476574253230612532304672656525323051756f74652532302545322538362539322d5265706c7925323077697468696e2532303234253230686f7572732d4443323632363f7374796c653d666f722d7468652d6261646765) ](https://kishansavaliya.com/get-quote) ### Kishan Savaliya [](#kishan-savaliya) **Top Rated Plus on Upwork** [![Hire on Upwork](https://camo.githubusercontent.com/b69353d3c6e192f4d03cc36bb8883612004e32f54dd2dbcc1e700dd791acd875/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f486972652532306f6e2532305570776f726b2d546f702532305261746564253230506c75732d3134613830303f7374796c653d666f722d7468652d6261646765266c6f676f3d7570776f726b266c6f676f436f6c6f723d7768697465)](https://www.upwork.com/freelancers/~016dd1767321100e21) 100% Job Success - 10+ Years Magento Experience Adobe Certified - Hyva Specialist ### Panth Infotech Agency [](#panth-infotech-agency) **Magento Development Team** [![Visit Agency](https://camo.githubusercontent.com/bbf04bdd2aff502082508568ec42ace3a7475c98756f596e2013056c89726ed6/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f56697369742532304167656e63792d50616e7468253230496e666f746563682d3134613830303f7374796c653d666f722d7468652d6261646765266c6f676f3d7570776f726b266c6f676f436f6c6f723d7768697465)](https://www.upwork.com/agencies/1881421506131960778/) Custom Modules - Theme Design - Migrations Performance - SEO - Adobe Commerce Cloud **Visit our website:** [kishansavaliya.com](https://kishansavaliya.com) | **Get a quote:** [kishansavaliya.com/get-quote](https://kishansavaliya.com/get-quote) --- Table of Contents ----------------- [](#table-of-contents) - [Who Is It For](#who-is-it-for) - [Key Features](#key-features) - [Screenshots](#screenshots) - [Compatibility](#compatibility) - [Installation](#installation) - [Configuration](#configuration) - [Supported LLM Bots](#supported-llm-bots) - [How It Works](#how-it-works) - [FAQ](#faq) - [Support](#support) - [About Panth Infotech](#about-panth-infotech) - [Quick Links](#quick-links) --- Who Is It For ------------- [](#who-is-it-for) - **Stores worried about AI training scrapers** that want to block GPTBot, Bytespider, CCBot, or other data-collection bots in one click rather than hand-editing a file on every deploy. - **SEO-conscious merchants** who need layered-nav pages, search result pages, and customer account paths excluded from indexing through proper HTTP headers, not just a meta tag. - **Multi-store setups** where each store view needs its own robots.txt body, noindex path list, and LLM bot policy. - **Stores upgrading from Panth\_AdvancedSEO** that want robots handling as a standalone, self-contained module without pulling in the full SEO suite. - **Developers** who need a structured, schema-backed policy grid instead of a single admin textarea with no validation. --- Key Features ------------ [](#key-features) ### Dynamic robots.txt Per Store View [](#dynamic-robotstxt-per-store-view) - **Router-level controller** takes over `/robots.txt` so the response is built from live config every time. No static file is ever served. - **LLM and AI bot blocks** are written as `User-agent: \nDisallow: /` sections when their toggle is set to No. - **Admin policy rows** from the `panth_seo_robots_policy` table are merged under the matching user-agent block. - **Crawl-delay, Sitemap, and Host** lines are appended automatically from config. - **Custom body override** lets you paste your own robots.txt verbatim and skip the generation pipeline entirely. ### X-Robots-Tag HTTP Response Header [](#x-robots-tag-http-response-header) - **Added to every frontend HTML response** by `Plugin\Response\XRobotsTagPlugin` before the response is sent. - **Automatic noindex for error pages** (404, 410, 500, 503), non-HTML assets (.pdf, .doc, .xls), layered-nav filter pages, and search result pages. - **Configurable noindex path list** with wildcard `*` support for private paths like `/customer/*`, `/checkout`, `/wishlist`, and more. - **max-image-preview and max-snippet** are appended to every header value, including the `large` setting recommended for Google Discover. ### 14 LLM and AI Crawler Toggles [](#14-llm-and-ai-crawler-toggles) - **One Yes/No toggle per bot** in the LLM Bot Policy config group. Turning a bot to No writes a `Disallow: /` block for that user-agent in robots.txt. - **Bots covered:** GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot (covers Claude-Web too), Anthropic-AI, Google-Extended (covers GoogleOther), PerplexityBot, Cohere-AI, CCBot, Bytespider, Amazonbot, Applebot-Extended, FacebookBot, Meta-ExternalAgent. - **CCBot and Bytespider are blocked by default** because they feed large-scale training pipelines and are known to ignore partial disallows. ### Admin Policy Grid and robots.txt Preview [](#admin-policy-grid-and-robotstxt-preview) - **`panth_seo_robots_policy` table** stores per-user-agent, per-path, per-store-view allow/disallow rows with a priority column. - **Full CRUD grid** at Admin - Panth Infotech - Robots & LLM Bots - Robots Policies with mass enable, disable, and delete actions. - **robots.txt Preview page** renders the live output for any store view so you can check the result before it goes public. - **Store-view scope** on every row and config value so each store can have its own policy. ### Security Built In [](#security-built-in) - **CRLF-injection-safe** validator runs on every directive string before it reaches a response header or the robots.txt body. `\r`, `\n`, and `\0` are rejected outright. - **User-agent and path validation** on every policy save. UAs must match `/^[A-Za-z0-9._\-+*\/ ]+$/`; paths must start with `/` and contain no control bytes. - **ACL on every admin controller.** All routes require a valid admin session and declare their own `ADMIN_RESOURCE`. - **XSS-safe Preview page** renders the robots.txt body through `escapeHtml()` so a hostile custom body cannot execute script in the admin browser. ### Built to Last [](#built-to-last) - **Constructor DI only** across all classes. No ObjectManager calls. - **Full Page Cache friendly.** The robots.txt controller and the X-Robots-Tag plugin do not break Varnish or Fastly. - **Translation ready.** All admin labels use Magento's `__()` function. - **Zero data loss on upgrade from Panth\_AdvancedSEO.** The `panth_seo_robots_policy` table name is preserved and the schema shapes match exactly. --- Screenshots ----------- [](#screenshots) ### Live Walkthrough [](#live-walkthrough) End-to-end admin flow: enable the module, toggle a few LLM bots, add a policy row, preview the generated robots.txt, curl `/robots.txt` on both Hyva and Luma, and confirm the `X-Robots-Tag` header on a customer account page. [![Panth Robots SEO demo](docs/images/demo.gif)](docs/images/demo.gif) ### Admin Configuration [](#admin-configuration) Global configuration: toggle the module, set the default meta robots value, configure layered-nav and catalogsearch noindex, edit the noindex path list, and set max-image-preview, max-snippet, and Crawl-delay. [![Admin configuration](docs/images/admin-config.png)](docs/images/admin-config.png) ### Robots Policies Grid [](#robots-policies-grid) One row per user-agent, path, directive, and store view combination. Filter by store, mass-enable, disable or delete, and set priority so the evaluator knows which rule wins when two patterns overlap. [![Admin grid](docs/images/admin-grid.png)](docs/images/admin-grid.png) ### Policy Edit Form [](#policy-edit-form) Pick a user-agent (`*` for the default block, or `GPTBot`, `ClaudeBot`, a custom crawler), pick allow or disallow, enter a path, scope to a store view, and set priority and active flag. [![Admin edit form](docs/images/admin-edit.png)](docs/images/admin-edit.png) ### robots.txt Preview [](#robotstxt-preview) Dedicated Panth Infotech - Robots & LLM Bots - robots.txt Preview page renders the live body for the selected store view, exactly as the frontend controller will serve it. [![robots.txt preview](docs/images/robots-txt-preview.png)](docs/images/robots-txt-preview.png) --- Compatibility ------------- [](#compatibility) RequirementVersions SupportedMagento Open Source2.4.4, 2.4.5, 2.4.6, 2.4.7, 2.4.8Adobe Commerce2.4.4, 2.4.5, 2.4.6, 2.4.7, 2.4.8Adobe Commerce Cloud2.4.4 to 2.4.8PHP8.1.x, 8.2.x, 8.3.x, 8.4.xHyva Theme1.0+ (fully compatible)Luma ThemeNative supportRequired Dependency`mage2kishan/module-core` (free)--- Installation ------------ [](#installation) ### Composer Installation (Recommended) [](#composer-installation-recommended) ``` composer require mage2kishan/module-robots-seo bin/magento module:enable Panth_Core Panth_RobotsSeo bin/magento setup:upgrade bin/magento setup:di:compile bin/magento setup:static-content:deploy -f bin/magento cache:flush ``` ### Manual Installation via ZIP [](#manual-installation-via-zip) 1. Download the latest release from [Packagist](https://packagist.org/packages/mage2kishan/module-robots-seo) or from the [product page](https://kishansavaliya.com/magento-2-robots-seo.html). 2. Extract it to `app/code/Panth/RobotsSeo/` in your Magento install. 3. Make sure `Panth_Core` is installed too (required dependency). 4. Run the commands above starting from `bin/magento module:enable`. ### Verify Installation [](#verify-installation) ``` bin/magento module:status Panth_RobotsSeo # Expected: Module is enabled curl -s -o /dev/null -w '%{http_code}\n' https://your-store.test/robots.txt # 200 curl -sI https://your-store.test/customer/account/login | grep -i x-robots-tag # X-Robots-Tag: noindex, nofollow, max-image-preview:large, max-snippet:-1 ``` After install, open: ``` Admin -> Panth Infotech -> Robots & LLM Bots ``` --- Configuration ------------- [](#configuration) Go to **Stores -> Configuration -> Panth Infotech -> Robots & LLM Bots**. ### General [](#general) SettingGroupDefaultDescriptionEnable ModuleGeneralYesMaster switch. When No, the X-Robots-Tag plugin is a no-op and `/robots.txt` serves a stock `User-agent: *\nAllow: /`.Debug LoggingGeneralNoWhen Yes, every header and meta decision is written to `var/log/panth_robots_seo.log`.Default Meta RobotsGeneral`index,follow`Baseline directive applied when no per-entity or per-path override fires.Noindex Layered-Nav Filtered PagesGeneralYesEmit `noindex, follow` when a catalog listing has layered-nav or sort/limit/page query parameters.Noindex Search Result PagesGeneralYesEmit `noindex, follow` on `/catalogsearch/result/*` pages.Noindex URL Paths (one per line)General18-line seeded listPrivate path patterns that must emit `noindex, nofollow`. Wildcards supported: `*` matches anything.max-image-preview DirectiveGeneral`large`Appended to every X-Robots-Tag. `large` is recommended for Google Discover eligibility.max-snippet DirectiveGeneral`-1``-1` = unlimited. A positive integer caps the SERP snippet length.Crawl-delay (seconds)General`0`Emitted under `User-agent: *` in robots.txt. `0` omits the directive.### LLM Bot Policy [](#llm-bot-policy) SettingGroupDefaultDescriptionAllow GPTBot (OpenAI)LLM Bot PolicyYesNo = emits `User-agent: GPTBot\nDisallow: /`.Allow ClaudeBot (Anthropic)LLM Bot PolicyYesCovers both `ClaudeBot` and `Claude-Web`.Allow Google-ExtendedLLM Bot PolicyYesCovers both `Google-Extended` and `GoogleOther`.Allow CCBot (Common Crawl)LLM Bot PolicyNoBlocked by default. CCBot feeds dataset-scale training pipelines.Allow PerplexityBotLLM Bot PolicyYesAllow Bytespider (ByteDance)LLM Bot PolicyNoBlocked by default. Bytespider is known to ignore partial disallows.Allow ChatGPT-UserLLM Bot PolicyYesAllow OAI-SearchBotLLM Bot PolicyYesAllow Anthropic-AILLM Bot PolicyYesAllow Cohere-AILLM Bot PolicyYesAllow AmazonbotLLM Bot PolicyYesAllow Applebot-ExtendedLLM Bot PolicyYesAllow FacebookbotLLM Bot PolicyYesAllow Meta-ExternalAgentLLM Bot PolicyYes### robots.txt Override [](#robotstxt-override) SettingGroupDefaultDescriptionUse Custom robots.txt Bodyrobots.txt OverrideNoWhen Yes, the custom body REPLACES the generated output. All LLM toggles and policy rows are ignored.Custom robots.txt Bodyrobots.txt Override(empty)Pasted verbatim into the response. CRLF is normalised to LF. Leave empty to use the generated output.Every setting resolves at store-view scope, so each store can have a different LLM policy, noindex path list, or custom body. --- Supported LLM Bots ------------------ [](#supported-llm-bots) BotUA string(s)DefaultGPTBot (OpenAI)`GPTBot`AllowChatGPT-User`ChatGPT-User`AllowOAI-SearchBot`OAI-SearchBot`AllowClaudeBot (Anthropic)`ClaudeBot`, `Claude-Web`AllowAnthropic-AI`anthropic-ai`AllowGoogle-Extended`Google-Extended`, `GoogleOther`AllowPerplexityBot`PerplexityBot`AllowCohere-AI`cohere-ai`AllowCCBot (Common Crawl)`CCBot`**Disallow**Bytespider (ByteDance)`Bytespider`**Disallow**Amazonbot`Amazonbot`AllowApplebot-Extended`Applebot-Extended`AllowFacebookBot`FacebookBot`AllowMeta-ExternalAgent`meta-externalagent`AllowBots not listed here (YouBot, PetalBot, Diffbot, AI2Bot, etc.) are not blocked by default. To block them, add a `Disallow: /` row in the Robots Policies grid with the UA as the user-agent string. --- How It Works ------------ [](#how-it-works) 1. **`Controller\Robots\Index`** serves `GET /robots.txt` with the generated or override body at `Content-Type: text/plain; charset=utf-8`. The core Magento robots router is disabled in `etc/frontend/di.xml` so this controller always wins. 2. **`Setup\Patch\Data\InstallRobotsTxtRewrite`** writes the `url_rewrite` row that maps `/robots.txt` to the module controller at install time. `RefreshRobotsTxtRewrite` re-points stale rows left behind by `Panth_AdvancedSEO`, so upgrades need no manual DB work. 3. **`Plugin\Response\XRobotsTagPlugin`** runs `beforeSendResponse` on `Magento\Framework\App\Response\Http`. It reads the request path, HTTP status code, and Content-Type, then sets `X-Robots-Tag` once per response using this precedence order: - Self-skip on `/robots.txt`. - Error-code override (404, 410, 500, 503) to `noindex, nofollow`. - Non-HTML asset override (.pdf, .doc, .xls, .xlsx) to `noindex, nofollow`. - Catalogsearch noindex when `noindex_search_results = Yes`. - Configured `noindex_paths` match via `Service\NoindexPathMatcher`. - Layered-nav or sort-filter parameters to `noindex, follow`. - Default directive from `panth_robots_seo/general/default_directive`. 4. **`Model\Robots\PolicyResolver`** aggregates LLM-bot toggles, rows from `panth_seo_robots_policy`, the configured crawl-delay, and sitemap references into the final robots.txt body for a given store. 5. **`Service\DirectiveValidator`** is the single chokepoint every directive string passes through before it reaches a response header or the robots.txt body. It rejects any string containing `\r`, `\n`, `\0`, or bytes outside printable ASCII. --- FAQ --- [](#faq) ### Does it work on Hyva themes? [](#does-it-work-on-hyva-themes) Yes. The module works at the controller and plugin layer, not through layout or template. Both Hyva and Luma stores get the same robots.txt output and X-Robots-Tag header with no extra configuration. ### Magento already has a robots.txt textarea in Content -> Design -> Configuration. Why replace it? [](#magento-already-has-a-robotstxt-textarea-in-content---design---configuration-why-replace-it) Magento's built-in option is a single store-wide textarea with no header control, no LLM bot awareness, and no path-level noindex logic. Panth Robots SEO adds per-store-view generation, 14 AI crawler toggles, a structured policy grid, and the X-Robots-Tag header that Magento does not provide at all. ### Will it affect my existing robots.txt content? [](#will-it-affect-my-existing-robotstxt-content) When you first install the module, the generated robots.txt starts from your LLM bot toggles and any policy rows you create. If you want to keep your existing content exactly, paste it into the Custom robots.txt Body field and enable the override. ### Can I block just one path for a specific bot, not the whole site? [](#can-i-block-just-one-path-for-a-specific-bot-not-the-whole-site) Yes. Open Admin - Panth Infotech - Robots & LLM Bots - Robots Policies and add a row with the user-agent, `disallow`, and the path you want blocked. You can scope the row to a specific store view and set a priority. ### Does it set the HTML `` tag too? [](#does-it-set-the-html-meta-namerobots-tag-too) The module sets the `X-Robots-Tag` HTTP response header on every page, which search engines treat as equivalent to the meta tag. The HTML meta tag is updated too if `Panth_AdvancedSEO` is installed alongside this module. Without AdvancedSEO, only the HTTP header is set. ### Is the noindex path list configurable? [](#is-the-noindex-path-list-configurable) Yes. Go to Stores - Configuration - Panth Infotech - Robots & LLM Bots - General - Noindex URL Paths. Enter one path per line. The `*` wildcard matches anything. The seeded default covers `/customer/*`, `/checkout`, `/wishlist`, `/sales/*`, and about 14 other private patterns. ### What happens on a 404 page? [](#what-happens-on-a-404-page) The X-Robots-Tag plugin hard-overrides to `noindex, nofollow` for HTTP status codes 404, 410, 500, and 503, regardless of any other config. Error pages can never appear in the index. ### Does it need Panth\_AdvancedSEO? [](#does-it-need-panth_advancedseo) No. The module is fully standalone. If AdvancedSEO is also installed, they share the `panth_seo_robots_policy` table and do not conflict with each other. ### Is it multi-store safe? [](#is-it-multi-store-safe) Yes. Every config value, every policy row, and every X-Robots-Tag decision resolves at store-view scope. A setting on one store view never affects another. --- Support ------- [](#support) ChannelContactProduct Page[kishansavaliya.com/magento-2-robots-seo.html](https://kishansavaliya.com/magento-2-robots-seo.html)EmailWebsite[kishansavaliya.com](https://kishansavaliya.com)WhatsApp+91 84012 70422GitHub Issues[github.com/mage2sk/module-robots-seo/issues](https://github.com/mage2sk/module-robots-seo/issues)Upwork (Top Rated Plus)[Hire Kishan Savaliya](https://www.upwork.com/freelancers/~016dd1767321100e21)Upwork Agency[Panth Infotech](https://www.upwork.com/agencies/1881421506131960778/)Response time: 1-2 business days. ### Need Custom Magento Development? [](#need-custom-magento-development) Looking for **custom Magento module development**, **Hyva theme work**, **store migrations**, or **performance tuning**? Get a free quote in 24 hours: [ ![Get a Free Quote](https://camo.githubusercontent.com/0d9512dc4f89ef40179e9f650fd0786b6bc846d289ab2cb04a223a43f2833eeb/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f254630253946253932254143253230476574253230612532304672656525323051756f74652d6b697368616e736176616c6979612e636f6d2532466765742d2d71756f74652d4443323632363f7374796c653d666f722d7468652d6261646765) ](https://kishansavaliya.com/get-quote) [ ![Hire on Upwork](https://camo.githubusercontent.com/fad56e8be7896e39593f090cd2ed32e6bf86b4b6e75974cf292aee99d45f5077/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f486972652532304b697368616e2d546f702532305261746564253230506c75732d3134613830303f7374796c653d666f722d7468652d6261646765266c6f676f3d7570776f726b266c6f676f436f6c6f723d7768697465) ](https://www.upwork.com/freelancers/~016dd1767321100e21) [ ![Visit Agency](https://camo.githubusercontent.com/fdf45a9d937d72b896f5e2557337d47efebc0e06b7ad4d4c162c0a64375a62fa/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f56697369742d50616e7468253230496e666f746563682532304167656e63792d3134613830303f7374796c653d666f722d7468652d6261646765266c6f676f3d7570776f726b266c6f676f436f6c6f723d7768697465) ](https://www.upwork.com/agencies/1881421506131960778/) [ ![View Product Page](https://camo.githubusercontent.com/c43ba12401ab077a6908855b2e7a3e7244555652a61d62434160092ca3e3a82a/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f5669657725323050726f64756374253230506167652d6d6167656e746f2d2d322d2d726f626f74732d2d73656f2d3044393438383f7374796c653d666f722d7468652d6261646765) ](https://kishansavaliya.com/magento-2-robots-seo.html) --- About Panth Infotech -------------------- [](#about-panth-infotech) Built and maintained by **Kishan Savaliya** ([kishansavaliya.com](https://kishansavaliya.com)), a **Top Rated Plus** Magento developer on Upwork with 10+ years of eCommerce experience. **Panth Infotech** is a Magento 2 development agency that builds high quality, security focused extensions and themes for both Hyva and Luma storefronts. The extension suite covers SEO, performance, checkout, product presentation, customer engagement, and store management, with each module built to MEQP standards and tested across Magento 2.4.4 to 2.4.8. Browse the full extension catalog on our [Magento extensions page](https://kishansavaliya.com/magento-extensions.html) or on [Packagist](https://packagist.org/packages/mage2kishan/). --- Quick Links ----------- [](#quick-links) ResourceLink**Product Page**[magento-2-robots-seo.html](https://kishansavaliya.com/magento-2-robots-seo.html)**Packagist**[mage2kishan/module-robots-seo](https://packagist.org/packages/mage2kishan/module-robots-seo)**GitHub**[mage2sk/module-robots-seo](https://github.com/mage2sk/module-robots-seo)**Website**[kishansavaliya.com](https://kishansavaliya.com)**Free Quote**[kishansavaliya.com/get-quote](https://kishansavaliya.com/get-quote)**Upwork (Top Rated Plus)**[Hire Kishan Savaliya](https://www.upwork.com/freelancers/~016dd1767321100e21)**Upwork Agency**[Panth Infotech](https://www.upwork.com/agencies/1881421506131960778/)**Email****WhatsApp**+91 84012 70422--- **Ready to take control of how bots and crawlers see your store?** [ ![See Robots SEO](https://camo.githubusercontent.com/006dbd4487dd827c90bcce6d2bc7192eb914a4257d8c42a592e1ce522ef9430c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f254630253946253941253830253230536565253230526f626f747325323053454f2532302545322538362539322d50726f647563742532305061676525323025323625323044657461696c732d4443323632363f7374796c653d666f722d7468652d6261646765) ](https://kishansavaliya.com/magento-2-robots-seo.html) --- **SEO Keywords:** magento 2 robots.txt, magento 2 robots seo, magento 2 x-robots-tag, magento 2 llm bot policy, magento 2 ai crawler control, magento 2 block gptbot, magento 2 block claudebot, magento 2 block perplexitybot, magento 2 block bytespider, magento 2 google-extended, magento 2 noindex, magento 2 noindex layered nav, magento 2 noindex search results, magento 2 crawl delay, magento 2 robots meta, magento 2 seo headers, hyva robots seo, luma robots seo, magento 2 robots extension, magento 2 robots module, mage2kishan robots seo, panth robots seo, panth infotech, hire magento developer, top rated plus upwork, kishan savaliya magento, custom magento development, magento 2.4.8 robots, php 8.4 magento seo ### Health Score 44 — FairBetter than 90% of packages Maintenance91 Actively maintained with recent releases Popularity9 Limited adoption so far Community7 Small or concentrated contributor base Maturity57 Maturing project, gaining track record Bus Factor1 Top contributor holds 100% of commits — single point of failure How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window. **Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores. **Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement. **Maturity (30%)** — Project age, version count, PHP version support, and release stability. ### Release Activity Cadence Every ~7 days Recently: every ~13 days Total 9 Last Release 36d ago ### Community Maintainers ![](https://www.gravatar.com/avatar/343e344aa298f189db888b32d62f9202d31ced1a5ea23411850a63dc4a30299c?d=identicon)[kishansavaliya](/maintainers/kishansavaliya) --- Top Contributors [![KishanSavaliya](https://avatars.githubusercontent.com/u/16853223?v=4)](https://github.com/KishanSavaliya "KishanSavaliya (13 commits)") --- Tags robots-txtseorobotsmagento2magento2 modulex-robots-taghyvalumapanthgptbotclaudebotllm-botsperplexitybot ### Embed Badge ![Health badge](/badges/mage2kishan-module-robots-seo/health.svg) ``` [![Health](https://phpackages.com/badges/mage2kishan-module-robots-seo/health.svg)](https://phpackages.com/packages/mage2kishan-module-robots-seo) ``` ### Alternatives [mollie/magento2 Mollie Payment Module for Magento 2 1131.9M16](/packages/mollie-magento2)[run-as-root/magento2-prometheus-exporter Magento2 Prometheus Exporter 68357.9k](/packages/run-as-root-magento2-prometheus-exporter)[loki/magento2-components Core module for defining Alpine.js components with advanced AJAX features 1011.8k26](/packages/loki-magento2-components)[opengento/module-category-import-export This module add the capability to import and export the categories from the back-office. 1310.9k2](/packages/opengento-module-category-import-export)[angeo/module-llms-txt Magento 2 module for AI Engine Optimization (AEO). Generates spec-compliant llms.txt and llms-full.txt per llmstxt.org standard, plus streaming JSONL for vector indexing. Multi-store, multi-website, CLI, cron, async admin UI, Page Builder-aware sanitization, customer-group pricing, atomic writes, ETag/Cache-Control, .md mirrors. 111.1k](/packages/angeo-module-llms-txt) PHPackages © 2026 [Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)