PHPackages                             daikazu/robotstxt - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Utility & Helpers](/categories/utility)
4. /
5. daikazu/robotstxt

ActiveLibrary[Utility & Helpers](/categories/utility)

daikazu/robotstxt
=================

Dynamically generate robots.txt content based on the current Laravel environment.

v1.1.0(1mo ago)028—0%MITPHPPHP ^8.4CI failing

Since Nov 20Pushed 1mo agoCompare

[ Source](https://github.com/daikazu/robotstxt)[ Packagist](https://packagist.org/packages/daikazu/robotstxt)[ Docs](https://github.com/daikazu/robotstxt)[ GitHub Sponsors](https://github.com/Daikazu)[ RSS](/packages/daikazu-robotstxt/feed)WikiDiscussions main Synced 1mo ago

READMEChangelog (2)Dependencies (13)Versions (4)Used By (0)

  ![Logo for ROBOTS.TXT](art/header-light.png)[![Latest Version on Packagist](https://camo.githubusercontent.com/d9ae655cfae45b53ec80df7aab6d14708c0bccc6dbcfa6ad6efab516587906d8/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f762f6461696b617a752f726f626f74737478742e7376673f7374796c653d666c61742d737175617265)](https://packagist.org/packages/daikazu/robotstxt)[![GitHub Tests Action Status](https://camo.githubusercontent.com/2d5b55809e6fb39f7d24722443b6b197239708010cef24e838cc1c597ff3ee17/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f616374696f6e732f776f726b666c6f772f7374617475732f6461696b617a752f726f626f74737478742f72756e2d74657374732e796d6c3f6272616e63683d6d61696e266c6162656c3d7465737473267374796c653d666c61742d737175617265)](https://github.com/daikazu/robotstxt/actions?query=workflow%3Arun-tests+branch%3Amain)[![GitHub Code Style Action Status](https://camo.githubusercontent.com/793d60f4b4ebbc5d484277e7c7a095e31405976a180c9fe8b539e5ddfb3f8bc2/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f616374696f6e732f776f726b666c6f772f7374617475732f6461696b617a752f726f626f74737478742f6669782d7068702d636f64652d7374796c652d6973737565732e796d6c3f6272616e63683d6d61696e266c6162656c3d636f64652532307374796c65267374796c653d666c61742d737175617265)](https://github.com/daikazu/robotstxt/actions?query=workflow%3A%22Fix+PHP+code+style+issues%22+branch%3Amain)[![Total Downloads](https://camo.githubusercontent.com/872bf227514583899ee61a6145b053e511aa50a3ba5094d678a14694263ea03a/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f64742f6461696b617a752f726f626f74737478742e7376673f7374796c653d666c61742d737175617265)](https://packagist.org/packages/daikazu/robotstxt)

Dynamic robots.txt for your Laravel app.
========================================

[](#dynamic-robotstxt-for-your-laravel-app)

A Laravel package for dynamically generating robots.txt files with environment-specific configurations. Control how search engines and AI crawlers interact with your site using modern features like Cloudflare's Content Signals Policy, per-environment rules, and flexible content signal directives.

Perfect for applications that need different robots.txt rules across environments (production, staging, local) or want granular control over AI training, search indexing, and content access.

Features
--------

[](#features)

- **Environment-Specific Configuration** - Different robots.txt rules for production, staging, local, etc.
- **Content Signals Support** - Implement Cloudflare's Content Signals Policy to control AI training, search indexing, and AI content usage
- **Flexible User-Agent Rules** - Define global rules or per-agent directives (disallow, allow, content signals)
- **Host Directive** - Specify preferred domain for crawlers
- **Sitemap Management** - Automatically include sitemap URLs
- **Custom Text** - Add custom content to your robots.txt file
- **Human-Readable Policies** - Optional policy comment blocks with custom or default text

Installation
------------

[](#installation)

You can install the package via composer:

```
composer require daikazu/robotstxt
```

You can publish the config file with:

```
php artisan vendor:publish --tag="robotstxt-config"
```

### Nginx Configuration (Required for Production)

[](#nginx-configuration-required-for-production)

If you're getting a 404 status (but still seeing content), you need to configure Nginx to pass robots.txt requests to Laravel:

**For Laravel Herd:**Add this to your site's Nginx config (via Herd UI or `~/.config/herd/Nginx/[site].conf`):

```
location = /robots.txt {
      try_files $uri /index.php?$query_string;
      access_log off;
      log_not_found off;
  }
```

**For Laravel Forge/Vapor:**Add the same location block to your Nginx configuration.

**For custom servers:**Add to your server block in your Nginx config file.

Then restart Nginx/Herd.

Usage
-----

[](#usage)

After installation, the package automatically registers a route at `/robots.txt` that serves your dynamically generated robots.txt file.

### Basic Configuration

[](#basic-configuration)

The default configuration file (`config/robotstxt.php`) provides environment-specific settings. Here's a simple example:

```
return [
    'environments' => [
        'production' => [
            'paths' => [
                '*' => [
                    'disallow' => ['/admin', '/api'],
                    'allow' => ['/'],
                ],
            ],
            'sitemaps' => [
                'sitemap.xml',
            ],
        ],
    ],
];
```

This generates:

```
Sitemap: https://example.com/sitemap.xml

User-agent: *
Disallow: /admin
Disallow: /api
Allow: /

```

### Content Signals (AI & Search Control)

[](#content-signals-ai--search-control)

Control how AI crawlers and search engines use your content with Cloudflare's Content Signals Policy:

```
'production' => [
    // Enable human-readable policy comment block
    'content_signals_policy' => [
        'enabled' => true,
        'custom_policy' => null, // or provide your own HEREDOC text
    ],

    // Global content signals (applied at top level)
    'content_signals' => [
        'search'   => true,   // Allow search indexing
        'ai_input' => false,  // Block AI input/RAG
        'ai_train' => false,  // Block AI training
    ],

    'paths' => [
        '*' => [
            'disallow' => [],
            'allow' => ['/'],
        ],
    ],
],
```

Generates:

```
# As a condition of accessing this website, you agree to abide by the following
# content signals:
# [Full policy text...]

Content-Signal: search=yes, ai-input=no, ai-train=no

User-agent: *
Allow: /

```

### Per-Agent Content Signals

[](#per-agent-content-signals)

You can also define content signals for specific user agents:

```
'paths' => [
    '*' => [
        'disallow' => [],
        'allow' => ['/'],
    ],
    'Googlebot' => [
        'content_signals' => [
            'search'   => true,
            'ai_input' => true,
            'ai_train' => false,
        ],
        'disallow' => ['/private'],
        'allow' => ['/'],
    ],
],
```

Generates:

```
Content-Signal: search=yes, ai-input=no, ai-train=no

User-agent: *
Allow: /

User-agent: Googlebot
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /private
Allow: /

```

### Host Directive

[](#host-directive)

Specify the preferred domain for crawlers:

```
'production' => [
    'host' => 'https://www.example.com',
    // ... other config
],
```

Generates:

```
Host: https://www.example.com

```

### Custom Text

[](#custom-text)

Add arbitrary custom content to the end of your robots.txt:

```
'production' => [
    'custom_text' =>  [
        'production' => [
            'paths' => [
                '*' => [
                    'disallow' => [],
                    'allow' => ['/'],
                ],
            ],
            'content_signals' => [
                'search' => true,
                'ai_input' => false,
                'ai_train' => false,
            ],
        ],
        'staging' => [
            'paths' => [
                '*' => [
                    'disallow' => ['/'],
                ],
            ],
        ],
        'local' => [
            'paths' => [
                '*' => [
                    'disallow' => ['/'],
                ],
            ],
        ],
    ],
];
```

### Content Signal Values

[](#content-signal-values)

- `true` or `'yes'` - Permission granted
- `false` or `'no'` - Permission denied
- `null` - No preference specified (signal not included)

### Content Signal Types

[](#content-signal-types)

- **search** - Building a search index and providing search results (excludes AI summaries)
- **ai\_input** - Inputting content into AI models (RAG, grounding, AI Overviews)
- **ai\_train** - Training or fine-tuning AI models

### Complete Configuration Example

[](#complete-configuration-example)

```
return [
    'environments' => [
        'production' => [
            // Content Signals Policy
            'content_signals_policy' => [
                'enabled' => true,
                'custom_policy' => null,
            ],

            // Global Content Signals
            'content_signals' => [
                'search'   => true,
                'ai_input' => false,
                'ai_train' => false,
            ],

            // User-Agent Rules
            'paths' => [
                '*' => [
                    'disallow' => ['/admin', '/api'],
                    'allow' => ['/'],
                ],
                'Googlebot' => [
                    'content_signals' => [
                        'search'   => true,
                        'ai_input' => true,
                        'ai_train' => false,
                    ],
                    'disallow' => [],
                    'allow' => ['/'],
                ],
            ],

            // Sitemaps
            'sitemaps' => [
                'sitemap.xml',
                'sitemap-news.xml',
            ],

            // Host
            'host' => 'https://www.example.com',

            // Custom Text
            'custom_text' =>