PHPackages                             crwlr/crwl-extension-utils - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. crwlr/crwl-extension-utils

ActiveLibrary

crwlr/crwl-extension-utils
==========================

Utils for extension packages for the crwl.io app.

v2.4.2(1y ago)02.4k↓100%2PHPPHP ^8.1

Since May 2Pushed 4mo ago1 watchersCompare

[ Source](https://github.com/crwlrsoft/crwl-extension-utils)[ Packagist](https://packagist.org/packages/crwlr/crwl-extension-utils)[ GitHub Sponsors](https://github.com/sponsors/otsch)[ RSS](/packages/crwlr-crwl-extension-utils/feed)WikiDiscussions main Synced 1mo ago

READMEChangelog (10)Dependencies (7)Versions (13)Used By (2)

crwl.io Extension Utils
=======================

[](#crwlio-extension-utils)

These utilities enable you to develop extensions for the crwl.io web crawling and scraping app.

How to Package Your Custom Steps for the crwl.io App
----------------------------------------------------

[](#how-to-package-your-custom-steps-for-the-crwlio-app)

The [crwlr.software documentation](https://www.crwlr.software/packages/crawler/steps-and-data-flow/custom-steps) provides a detailed explanation of how to build your own steps for the crwlr/crawler library.

Once you've successfully created a working step, the most challenging part of your task is already complete. To integrate your custom steps into the crwl.io app, follow these simple steps:

- Package it as a Composer package in a GitHub repository.
- Create a `StepBuilder`
- Create a laravel `ServiceProvider` and register your extension steps using the `ExtensionPackageManager` included in this package.

### Composer Package Setup

[](#composer-package-setup)

Ensure your package repository's `composer.json` file resembles the following template:

```
{
  "name": "my-vendor/my-crwl-extension",
  "description": "Extension package with custom steps for the crwl.io app",
  "type": "library",
  "autoload": {
    "psr-4": {
      "MyVendor\\MyCrwlExtension\\": "src/"
    }
  },
  "require": {
    "crwlr/crawler": "^1.5",
    "illuminate/support": "^9.27|^10.0",
    "crwlr/crwl-extension-utils": "^1.0"
  }
}
```

Feel free to customize it according to your preferences, but ensure that it includes dependencies on the following three packages: `crwlr/crawler`, `crwlr/crwl-extension-utils` and `illuminate/support`.

For a well-organized project structure, we recommend the following folder arrangement:

```
/
├─ src/
│  ├─ StepBuilders/
│  ├─ Steps/
│  ├─ ServiceProvider.php
├─ .gitignore
├─ composer.json
├─ README.md

```

In your `.gitignore`, include at least the following entries:

```
/vendor/
composer.lock
```

### Step Builders

[](#step-builders)

To create a `StepBuilder` for your step, follow the example below:

```
namespace MyVendor\MyCrwlExtension\StepBuilders;

use Crwlr\Crawler\Steps\StepInterface;
use Crwlr\Crawler\Steps\StepOutputType;
use Crwlr\CrwlExtensionUtils\ConfigParam;
use Crwlr\CrwlExtensionUtils\StepBuilder;
use MyVendor\MyCrwlExtension\Steps\MyStep;

class MyStepBuilder extends StepBuilder
{
    public function stepId(): string
    {
        return 'my-extension.my-step';
    }

    public function label(): string
    {
        return 'This step does X.';
    }

    public function configToStep(array $stepConfig): StepInterface
    {
        $fooConfigValue = $this->getValueFromConfigArray('foo', $stepConfig);

        $barConfigValue = $this->getValueFromConfigArray('bar', $stepConfig);

        $bazConfigValue = $this->getValueFromConfigArray('baz', $stepConfig);

        return new MyStep($fooConfigValue, $barConfigValue);
    }

    public function configParams(): array
    {
        return [
            ConfigParam::string('foo')
                ->inputLabel('Your foo'),
            ConfigParam::int('bar')
                ->default(5)
                ->inputLabel('Number of bar')
                ->description('Provide the number of bar, so the step can do X.'),
            ConfigParam::bool('baz')
                ->inputLabel('Baz?'),
        ];
    }

    /**
     * Define the possible output type. More info at
     * https://www.crwlr.software/packages/crawler/v1.10/steps-and-data-flow/custom-steps#step-output-types
     * If the underlying step can yield different output types based on the configuration,
     * just return StepOutputType::Mixed.
     */
    public function outputType(): StepOutputType
    {
        return StepOutputType::AssociativeArrayOrObject;
    }
}
```

If your step requires configuration, define the necessary parameters in the `configParams()` method. The crwl.io app's crawler creation/editing form will display corresponding inputs for these config options. When the crawler runs, the `StepBuilder::configToStep()` method is invoked with the user-saved configuration data. In this method, construct your custom step with the configured values and return it.

Currently, the available config param types are `string`, `int`, and `bool`. Optionally, you can specify a default value (`default()`), an input label (`inputLabel()`), and a description text (`description()`) for each configuration parameter.

If your step doesn't require any settings, the `StepBuilder` looks rather minimalistic:

```
namespace MyVendor\MyCrwlExtension\StepBuilders;

use Crwlr\Crawler\Steps\StepInterface;
use Crwlr\Crawler\Steps\StepOutputType;use Crwlr\CrwlExtensionUtils\StepBuilder;
use MyVendor\MyCrwlExtension\Steps\MyStep;

class MyStepBuilder extends StepBuilder
{
    public function stepId(): string
    {
        return 'my-extension.my-step';
    }

    public function label(): string
    {
        return 'This step does X.';
    }

    public function configToStep(array $stepConfig): StepInterface
    {
        return new MyStep();
    }

    public function outputType(): StepOutputType
    {
        return StepOutputType::Scalar;
    }
}
```

If your step needs a filesystem path, where it can store files, you can use `$this->fileStoragePath` inside the builder. The crwl.io app sets this path for all step builders before building any steps.

```
public function configToStep(array $stepConfig): StepInterface
{
    return new MyStep($this->fileStoragePath);
}
```

When creating a step builder for an HTTP loading step, please implement the following method:

```
public function isLoadingStep(): bool
{
    return true;
}
```

### ServiceProvider and Registering Package and Steps

[](#serviceprovider-and-registering-package-and-steps)

To make your steps accessible in the crwl.io app, the final step is to register an extension package and all your steps using the `ExtensionPackageManager` included in this package. Since crwl.io is a [Laravel](https://laravel.com/) application, this is accomplished through a `ServiceProvider` class:

```
namespace MyVendor\MyCrwlExtension;

use Crwlr\CrwlExtensionUtils\ExtensionPackageManager;
use MyCrwlExtension\StepBuilders\FooStepBuilder;
use MyCrwlExtension\StepBuilders\BarStepBuilder;
use MyCrwlExtension\StepBuilders\BazStepBuilder;

class ServiceProvider extends \Illuminate\Support\ServiceProvider
{
    public function register()
    {
        $this->app->make(ExtensionPackageManager::class)
            ->registerPackage('my-vendor-name/my-crwl-extension')
            ->registerStep(FooStepBuilder::class)
            ->registerStep(BarStepBuilder::class)
            ->registerStep(BazStepBuilder::class);
    }
}
```

To complete the setup, add the `ServiceProvider` to the `extra` section in the `composer.json` file:

```
{
  "name": "my-vendor/my-crwl-extension",
  "description": "Extension package with custom steps for the crwl.io app",
  "type": "library",
  "autoload": {
    "psr-4": {
      "MyVendor\\MyCrwlExtension\\": "src/"
    }
  },
  "require": {
    "crwlr/crawler": "^1.4",
    "illuminate/support": "^9.27|^10.0",
    "crwlr/crwl-extension-utils": "^1.0"
  },
  "extra": {
    "laravel": {
      "providers": [
        "MyVendor\\MyCrwlExtension\\ServiceProvider"
      ]
    }
  }
}
```

With these configurations in place, your extension package is ready for use. If your extension is private, ensure you grant access to the [crwlrsoft GitHub organization](https://github.com/crwlrsoft). As a super-user on your crwl.io instance, you can then install your extension via the extensions page in the app.

Custom Steps Performing HTTP Requests Without the crwlr/crawler HttpLoader
--------------------------------------------------------------------------

[](#custom-steps-performing-http-requests-without-the-crwlrcrawler-httploader)

In scenarios where your custom steps need to execute HTTP requests that cannot leverage the `HttpLoader` from the `crwlr/crawler` package—such as when utilizing a REST API SDK to retrieve data from an API—you'll need to ensure that every HTTP request is tracked when executing your custom steps within the crwl.io app.

To accomplish this, you have two options:

- Use a Guzzle Client instance generated by the `TrackingGuzzleClientFactory`.
- Alternatively, manually invoke the `trackHttpResponse()` or `trackHeadlessBrowserResponse()` methods of the `RequestTracker` following each request.

### Using a guzzle Client instance

[](#using-a-guzzle-client-instance)

If you want to use a Guzzle `Client` instance for the requests (it's common practice for PHP API SDKs to let you provide your own Guzzle instance), use the `TrackingGuzzleClientFactory` from this package:

```
use Crwlr\CrwlExtensionUtils\TrackingGuzzleClientFactory;

// Let the factory be resolved by the laravel service container.
$factory = app()->make(TrackingGuzzleClientFactory::class);

$client = $factory->getClient();
```

You can also pass your custom Guzzle configuration as an argument:

```
$client = $factory->getClient(['allow_redirects' => false]);
```

### Using the RequestTracker

[](#using-the-requesttracker)

In scenarios where utilizing a Guzzle `Client` instance for requests is not feasible, you need to call either `RequestTracker::trackHttpResponse()` or if your request was executed using a headless browser `RequestTracker::trackHeadlessBrowserResponse()`.

```
use Crwlr\CrwlExtensionUtils\RequestTracker;

// Let the tracker be resolved by the laravel service container.
$tracker = app()->make(RequestTracker::class);

// Execute your request however you want...

$tracker->trackHttpResponse();

// or

$tracker->trackHeadlessBrowserResponse();
```

If you can provide request/response instances implementing the PSR-7 `RequestInterface` and/or `ResponseInterface`, please do so:

```
$tracker->trackHttpResponse($request, $response);

// or

$tracker->trackHeadlessBrowserResponse($request, $response);
```

###  Health Score

40

—

FairBetter than 88% of packages

Maintenance60

Regular maintenance activity

Popularity18

Limited adoption so far

Community11

Small or concentrated contributor base

Maturity60

Established project with proven stability

 Bus Factor1

Top contributor holds 100% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Every ~53 days

Recently: every ~66 days

Total

12

Last Release

518d ago

Major Versions

v0.1.1 → v1.0.02024-01-31

v1.1.0 → v2.0.02024-02-07

### Community

Maintainers

![](https://www.gravatar.com/avatar/3074cef6e2926ede6d4c9c39a0cf29e2e86e6927255a17c103114d0a5957e1a7?d=identicon)[crwlr](/maintainers/crwlr)

---

Top Contributors

[![otsch](https://avatars.githubusercontent.com/u/4062813?v=4)](https://github.com/otsch "otsch (40 commits)")

###  Code Quality

TestsPest

Static AnalysisPHPStan

Code StylePHP CS Fixer

Type Coverage Yes

### Embed Badge

![Health badge](/badges/crwlr-crwl-extension-utils/health.svg)

```
[![Health](https://phpackages.com/badges/crwlr-crwl-extension-utils/health.svg)](https://phpackages.com/packages/crwlr-crwl-extension-utils)
```

###  Alternatives

[fumeapp/modeltyper

Generate TypeScript interfaces from Laravel Models

196277.9k](/packages/fumeapp-modeltyper)[aedart/athenaeum

Athenaeum is a mono repository; a collection of various PHP packages

255.2k](/packages/aedart-athenaeum)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)