PHPackages                             ttree/contentrepositoryimporter - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Utility & Helpers](/categories/utility)
4. /
5. ttree/contentrepositoryimporter

ActiveNeos-package[Utility & Helpers](/categories/utility)

ttree/contentrepositoryimporter
===============================

Helper package to import data in the Neos content repository

4.1.3(4y ago)1510.2k14[1 issues](https://github.com/ttreeagency/ContentRepositoryImporter/issues)MITPHP

Since Sep 23Pushed 4y ago5 watchersCompare

[ Source](https://github.com/ttreeagency/ContentRepositoryImporter)[ Packagist](https://packagist.org/packages/ttree/contentrepositoryimporter)[ RSS](/packages/ttree-contentrepositoryimporter/feed)WikiDiscussions master Synced 1mo ago

READMEChangelog (4)Dependencies (3)Versions (24)Used By (0)

ContentRepositoryImporter
=========================

[](#contentrepositoryimporter)

This package contains generic utility to help importing data in the Neos Content Repository.

What's included ?
-----------------

[](#whats-included-)

- A command controller (CLI) to launch your import presets
- Based on simple conventions
- DataProvider: used to prepare and cleanup data from the external source
- Importer: get the data from the DataProvider and push everything in the CR
- DataType: Simple object used to cleanup value and share code between DataProvider
- Split your import in multiple sub commands to avoid high memory usage
- No big magic, you can always take control by overriding the default configuration and methods

A basic DataProvider
--------------------

[](#a-basic-dataprovider)

Every data provider must extend the `DataProvider` abstract class or implement the interface `DataProviderInterface`. Check the source code of the abstract data provider, there are some useful things to discover.

It's important to update the `count` property when you process data from the external source. During the processing, you can decide to skip some data (invalid data, missing values, ...) so we can not use the SQL count feature.

Try to do most of the data cleaning up in the data provider, so the data would arrive to the importer ready for insertion. Basically the array build by the provider should contains the data with the property name that match your node type property name. If you need to transport value that will not match the node properties, please prefix them with '\_'.

There is some magic value, those values MUST be on the first level of the array:

- **\_\_identifier** (optional) This UUID will be used in the imported node, you should use `AbstractImporter::applyProperties` to have this feature, used by default
- **\_\_externalIdentifier** (required) The external identifier of the data, this one is really important. The package keep track of imported data
- **\_\_label** (required) The label of this record used by the importer mainly for logging (this value is not imported, but useful to follow the process) if you run twice the same import, the imported node will be updated and not created.

**Tips**: If the properties of your nodes are not at the first level of the array, you can override the method `AbstractImporter::getPropertiesFromDataProviderPayload`

### Output of the provider

[](#output-of-the-provider)

Your provider should output something like this:

```
	[
		'__label' => 'The external content lable, for internal use'
		'__externalIdentifier' => 'The external external identifier, for internal use'
		'title' => 'My title'
		'year' => 1999
		'text' => '...'
	]

```

**Tips**: If your provider does not return an array, you MUST registrer a TypeConverter to convert it to an array. The property mapper is used automatically by the Importer.

### Content Dimensions support

[](#content-dimensions-support)

If your data provider follow this convention, the importer can automatically create variants of your nodes:

```
	[
		'__label' => 'The external content lable, for internal use'
		'__externalIdentifier' => 'The external external identifier, for internal use'
		'title' => 'My title'
		'year' => 1999
		'text' => '...',

	    '@dimensions' => [
		   '@en' => [
			   '@strategy' => 'merge',
			   'title' => '...',
		   ],
		   '@fr' => [
			   '@strategy' => 'merge',
			   'title' => '...',
		   ],
	    ]
	]

```

The `@en` is a preset name, you must configuration the presets on your `Settings.yaml`:

```
Ttree:
  ContentRepositoryImporter:
    dimensionsImporter:
      presets:
        fr:
	  language: ['fr', 'en', 'de']
        en:
	  language: ['en', 'de']
        de:
	  language: ['de']

```

### Share data between preset parts

[](#share-data-between-preset-parts)

You can split your import in multiple parts. Each parts is executed in a separate request. Sometimes it's useful to share data between parts (ex. in the first part you import the taxonomy, and in the second parts you map documents with the taxonomy). Those solve this use case, we integrate a feature called **Vault**. The Vault is simply a cache accessible in the importer and data provider by calling `$this->vault->set($key, $name)` and `$this->vault->get($key)`. The current preset is the namespace, so you can use simple keys like name, ids, ...

The cache is flushed if you call `flow import:init --preset your-preset`.

### Basic provider

[](#basic-provider)

```
class BasicDataProvider extends DataProvider {

	/**
	 * @return array
	 */
	public function fetch() {
		$result = [];
		$query = $this->createQuery()
			->select('*')
			->from('demo_table', 'd')
			->orderBy('d.name');

		$statement = $query->execute();
		while ($demoRecord = $statement->fetch()) {
			$result[] = [
				'__externalIdentifier' => (integer)$demoRecord['id'],
				'name' => String::create($demoRecord['name'])->getValue()
			];
		}

		$this->count = count($result);

		return $result;
	}

}
```

A basic Importer
----------------

[](#a-basic-importer)

Every data importer must extend the `AbstractImporter` abstract class or implement the interface `ImporterInterface`.

In the `processRecord` method you handle the processing of every record, such as creating Content Repository node for each incoming data record.

Do not forget to register the processed nodes with `registerNodeProcessing`. The method will handle feature like logging and tracking of imported node to decide if the local node need to be created or updated.

```
class ProductImporter extends AbstractImporter
{

    /**
     * @var string
     */
    protected $externalIdentifierDataKey = 'productNumber';

    /**
     * @var string
     */
    protected $labelDataKey = 'properties.name';

    /**
     * @var string
     */
    protected $nodeNamePrefix = 'product-';

    /**
     * @var string
     */
    protected $nodeTypeName = 'Acme.Demo:Product';

    /**
     * Starts batch processing all commands
     *
     * @return void
     * @api
     */
    public function process()
    {
        $this->initializeStorageNode('shop/products', 'products', 'Products', 'products');
        $this->initializeNodeTemplates();

        $nodeTemplate = new NodeTemplate();
        $this->processBatch($nodeTemplate);
    }

}
```

A basic preset
--------------

[](#a-basic-preset)

You can configure an import preset in your `Settings.yaml`. A preset is split in multiple parts. If you use the `batchSize`, the current part will be executed by batch, by using a sub CLI request. This can solve memory or performance issue for big imports.

```
Ttree:
  ContentRepositoryImporter:
    sources:
      default:
        host: localhost
        driver: pdo_mysql
        dbname: database
        user: user
        password: password
      extraSourceDatabase:
        host: localhost
        driver: pdo_mysql
        dbname: database
        user: user
        password: password

    presets:
      'base':
        parts:
          'news':
            label: 'News Import'
            dataProviderClassName: 'Your\Package\Importer\DataProvider\NewsDataProvider'
            importerClassName: 'Your\Package\Importer\Importer\NewsImporter'
          'page':
            label: 'Page Import'
            dataProviderClassName: 'Your\Package\Importer\DataProvider\PageDataProvider'
            dataProviderOptions:
              source: 'extraSourceDatabase'
              someOption: 'Some option that will be available in the options property of the data provider'
            importerClassName: 'Your\Package\Importer\Importer\PageImporter'
            importerOptions:
              siteNodePath: '/sites/my-site'
              someOption: 'Some option that will be available in the options property of the importer'
            batchSize': 120

          'pageContent':
            label: 'Page Content Import'
            dataProviderClassName: 'Your\Package\Importer\DataProvider\PageContentDataProvider'
            importerClassName: 'Your\Package\Importer\Importer\PageContentImporter'
            batchSize: 120
```

Start your import process
-------------------------

[](#start-your-import-process)

**Tips**: Do not forget to require this package from the package in which you do the importing, to ensure the correct loading order, so the settings would get overriden correctly.

From the CLI:

```
flow import:batch --preset base

```

You can also filter the preset steps:

```
flow import:batch --preset base --parts page,pageContent

```

For testing purposes, or if you would like to override the value defined in your preset, you can also specify the number of records which should be imported at a time in an isolated sub-process:

```
flow import:batch --preset base --batch-size 50

```

Passing exceeding arguments to the DataProvider
-----------------------------------------------

[](#passing-exceeding-arguments-to-the-dataprovider)

The import process supports passing unnamed exceeding arguments to the `DataProvider`. This can be useful if you e.g. want to allow importing only a single record

```
flow import:batch --preset base recordIdentifier:1234

```

Exceeding arguments will be available in the `DataProvider` through `$this->getExceedingArguments()`. You need to process this data yourself and apply it to your fetching logic.

Command based importers
-----------------------

[](#command-based-importers)

Some data sources may consist of commands rather than data records. For example, a JSON file may contain `create`, `update` and `delete` instructions which reduce the guess-work on the importer's side, which records may be new, which should be updated and if the absence of a record means that the corresponding node should be deleted from the content repository.

For these cases you can extend the `AbstractCommandBasedImporter`. If your data records contain a `mode` field, the importer will try to call a corresponding command method within the same class.

Consider the following data source file as an example:

```
[
    {
        "mode": "create",
        "mpn": "1081251137",
        "languageIdentifier": "de",
        "properties": {
            "label": "Coffee Machine",
            "price": "220000",
            "externalKey": "1081251137"
        }
    },
    {
        "mode": "delete",
        "mpn": "591500202"
    }
]
```

A corresponding `ProductImporter` might look like this:

```
/**
 * Class ProductImporter
 */
class ProductImporter extends AbstractCommandBasedImporter
{

    /**
     * @var string
     */
    protected $storageNodeNodePath = 'products';

    /**
     * @var string
     */
    protected $storageNodeTitle = 'Products';

    /**
     * @var string
     */
    protected $externalIdentifierDataKey = 'mpn';

    /**
     * @var string
     */
    protected $labelDataKey = 'properties.Label';

    /**
     * @var string
     */
    protected $nodeNamePrefix = 'product-';

    /**
     * @var string
    */
    protected $nodeTypeName = 'Acme.MyShop:Product';

    /**
     * Creates a new product
     *
     * @param string $externalIdentifier
     * @param array $data
     * @return void
     */
    protected function createCommand($externalIdentifier, array $data)
    {
        $this->applyProperties($data['properties'], $this->nodeTemplate);

        $node = $this->storageNode->createNodeFromTemplate($this->nodeTemplate);
        $this->registerNodeProcessing($node, $externalIdentifier);
    }

    /**
     * Updates a product
     *
     * @param string $externalIdentifier
     * @param array $data
     * @return void
     */
    protected function updateCommand($externalIdentifier, array $data)
    {
        $this->applyProperties($data['properties'], $this->nodeTemplate);

        $node = $this->storageNode->createNodeFromTemplate($this->nodeTemplate);
        $this->registerNodeProcessing($node, $externalIdentifier);
    }

    /**
     * Deletes a product
     *
     * @param string $externalIdentifier
     * @param array $data
     */
    protected function deleteCommand($externalIdentifier, array $data)
    {
        // delete the product node
    }
}
```

CSV Data Provider
-----------------

[](#csv-data-provider)

This package comes with a basic data provider for CSV files which will suffice for many scenarios. The class name for this data provider is `Ttree\ContentRepositoryImporter\DataProvider\CsvDataProvider`.

The following options can be passed to the data provider:

- `csvFilePath`: the full path and filename leading to the file to import
- `csvDelimiter`: the delimiter used in the CSV file (default: `,`)
- `csvEnclosure`: the character which is used for enclosing the values (default: `"`)
- `skipHeader`: if the first line in the CSV file should be ignored (default: false)

Here is an example for a preset using the CSV Data Provider:

```
Ttree:
  ContentRepositoryImporter:
    presets:
      'products':
        parts:
          'products':
            label: 'Product Import'
            batchSize: 100
            dataProviderClassName: 'Ttree\ContentRepositoryImporter\DataProvider\CsvDataProvider'
            dataProviderOptions:
              csvFilePath: '/tmp/Products.csv'
              csvDelimiter: ';'
              csvEnclosure: '"'
              skipHeader: true
            importerClassName: 'Acme\MyProductImporter\Service\Import\ProductImporter'
            importerOptions:
              siteNodePath: '/sites/wwwacmecom'
```

Acknowledgments
---------------

[](#acknowledgments)

Development sponsored by [ttree ltd - neos solution provider](http://ttree.ch).

We try our best to craft this package with a lots of love, we are open to sponsoring, support request, ... just contact us.

License
-------

[](#license)

Licensed under GPLv3+, see [LICENSE](LICENSE)

###  Health Score

40

—

FairBetter than 87% of packages

Maintenance19

Infrequent updates — may be unmaintained

Popularity34

Limited adoption so far

Community21

Small or concentrated contributor base

Maturity73

Established project with proven stability

 Bus Factor1

Top contributor holds 58.8% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Every ~134 days

Recently: every ~337 days

Total

19

Last Release

1465d ago

Major Versions

2.0.3 → 3.0.02017-07-11

3.1.x-dev → 4.0.02018-08-23

### Community

Maintainers

![](https://www.gravatar.com/avatar/cd3279e45009314122be6b398a54418a037e1b248fb4740cb66bb742d05e3512?d=identicon)[ttree](/maintainers/ttree)

---

Top Contributors

[![dfeyer](https://avatars.githubusercontent.com/u/221173?v=4)](https://github.com/dfeyer "dfeyer (161 commits)")[![daniellienert](https://avatars.githubusercontent.com/u/642226?v=4)](https://github.com/daniellienert "daniellienert (83 commits)")[![robertlemke](https://avatars.githubusercontent.com/u/95582?v=4)](https://github.com/robertlemke "robertlemke (13 commits)")[![lcherpit](https://avatars.githubusercontent.com/u/490499?v=4)](https://github.com/lcherpit "lcherpit (5 commits)")[![andrehoffmann30](https://avatars.githubusercontent.com/u/23524251?v=4)](https://github.com/andrehoffmann30 "andrehoffmann30 (3 commits)")[![remuslazar](https://avatars.githubusercontent.com/u/6108134?v=4)](https://github.com/remuslazar "remuslazar (3 commits)")[![lorenzulrich](https://avatars.githubusercontent.com/u/1816023?v=4)](https://github.com/lorenzulrich "lorenzulrich (2 commits)")[![fnkr](https://avatars.githubusercontent.com/u/616991?v=4)](https://github.com/fnkr "fnkr (2 commits)")[![skurfuerst](https://avatars.githubusercontent.com/u/190777?v=4)](https://github.com/skurfuerst "skurfuerst (1 commits)")[![dlubitz](https://avatars.githubusercontent.com/u/13046100?v=4)](https://github.com/dlubitz "dlubitz (1 commits)")

---

Tags

flowframeworkneoscms

### Embed Badge

![Health badge](/badges/ttree-contentrepositoryimporter/health.svg)

```
[![Health](https://phpackages.com/badges/ttree-contentrepositoryimporter/health.svg)](https://phpackages.com/packages/ttree-contentrepositoryimporter)
```

###  Alternatives

[civicrm/civicrm-core

Open source constituent relationship management for non-profits, NGOs and advocacy organizations.

728272.9k17](/packages/civicrm-civicrm-core)[getdkan/dkan

DKAN Open Data Catalog

385135.4k2](/packages/getdkan-dkan)[verbb/formie

The most user-friendly forms plugin for Craft.

101372.9k40](/packages/verbb-formie)[kaufmanndigital/gdpr-cookieconsent

A ready-to-run package, that integrates an advanced cookie consent banner into your Neos CMS site.

2540.7k](/packages/kaufmanndigital-gdpr-cookieconsent)[shel/neos-colorpicker

A plugin for Neos CMS which provides a colorpicker editor

1494.4k6](/packages/shel-neos-colorpicker)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)