PHPackages                             avtonom/mathielen-import-engine - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Database & ORM](/categories/database)
4. /
5. avtonom/mathielen-import-engine

ActiveLibrary[Database & ORM](/categories/database)

avtonom/mathielen-import-engine
===============================

Full-blown importer stack for importing almost any data into your application

v0.9.3(8y ago)0268MITPHPPHP >=5.5.0

Since Aug 7Pushed 8y ago1 watchersCompare

[ Source](https://github.com/Avtonom/import-engine)[ Packagist](https://packagist.org/packages/avtonom/mathielen-import-engine)[ Docs](https://github.com/Avtonom/import-engine)[ RSS](/packages/avtonom-mathielen-import-engine/feed)WikiDiscussions master Synced yesterday

READMEChangelogDependencies (12)Versions (21)Used By (0)

Mathielen Import Engine
=======================

[](#mathielen-import-engine)

[![Build Status](https://camo.githubusercontent.com/d63dc312c1f00c6b0336b68674dc2c001b111bcb63445c166f81b23d94a521c7/68747470733a2f2f7472617669732d63692e6f72672f6d61746869656c656e2f696d706f72742d656e67696e652e706e673f6272616e63683d6d6173746572)](https://travis-ci.org/mathielen/import-engine)[![Scrutinizer Code Quality](https://camo.githubusercontent.com/d565c71285704bc05a2ee8ed7ce1523e3108dd82dd08fe6f665b08d660a5d862/68747470733a2f2f7363727574696e697a65722d63692e636f6d2f672f6d61746869656c656e2f696d706f72742d656e67696e652f6261646765732f7175616c6974792d73636f72652e706e673f733d63306135356163353338316133663866646163643935656561636335653561643839323636393566)](https://scrutinizer-ci.com/g/mathielen/import-engine/)[![Code Coverage](https://camo.githubusercontent.com/c6e149b3072b88df03be396d52dd42a69aa28d538047627795532f3a1c333890/68747470733a2f2f7363727574696e697a65722d63692e636f6d2f672f6d61746869656c656e2f696d706f72742d656e67696e652f6261646765732f636f7665726167652e706e673f733d35663038336435353030643365633935366435666338366138353730613937653262623963366464)](https://scrutinizer-ci.com/g/mathielen/import-engine/)[![SensioLabsInsight](https://camo.githubusercontent.com/7257aa6e67b6d29100b4ddc207b6567340350359f14bac15aad6b2b036454d05/68747470733a2f2f696e73696768742e73656e73696f6c6162732e636f6d2f70726f6a656374732f39383566373534312d326566392d346239322d393864332d3863663766343134346537342f6d696e692e706e67)](https://insight.sensiolabs.com/projects/985f7541-2ef9-4b92-98d3-8cf7f4144e74)[![Latest Stable Version](https://camo.githubusercontent.com/cbcac5d1c6cd93d50d95601e552a0f73bb36810707aca9c4bea372cd02ea101a/68747470733a2f2f706f7365722e707567782e6f72672f6d61746869656c656e2f696d706f72742d656e67696e652f762f737461626c652e706e67)](https://packagist.org/packages/mathielen/import-engine)

Full-blown importer stack for importing almost any data into your application. Can be used for exports, too.

Introduction
------------

[](#introduction)

This library implements some high-level functionality based on the great [Ddeboer Data Import library](https://github.com/ddeboer/data-import). As the Data-Import library offers a great toolkit to implement a data import/export process in a quick and clean way, there is still a lot of work to do to have a full blown importer set up for your application. This library helps you with that.

If you are developing a Symfony2 project, you may want to use the comfortable configuration of the [ImportEngineBundle](https://github.com/mathielen/ImportEngineBundle).

Features
--------

[](#features)

- A Storage Abstraction-layer that supports nice features like [automatic delimiter-discovering](#automatic-csv-delimiter-discovery-for-filestorageproviders) or processing compressed files. Currently these storages are supported:
    - Structured Files
        - CSV, XML, JSON, Excel
        - File may be compressed
    - Doctrine2 queries
    - Service endpoints
- Storage Provisioning. Provide a list of possible storage-containers for your import. I.e. local files, remote files, uploaded files, database connections, service endpoints and more.
- A mapping sub-system, for building various mappings for your import: field-field, field-converter-field, field-converter-object and more.
- Automatic mapping into object tree's using the [JMSSerializer](http://jmsyst.com/libs/serializer)
- Source (read) and Target (write) validation using [Symfony Validation](http://symfony.com/doc/current/book/validation.html). Annotations can be used.
- Integrated Eventsystem using [Symfony EventDispatcher](http://symfony.com/doc/current/components/event_dispatcher/introduction.html)
- Keeping almost every flexibility that is offered by the Ddeboer Data Import library.
- Well-tested code.

Installation
------------

[](#installation)

This library is available on [Packagist](https://packagist.org/packages/mathielen/import-engine). The recommended way to install it is through [Composer](http://getcomposer.org):

```
$ composer require mathielen/import-engine
```

Then include Composer’s autoloader:

```
require_once 'vendor/autoload.php';
```

If you want to make use of excel files, please also make sure to include phpoffice/phpexcel in your project:

```
$ composer require phpoffice/phpexcel
```

Quickstart
----------

[](#quickstart)

#### Import an arbitrary file into your system

[](#import-an-arbitrary-file-into-your-system)

Using the \*Provider facilities enables you to let the importer-system figure out what format the file has and what abstraction-classes should be used.

```
$service = new TestEntities\Dummy(); //your domain service

$fileStorageProvider = new Mathielen\ImportEngine\Storage\Provider\FileStorageProvider();
$storageSelection = $fileStorageProvider->select('tests/metadata/testfiles/flatdata.csv');
$sourceStorage = $fileStorageProvider->storage($storageSelection);

$targetStorage = new Mathielen\ImportEngine\Storage\ServiceStorage(array($service, 'onNewData'));
$importer = Mathielen\ImportEngine\Importer\Importer::build($targetStorage);

$import = Mathielen\ImportEngine\Import\Import::build($importer, $sourceStorage);

$importRunner = new Mathielen\ImportEngine\Import\Run\ImportRunner();
$importRunner->run($import);
```

#### More examples

[](#more-examples)

Have a look at:

Usage
-----

[](#usage)

### Terminology

[](#terminology)

- An [Importer](#importer) is the basic definition of the whole import-process. It says *what* may be imported and *where* to. It consists of:
    - (optional) A [StorageProvider](#storageprovider), that represents a "virtual file system" for selecting a SourceStorage
    - (optional) A [SourceStorage](#storage) that may be a file, a database table, an array, an object-tree, etc
    - A [TargetStorage](#storage) that may be a file, a database table, an array, an object-tree, etc
    - A [Mapping](#mapping), which may contain converters, field-mappings, etc
    - A [Validation](#validation), that may contain validation-rules for data read from the SourceStorage and/or validation-rules for data that will be written to the TargetStorage.
    - An [Eventsystem](#eventsystem) for implementing detailed [Logging](#eventsystem) or other interactions within the process.
- An [Import](#import) is a specific definition of the import-process. It uses the [Importer](#importer) and has all the specific informations that is mandatory for processing the data. That is a specific SourceStorage and a [Mapping](#mapping).
- The [ImportRunner](#importrunner) is used to process the Import.
- Every run of an Import is represented by an [ImportRun](#importrun)

### StorageProvider

[](#storageprovider)

StorageProviders represents a "virtual file system" for selecting a [SourceStorage](#storage) that can be used as a source or target of the import.

#### FinderFileStorageProvider

[](#finderfilestorageprovider)

Using the [Symfony Finder Component](http://symfony.com/doc/current/components/finder.html) as a collection of possible files that can be imported.

```
use Symfony\Component\Finder\Finder;
use Mathielen\ImportEngine\Storage\Provider\FinderFileStorageProvider;

$finder = Finder::create()
  ->in('tests/metadata/testfiles')
  ->name('*.csv')
  ->name('*.tab')
  ->size('>0K')
;

$ffsp = new FinderFileStorageProvider($finder);
```

#### DoctrineQueryStorageProvider

[](#doctrinequerystorageprovider)

You can use specific Doctrine Queries or only Entity-Classnames (the query will be SELECT \* FROM then) as possible Source-Storages.

```
use Symfony\Component\Finder\Finder;
use Mathielen\ImportEngine\Storage\Provider\DoctrineQueryStorageProvider;

$em = ... //Doctrine2 EntityManager
$qb = $em->createQueryBuilder()
  ->select('a')
  ->from('MySystem\Entity\Address', 'a')
  ->andWhere('a.id > 10')
;

$queries = array(
  'MySystem/Entity/MyEntity',
  $qb
);

$desp = new DoctrineQueryStorageProvider($em, $queries);
```

#### UploadFileStorageProvider

[](#uploadfilestorageprovider)

You can use a Provider to facilitate a File-Upload.

```
use Mathielen\ImportEngine\Storage\Provider\UploadFileStorageProvider;

$ufsp = new UploadFileStorageProvider('/tmp'); //path to where the uploaded files will be transferred to
```

#### Automatic CSV Delimiter Discovery for FileStorageProviders

[](#automatic-csv-delimiter-discovery-for-filestorageproviders)

FileStorageProviders may use StorageFactories for constructing Storage objects. By default the FormatDiscoverLocalFileStorageFactory is used. This StorageFactory uses a MimeTypeDiscoverStrategy to determine the mime-type of the selected file and use it to create the correct storage-handler. You can change this behavior or extend it. There is a CsvAutoDelimiterTypeFactory that you can use to automaticly guess the correct delimiter of a CSV file.

```
use Mathielen\ImportEngine\Storage\Format\Factory\CsvAutoDelimiterFormatFactory;
use Mathielen\ImportEngine\Storage\Factory\FormatDiscoverLocalFileStorageFactory;
use Mathielen\ImportEngine\Storage\Format\Discovery\MimeTypeDiscoverStrategy;

$ffsp = ...
$ffsp->setStorageFactory(
  new FormatDiscoverLocalFileStorageFactory(
    new MimeTypeDiscoverStrategy(array(
      'text/plain' => new CsvAutoDelimiterFormatFactory()
))));
```

This way any file that has the text/plain mime-type will be passed to the CsvAutoDelimiterFormatFactory to determine the delimiter.

### Storage

[](#storage)

A storage is a container of data. Storages provide a reader and writer implementation for itself.

```
use Mathielen\ImportEngine\Storage\ArrayStorage;
use Mathielen\ImportEngine\Storage\DoctrineStorage;
use Mathielen\ImportEngine\Storage\LocalFileStorage;
use Mathielen\ImportEngine\Storage\Format\CsvFormat;

$em = ... //Doctrine2 EntityManager

$array = array(1,2,3);
$storage = new ArrayStorage($array);
$storage = new DoctrineStorage($em, 'MyEntities\Entity');
$storage = new LocalFileStorage('tests/metadata/testfiles/flatdata.csv', new CsvFormat());
$storage = new ServiceStorage(array($service, 'myMethod')); //callable
```

### Validation

[](#validation)

You can get the source and target validation errors with:

```
$import = ...
$import->importer()->validation()->getViolations();
```

#### Source data validation

[](#source-data-validation)

```
use Mathielen\ImportEngine\Validation\ValidatorValidation;
use Mathielen\DataImport\Filter\ClassValidatorFilter;
use Symfony\Component\Validator\Constraints\NotBlank;
use Symfony\Component\Validator\Constraints\Regex;

$validator = ... //Symfony Validator

$validation = ValidatorValidation::build($validator)
  ->addSourceConstraint('salutation', new NotBlank()) //source field 'salutation' should not be empty
  ->addSourceConstraint('zipcode', new Regex("/[0-9]{5}/")) //source field 'zipcode' should be 5 digits
;
```

#### Target data Validation

[](#target-data-validation)

##### ClassValidatorFilter

[](#classvalidatorfilter)

You can use the ClassValidatorFilter to map the data to an object-tree and validate the objects (using annotations, or [differently configurated validation rules](http://symfony.com/doc/current/book/validation.html#constraint-configuration)). Therefore you must provide an ObjectFactory. There is a JmsSerializerObjectFactory you may want to use.

```
use Mathielen\ImportEngine\Validation\ValidatorValidation;
use Mathielen\DataImport\Filter\ClassValidatorFilter;
use Mathielen\DataImport\Writer\ObjectWriter\JmsSerializerObjectFactory;

$validator = ... //Symfony Validator

$jms_serializer = ...
$objectFactory = new JmsSerializerObjectFactory(
  'Entity\Address',
  $jms_serializer);

$validation = ValidatorValidation::build($validator)
  ->setTargetValidatorFilter(new ClassValidatorFilter($validator, $objectFactory));
```

### Importer

[](#importer)

```
use Mathielen\ImportEngine\Importer\Importer;
use Mathielen\ImportEngine\Storage\ArrayStorage;

$ffsp = ...
$validation = ...
$targetStorage = ...

$array = array(1,2,3);
$importer = Importer::build($targetStorage)
  ->setSourceStorage(new ArrayStorage($array))
  ->validation($validation)
;
```

### Import / Source Storage

[](#import--source-storage)

You can either use a StorageProvider (see above) and set the selection-id or you can use a specific Storage-Handler directly:

```
use Mathielen\ImportEngine\Storage\ArrayStorage;
use Mathielen\ImportEngine\Storage\LocalFileStorage;
use Mathielen\ImportEngine\Import\Import;
use Mathielen\ImportEngine\Importer\Importer;
use Mathielen\ImportEngine\Storage\Format\CsvFormat;

$targetArray = array();
$importer = Importer::build(new ArrayStorage($targetArray));
$import = Import::build(
    $importer,
    new LocalFileStorage(new \SplFileObject(__DIR__ . '/../../../metadata/testfiles/flatdata.csv'), new CsvFormat())
);
```

### Mapping

[](#mapping)

Also see orginal documentation [here](https://github.com/ddeboer/data-import#mappingitemconverter).

```
$import = ...

$import->mappings()
  ->add('foo', 'fooloo')
  ->add('baz', array('some' => 'else'));
;
```

### Converting fields

[](#converting-fields)

There are a some field-level build-in converters available:

- upperCase
- lowerCase
- @TODO

```
$import = ...

$import->mappings()
  ->add('SALUTATION_FIELD', 'salutation', 'upperCase')
;
```

### Custom fieldlevel-converting

[](#custom-fieldlevel-converting)

You have to register more complex converters to the importer for selecting them in your import.

```
use Mathielen\ImportEngine\Mapping\Converter\Provider\DefaultConverterProvider;
use Ddeboer\DataImport\ValueConverter\CallbackValueConverter;
use Mathielen\ImportEngine\Import\Import;
use Mathielen\ImportEngine\Storage\ArrayStorage;
use Mathielen\ImportEngine\Importer\Importer;

$converterProvider = new DefaultConverterProvider();
$converterProvider
  ->add('salutationToGender', new CallbackValueConverter(function ($item) {
      switch ($item) {
        case 'Mr.': return 'male';
        case 'Miss':
        case 'Mrs.': return 'femaile';
      }
  }));

$targetStorage = ...

$importer = Importer::build($targetStorage);
$importer
  ->transformation()
  ->setConverterProvider($converterProvider);

$array = array();
$import = Import::build($importer, new ArrayStorage($array))
  ->mappings()
  ->add('salutation', 'gender', 'salutationToGender')
;
```

### Custom rowlevel-converting

[](#custom-rowlevel-converting)

Like the fieldlevel converters, you have to register your converters first.

```
use Mathielen\ImportEngine\Mapping\Converter\Provider\DefaultConverterProvider;
use Ddeboer\DataImport\ItemConverter\CallbackItemConverter;
use Mathielen\ImportEngine\Import\Import;
use Mathielen\ImportEngine\Storage\ArrayStorage;
use Mathielen\ImportEngine\Importer\Importer;

$converterProvider = new DefaultConverterProvider();
$converterProvider
  ->add('splitNames', new CallbackItemConverter(function ($item) {
      list($firstname, $lastname) = explode(' ', $item['name']);

      $item['first_name'] = $firstname;
      $item['lastname'] = $lastname;

      return $item;
  }));

$targetStorage = ...

$importer = Importer::build($targetStorage);
$importer
  ->transformation()
  ->setConverterProvider($converterProvider);

$array = array();
$import = Import::build($importer, new ArrayStorage($array))
  ->mappings()
  ->add('fullname', null, 'splitNames')
;
```

### ImportRunner

[](#importrunner)

For running a configured Import you need an ImportRunner. Internally the ImportRunner builds a workflow and runs it. You can change the way how the workflow is built by supplying a different WorkflowFactory.

```
use Symfony\Component\EventDispatcher\EventDispatcher;
use Mathielen\ImportEngine\Import\Run\ImportRunner;
use Mathielen\ImportEngine\Import\Workflow\DefaultWorkflowFactory;
use Mathielen\ImportEngine\ValueObject\ImportConfiguration;
use Mathielen\ImportEngine\Storage\LocalFileStorage;
use Mathielen\ImportEngine\Storage\Format\CsvFormat;
use Mathielen\ImportEngine\Importer\ImporterRepository;

$import = ...

$importRunner = new ImportRunner(new DefaultWorkflowFactory(new EventDispatcher()));

//sneak peak a row
$previewData = $importRunner->preview($import);

//dont really write, just validate
$importRun = $importRunner->dryRun($import);

//do the import
$importRun = $importRunner->run($import);
```

### ImportRun statistics

[](#importrun-statistics)

If you use the DefaultWorkflowFactory with your ImportRunner you get basic statistics from dryRun() and run() invocations.

```
$importRun = ...
$importRunner = ...

$importRunner->dryRun($import);
$stats = $importRun->getStatistics();

/*
Array
(
    [processed] => 1
    [written] => 1
    [skipped] => 0
    [invalid] => 0
)
*/
```

### Eventsystem

[](#eventsystem)

You can interact with the running import via the [Symfony Eventdispatcher](http://symfony.com/doc/current/components/event_dispatcher/introduction.html).

```
use Symfony\Component\EventDispatcher\EventDispatcher;
use Mathielen\ImportEngine\Import\Run\ImportRunner;
use Mathielen\DataImport\Event\ImportProcessEvent;
use Mathielen\DataImport\Event\ImportItemEvent;
use Mathielen\ImportEngine\Import\Workflow\DefaultWorkflowFactory;

$myListener = function ($event) {
    if ($event instanceof ImportItemEvent) {
    	$currentResult = $event->getCurrentResult(); //readonly access to current result in the process (might be false)
    }
};

$eventDispatcher = new EventDispatcher();
$eventDispatcher->addListener(ImportProcessEvent::AFTER_PREPARE, $myListener);
$eventDispatcher->addListener(ImportItemEvent::AFTER_READ, $myListener);
$eventDispatcher->addListener(ImportItemEvent::AFTER_FILTER, $myListener);
$eventDispatcher->addListener(ImportItemEvent::AFTER_CONVERSION, $myListener);
$eventDispatcher->addListener(ImportItemEvent::AFTER_CONVERSIONFILTER, $myListener);
$eventDispatcher->addListener(ImportItemEvent::AFTER_VALIDATION, $myListener);
$eventDispatcher->addListener(ImportItemEvent::AFTER_WRITE, $myListener);
$eventDispatcher->addListener(ImportProcessEvent::AFTER_FINISH, $myListener);

$workflowFactory = new DefaultWorkflowFactory($eventDispatcher);
$importRunner = new ImportRunner($workflowFactory);

$import = ...
$importRunner->run($import);
```

License
-------

[](#license)

Import-Engine is released under the MIT license. See the [LICENSE](LICENSE) file for details.

###  Health Score

26

—

LowBetter than 43% of packages

Maintenance20

Infrequent updates — may be unmaintained

Popularity11

Limited adoption so far

Community7

Small or concentrated contributor base

Maturity57

Maturing project, gaining track record

 Bus Factor1

Top contributor holds 100% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Every ~63 days

Recently: every ~145 days

Total

18

Last Release

3219d ago

PHP version history (2 changes)0.1PHP >=5.4.0

v0.9.0PHP >=5.5.0

### Community

Maintainers

![](https://www.gravatar.com/avatar/851a2d4f8d02604c04bd2c4331a901e4fc92ed7ec1371cbceafb48c2495efc8c?d=identicon)[avtonomspb](/maintainers/avtonomspb)

---

Top Contributors

[![mathielen](https://avatars.githubusercontent.com/u/1571485?v=4)](https://github.com/mathielen "mathielen (155 commits)")

---

Tags

xmldatayamldoctrineexcelcsvserviceimportmappingjmsimporter

###  Code Quality

TestsPHPUnit

### Embed Badge

![Health badge](/badges/avtonom-mathielen-import-engine/health.svg)

```
[![Health](https://phpackages.com/badges/avtonom-mathielen-import-engine/health.svg)](https://phpackages.com/packages/avtonom-mathielen-import-engine)
```

###  Alternatives

[mathielen/import-engine

Full-blown importer stack for importing almost any data into your application

2711.9k2](/packages/mathielen-import-engine)[ddeboer/data-import

Import data from, and export data to, a range of file formats and media

5604.3M9](/packages/ddeboer-data-import)[portphp/portphp

Data import/export workflow

2702.9M22](/packages/portphp-portphp)[cyber-duck/laravel-excel

This package provides a way to export an Eloquent collection as an excel file and to import a Excel file as an Eloquent collection.

74225.0k](/packages/cyber-duck-laravel-excel)[dracoblue/craur

A lossless xml to json and json to xml converter (and csv/xlsx/yaml). Writing PHP Json/Xml/Csv/Yaml/excel Importers made easy

4643.1k2](/packages/dracoblue-craur)[archon/dataframe

Archon: PHP Data Analysis Library

9824.2k1](/packages/archon-dataframe)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)