PHPackages pbritka/m2-module-import - PHPackages - PHPackages [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register) 1. [Directory](/) 2. / 3. pbritka/m2-module-import ActiveMagento2-module pbritka/m2-module-import ======================== Import module for Magento 2 - provides commands, utilities and abstractions for building imports for Magento 2 projects. Forked from WeareJH/Import. 2.7.0(8mo ago)0159PHPPHP ^7.4 || ^8.0 Since Jun 14Pushed 4mo agoCompare [ Source](https://github.com/pbritka/m2-module-import)[ Packagist](https://packagist.org/packages/pbritka/m2-module-import)[ RSS](/packages/pbritka-m2-module-import/feed)WikiDiscussions master Synced 1mo ago READMEChangelogDependencies (15)Versions (38)Used By (0) Magento 2 Import ================ [](#magento-2-import) [ ![](https://camo.githubusercontent.com/ec050b99e7b6f815445ed4b808df99d291f3cfa064000d5c768ad59ccf0795a0/68747470733a2f2f696d672e736869656c64732e696f2f7472617669732f57656172654a482f6d322d6d6f64756c652d696d706f72742f6d61737465722e7376673f7374796c653d666c61742d737175617265266c6162656c3d4c696e7578) ](https://travis-ci.org/WeareJH/m2-module-import "Build Status") [ ![](https://camo.githubusercontent.com/65f0c22341544610bc415b4577247a8d52d63e79ee853670494931fca3ed93c3/68747470733a2f2f696d672e736869656c64732e696f2f636f6465636f762f632f6769746875622f57656172654a482f6d322d6d6f64756c652d696d706f72742e7376673f7374796c653d666c61742d737175617265) ](https://codecov.io/github/WeareJH/m2-module-import "Coverage Status") Import module for Magento 2 - provides commands, logs, reporting, utilities and abstractions for building imports for Magento 2 projects . Installation ------------ [](#installation) ``` $ composer config repositories.jh-import vcs git@github.com:WeareJH/m2-module-import.git $ composer require wearejh/m2-module-import $ php bin/magento setup:upgrade ``` Documentation ------------- [](#documentation) Table of Contents ----------------- [](#table-of-contents) - [Creating a new import](#creating-a-new-import) - [Create a module](#create-a-module) - [Define the import configuration](#define-the-import-configuration) - [Files import type](#files-import-type) - [source](#source) - [incoming\_directory](#incoming_directory) - [archived\_directory](#archived_directory) - [failed\_directory](#failed_directory) - [match\_files](#match_files) - [specification](#specification) - [writer](#writer) - [id\_field](#id_field) - [DB import type](#db-import-type) - [source](#source) - [connection\_name](#connection_name) - [specification](#specification) - [writer](#writer) - [id\_field](#id_field) - [source\_id](#source_id) - [select\_sql](#select_sql) - [count\_sql](#count_sql) - [Webapi import type](#webapi-import-type) - [source](#source) - [specification](#specification) - [writer](#writer) - [id\_field](#id_field) - [source\_id](#source_id) - [Create the specification](#create-the-specification) - [Transformers](#transformers) - [Filters](#filters) - [Create the writer](#create-the-writer) - [Indexing](#indexing) - [Report Handlers](#report-handlers) - [Bundled Handlers](#bundled-handlers) - [Usage](#bundled-handler-usage) - [Database Handler](#jhimportreporthandlerdatabasehandler) - [Console Handler](#jhimportreporthandlerconsolehandler) - [Mail Handler](#jhimportreporthandleremailhandler) - [Creating custom handlers](#creating-custom-handlers) - [Triggering an import](#triggering-an-import) - [Running an import manually](#running-an-import-manually) - [Where do the files go?](#where-do-the-files-go) - [Viewing Import Configuration](#viewing-import-configuration) - [Listing Imports](#listing-imports) - [Viewing an Import](#viewing-an-import) - [Check Incoming Files](#check-incoming-files) - [Downloading Files](#downloading-files) - [Deleteing Files](#deleting-files) - [Viewing import logs](#viewing-import-logs) - [Item level logs](#item-level-logs) - [Import level logs](#import-level-logs) - [Sequence detection](#sequence-detection) - [Force running the same source again](#force-running-the-same-source-again) - [Import Locking](#import-locking) - [Archiving](#archiving) - [What causes an import to be failed?](#what-causes-an-import-to-be-failed) - [File Cleanup](#file-cleanup) - [Zip Cron](#zip-cron) - [Delete Cron](#zip-cron) Creating a new Import --------------------- [](#creating-a-new-import) An import consists of a bunch of components: - A source - Responsible for reading the data - A specification - Responsible for transforming and filtering the data - A Writer - Responsible for saving the data In this section we will create an import named `price`. ### Create a module [](#create-a-module) The first step of using the import module is to create a new module in your project to contain the import configuration and any code you may have to write. Go ahead and create that - our convention is `Vendor/Import`. We will create an example module in this documentation to help show the concepts. We will use [N98 magerun2](https://github.com/netz98/n98-magerun2) to do this: ``` $ n98 dev:module:create MyVendor Import ``` Delete the `events.xml` & `crontab.xml` files from the modules `etc` directory, we don't need those. ### Define the import configuration [](#define-the-import-configuration) Define the import in your module in an `imports.xml` file: In our example that would be `app/code/MyVendor/Import/etc/imports.xml` ``` ``` The `files` key here represents the type of import. In this case a files type. Supported import types: - `files` - `db` - `webapi` You can see the supported types in the `$types` property of [src/Import/Manager.php](src/Import/Manager.php). ### Files import type [](#files-import-type) The `name` attribute is the unique name for your import and is how you execute it from `\Jh\Import\Import\Manager`. Here we named it `price`. The required configuration values for your import are: - source - match\_files - specification - writer - id\_field Optional values are: - incoming\_directory - default: `jh_import/incoming` - archived\_directory - default: `jh_import/archived` - failed\_directory - default: `jh_import/failed` - archive\_already\_imported\_files - default: `false` - process\_only\_last\_file - default: `false` The above directories will be created in the `var` directory of the magento instance if they do not already exist. #### source [](#source) This should be the name of a class or a virtual type. It should point to a class which implements `Jh\Import\Source\Source`. The only source type currently available is `Jh\Import\Source\Csv` for reading CSV files. You can customise the constructor arguments by using a virtual type. #### incoming\_directory [](#incoming_directory) If this value is left unspecified it will default to: `jh_import/incoming` This is where your files will be read from. This directory will be prefixed with the absolute path of the Magento installations `var` directory. For example, you specify: `vendor_import/incoming` translates to: `/var/www/magento/var/vendor_import/incoming` #### archived\_directory [](#archived_directory) If this value is left unspecified it will default to: `jh_import/archived` This is where your files will be moved to if they successfully import. This directory will be prefixed with the absolute path of the Magento installations `var` directory. For example, you specify: `vendor_import/archived` translates to: `/var/www/magento/var/vendor_import/archived` #### failed\_directory [](#failed_directory) If this value is left unspecified it will default to: `jh_import/failed` This is where your files will be moved to if they fail to import. This directory will be prefixed with the absolute path of the Magento installations `var` directory. For example, you specify: `vendor_import/failed` translates to: `/var/www/magento/var/vendor_import/failed` #### match\_files [](#match_files) Here you can configure which files should be read from the directory based on their name, the values can be: - `*` - read all files in the directory - a regex, for example: `/[a-z]+\.csv/` - read all files which have a csv extension and only contain lower case alpha chars in the name - a static name, for example `stock.csv` - read only a file named `stock.csv`, if it exists #### specification [](#specification) This should be the name of a class or a virtual type. It should point to a class which implements `Jh\Import\Specification\ImportSpecification`. This is the class which takes care of manipulating the incoming data in to a generic format. This will be explained in more detail later on. #### writer [](#writer) This should be the name of a class or a virtual type. It should point to a class which implements `Jh\Import\Writer\Writer`. This is the class which takes care of saving the data in to Magento. This will be explained in more detail later on. #### id\_field [](#id_field) This should be the name of a field that exists in every row of the data and is unique. This is used for logging purposes. For example it would probably be `sku` for a product import. The finished config my look like: ``` Jh\Import\Source\Csv jh_import/incoming /price_\d{8}.csv/ MyVendor\Import\Specification\Price MyVendor\Import\Writer\Price sku ``` #### archive\_already\_imported\_files [](#archive_already_imported_files) By enabling this option all files reported as already imported will be moved into `failed_directory`instead of being left in `incoming_directory`. #### process\_only\_last\_file [](#process_only_last_file) This option enabled will process only last file from matched batch of files. Useful when you have multiple files of same content and you want to save your processing time by importing data only from last one of them. Other matched files from batch will be moved into `failed_directory`. ### DB import type [](#db-import-type) The `name` attribute is the unique name for your import and is how you execute it from `\Jh\Import\Import\Manager`. Here we named it `price`. The required configuration values for your import are: - connection\_name - source - specification - writer - id\_field - source\_id - select\_sql - count\_sql #### connection\_name [](#connection_name) Name of the DB connection to use. The connection should be specified in `app/etc/env.php`. E.g. ``` 'db' => [ 'table_prefix' => '', 'connection' => [ 'default' => [ 'host' => 'db', 'dbname' => 'docker', 'username' => 'docker', 'password' => 'docker', 'model' => 'mysql4', 'engine' => 'innodb', 'initStatements' => 'SET NAMES utf8;', 'active' => '1', 'driver_options' => [ 1014 => false ] ], 'migration' => [ 'host' => 'db', 'dbname' => 'migration', 'username' => 'root', 'password' => 'docker', 'model' => 'mysql4', 'engine' => 'innodb', 'initStatements' => 'SET NAMES utf8;', 'active' => '1', 'driver_options' => [ 1014 => false ] ] ] ], 'resource' => [ 'default_setup' => [ 'connection' => 'default' ], 'migration' => [ 'connection' => 'migration' ] ], ``` #### source [](#source-1) This should be the name of a class or a virtual type. It should point to a class which implements `Jh\Import\Source\Source`. The core source type currently available is `Jh\Import\Source\Db` for reading data from a database. It can be extended further to customise it. #### specification [](#specification-1) This should be the name of a class or a virtual type. It should point to a class which implements `Jh\Import\Specification\ImportSpecification`. This is the class which takes care of manipulating the incoming data in to a generic format. This will be explained in more detail later on. #### writer [](#writer-1) This should be the name of a class or a virtual type. It should point to a class which implements `Jh\Import\Writer\Writer`. This is the class which takes care of saving the data in to Magento. This will be explained in more detail later on. #### id\_field [](#id_field-1) This should be the name of a field that exists in every row of the data and is unique. This is used for logging purposes. For example it would probably be `sku` for a product import. #### source\_id [](#source_id) A unique ID for the import type. #### select\_sql [](#select_sql) SQL query used for loading data from the DB. If you need a more advanced way of loading data, overwrite the source class. #### count\_sql [](#count_sql) SQL query used for counting the data to update. ================================================================================================================== The finished config could look like: ``` migration Jh\Import\Source\Db MyVendor\Import\Specification\Customer MyVendor\Import\Writer\Customer id_customer customer_migration SELECT * FROM ps_customer SELECT COUNT(1) FROM ps_customer customer_grid ``` ### Webapi import type [](#webapi-import-type) The `name` attribute is the unique name for your import and is how you execute it from `\Jh\Import\Import\Manager`. Here we named it `test`. The configuration values for your import are: - source - specification - writer - id\_field - source\_id - count\_request\_factory - count\_response\_handler - data\_request\_factory - data\_request\_paging\_decorator - data\_request\_filter\_decorator - data\_request\_page\_decorator - data\_request\_filter\_decorator (optional) - data\_request\_response\_handler #### source [](#source-2) This should be the name of a class or a virtual type. It should point to a class which implements `Jh\Import\Source\Source`. The core source type currently available is `Jh\Import\Source\Webapi` for reading data from a database. It can be extended further to customise it. #### specification [](#specification-2) This should be the name of a class or a virtual type. It should point to a class which implements `Jh\Import\Specification\ImportSpecification`. This is the class which takes care of manipulating the incoming data in to a generic format. This will be explained in more detail later on. #### writer [](#writer-2) This should be the name of a class or a virtual type. It should point to a class which implements `Jh\Import\Writer\Writer`. This is the class which takes care of saving the data in to Magento. This will be explained in more detail later on. #### id\_field [](#id_field-2) This should be the name of a field that exists in every row of the data and is unique. This is used for logging purposes. For example it would probably be `sku` for a product import. #### source\_id [](#source_id-1) A unique ID for the import type. #### count\_request\_factory [](#count_request_factory) This should be a class implementing `Jh\Import\Source\Webapi\RequestFactoryInterface`. It should return object of `Psr\Http\Message\RequestInterface` built to be sent to get entities amount. This building process is encapsulated inside object, so it can be implemented inside migration module and prevent changing abstraction by import module (open-closed principle). #### count\_response\_handler [](#count_response_handler) This should be a class implementing `Jh\Import\Source\Webapi\CountResponseHandlerInterface`. It should return integer telling how many entites do we have to migrate. The handling process is encapsulated inside object, so it can be implemented inside migration module and prevent changing abstraction by import module. (open-closed principle) #### data\_request\_factory [](#data_request_factory) This should be a class implementing `Jh\Import\Source\Webapi\RequestFactoryInterface`. It should return object of `Psr\Http\Message\RequestInterface` built to be sent to get actual data. This building process is encapsulated inside object, so it can be implemented inside migration module and prevent changing abstraction by import module (open-closed principle). #### data\_request\_paging\_decorator [](#data_request_paging_decorator) This should be a class implementing `Jh\Import\Source\Webapi\DataRequest\PagingDecoratorInterface`. It should return object of `Psr\Http\Message\RequestInterface` built which is modified to contain paging parameters. This building process is encapsulated inside object, so it can be implemented inside migration module and prevent changing abstraction by import module (open-closed principle). #### data\_request\_filter\_decorator (optional) [](#data_request_filter_decorator-optional) This should be a class implementing `Jh\Import\Source\Webapi\DataRequest\FilterDecoratorInterface`. It should return object of `Psr\Http\Message\RequestInterface` built which is modified to contain filtering parameters. This building process is encapsulated inside object, so it can be implemented inside migration module and prevent changing abstraction by import module (open-closed principle). #### data\_request\_page\_size [](#data_request_page_size) This should be integer telling how many entities should be retrieved in one page from webapi. #### data\_response\_handler [](#data_response_handler) This should be a class implementing `Jh\Import\Source\Webapi\DataResponseHandlerInterface`. It should return iterable, so data can be iterated and persisted into db. The handling process is encapsulated inside object, so it can be implemented inside migration module and prevent changing abstraction by import module. (open-closed principle). ================================================================================================================== The finished config could look like: ``` Jh\Import\Source\Webapi test_migration MyVendor\Import\Specification\Test MyVendor\Import\Writer\Test id MyVendor\Import\Specification\Request\CountRequestFactory MyVendor\Import\Specification\Response\CountResponseHandler MyVendor\Import\Specification\Request\DataRequestFactory MyVendor\Import\Specification\Request\DataRequest\PagingDecorator MyVendor\Import\Specification\Request\DataRequest\FilterDecorator 250 MyVendor\Import\Specification\Response\DataResponseHandler ``` Example module can be found in [examples](examples/webapi). ### Create the specification [](#create-the-specification) Create the class from the configuration we just mentioned: `MyVendor\Import\Specification\Price`. This class should implement `Jh\Import\Specification\ImportSpecification`. There is one method which should be fulfilled as part of the contract: `public function configure(Importer $import);`This method allows you to configure the importer by adding transformers and filters. A transformer and a filter are simply PHP callables. #### Transformers [](#transformers) A transformer is used to rename columns, manipulate data and organise it. For example if the sku column comes in as `Product_Number`from the CSV file (`Product_Number` would be the column header in the CSV), you might want to rename it to `sku` so your writer accepts generic data. You could do this like: ```