PHPackages                             keboola/db-import-export - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Database & ORM](/categories/database)
4. /
5. keboola/db-import-export

ActiveLibrary[Database & ORM](/categories/database)

keboola/db-import-export
========================

Package allows to import files to Snowflake from multiple cloud storages

4.5.0(1mo ago)118.9k↓12.3%[5 issues](https://github.com/keboola/php-db-import-export/issues)[1 PRs](https://github.com/keboola/php-db-import-export/pulls)3MITPHPPHP ^8.3CI failing

Since Nov 12Pushed 1mo ago14 watchersCompare

[ Source](https://github.com/keboola/php-db-import-export)[ Packagist](https://packagist.org/packages/keboola/db-import-export)[ RSS](/packages/keboola-db-import-export/feed)WikiDiscussions main Synced 1mo ago

READMEChangelog (10)Dependencies (44)Versions (293)Used By (3)

DB Import export library
========================

[](#db-import-export-library)

Supported operations
--------------------

[](#supported-operations)

- Load/Import csv from `ABS` to `Snowflake`
- Load/Import csv from `GCS` to `Bigquery`
- Unload/Export table from `Snowflake` to `ABS`

Features
--------

[](#features)

### Import

[](#import)

- Full load - destination table is truncated before load
- Incremental load - data are merged
- Primary key dedup for all engines
- Convert empty values to NULL (using convertEmptyValuesToNull option)

Export
------

[](#export)

- Full unload - destination csv is always rewriten

Development
-----------

[](#development)

### Docker

[](#docker)

Prepare `.env` (copy of `.env.dist`) and set up AWS keys which has access to `keboola-drivers` bucket in order to build this image. Also add this user to group `ci-php-import-export-lib` witch will allow you to work with newly created bucket for tests.

User can be created in `Dev - Main legacy`, where are also groups for `keboola-drivers` and `ci-php-import-export-lib`.

Then run `docker compose build`

The AWS credentials have to also have access to bucket specified in `AWS_S3_BUCKET`. This bucket has to contain testing data. Run `docker compose run --rm dev  composer loadS3` to load them up.

### Preparation

[](#preparation)

#### AWS

[](#aws)

Run terraform to create resources for tests. See [Terraform.md](./docs/Terraform.md) for details.

#### Azure

[](#azure)

- Create [storage account](https://portal.azure.com/#create/Microsoft.StorageAccount-ARM) template can be found in provisioning ABS [create template](https://portal.azure.com/#create/Microsoft.Template)
- Create container in storage account `Blob service -> Containers` *note: for tests this step can be skiped container is created with `loadAbs` cmd*
- Fill env variables in .env file

```
ABS_ACCOUNT_NAME=storageAccount
ABS_ACCOUNT_KEY=accountKey
ABS_CONTAINER_NAME=containerName

```

- Upload test fixtures to ABS `docker compose run --rm dev composer loadAbs`

#### Google cloud storage

[](#google-cloud-storage)

- Create bucket in [GCS](https://console.cloud.google.com/storage) set bucket name in .env variable `GCS_BUCKET_NAME`
- Create service account in [IAM](https://console.cloud.google.com/iam-admin/serviceaccounts)
- In bucket permissions grant service account admin access to bucket
- Create new service account key
- Convert key to string `awk -v RS= '{$1=$1}1' .json >> .env` (or `cat file.json | jq -c | jq -R`)
- Set content on last line of .env as variable `GCS_CREDENTIALS`
- Upload test fixtures to GCS `docker compose run --rm dev composer loadGcs-bigquery` or `docker compose run --rm dev composer loadGcs-snowflake` (depending on backend)

#### SNOWFLAKE

[](#snowflake)

Role, user, database and warehouse are required for tests. You can create them:

```
CREATE ROLE "KEBOOLA_DB_IMPORT_EXPORT";
CREATE DATABASE "KEBOOLA_DB_IMPORT_EXPORT";

GRANT ALL PRIVILEGES ON DATABASE "KEBOOLA_DB_IMPORT_EXPORT" TO ROLE "KEBOOLA_DB_IMPORT_EXPORT";
GRANT USAGE ON WAREHOUSE "DEV" TO ROLE "KEBOOLA_DB_IMPORT_EXPORT";

CREATE USER "KEBOOLA_DB_IMPORT_EXPORT"
PASSWORD = 'Password'
DEFAULT_ROLE = "KEBOOLA_DB_IMPORT_EXPORT";

GRANT ROLE "KEBOOLA_DB_IMPORT_EXPORT" TO USER "KEBOOLA_DB_IMPORT_EXPORT";

-- For GCS create storage integration https://docs.snowflake.com/en/user-guide/data-load-gcs-config.html#creating-a-custom-iam-role
CREATE STORAGE INTEGRATION "KEBOOLA_DB_IMPORT_EXPORT"
  TYPE = EXTERNAL_STAGE
  STORAGE_PROVIDER = GCS
  ENABLED = TRUE
  STORAGE_ALLOWED_LOCATIONS = ('gcs:///');
-- set integration name to env GCS_INTEGRATION_NAME in .env file
-- get service account id `STORAGE_GCP_SERVICE_ACCOUNT`
DESC STORAGE INTEGRATION "KEBOOLA_DB_IMPORT_EXPORT";
-- continue according manual ^ in snflk documentation assign roles for Data loading and unloading
```

#### Bigquery

[](#bigquery)

Install [Google Cloud client](https://cloud.google.com/sdk/docs/install-sdk) (via [Brew](https://formulae.brew.sh/cask/google-cloud-sdk#default)), initialize it and log in to [generate default credentials](https://cloud.google.com/docs/authentication/application-default-credentials#personal).

To prepare the backend you can use [Terraform template](./bq-storage-backend-init.tf). You must have the `resourcemanager.folders.create` permission for the organization.

```
# you can copy it to a folder somewhere and make an init
terraform init
```

Run `terraform apply` with following variables:

- folder\_id: Go to [GCP Resource Manager](https://console.cloud.google.com/cloud-resource-manager) and select your team dev folder ID (e.g. find 'KBC Team Dev' and copy ID)
- backend\_prefix: your\_name, all resources will create with this prefix
- billing\_account\_id: Go to [Billing](https://console.cloud.google.com/billing/) and copy your Billing account ID

```
terraform apply -var folder_id= -var backend_prefix= -var billing_account_id=
```

For missing pieces see [Connection repository](https://github.com/keboola/connection/blob/master/DOCKER.md#bigquery). After terraform apply ends go to the service project in folder created by terraform.

1. convert key to string and save to `.env` file: `awk -v RS= '{$1=$1}1' .json >> .env`
2. set content on the last line of `.env` as variable `BQ_KEY_FILE`
3. set env variable `BQ_BUCKET_NAME` generated from TF template `file_storage_bucket_id`

### Tests

[](#tests)

Run tests with following command.

*note: azure credentials must be provided and fixtures uploaded*

```
docker compose run --rm dev composer tests

```

Unit and functional test can be run sepparetly

```
#unit test
docker compose run --rm dev composer tests-unit

#functional test
docker compose run --rm dev composer tests-functional

```

### Code quality check

[](#code-quality-check)

```
#phplint
docker compose run --rm dev composer phplint

#phpcs
docker compose run --rm dev composer phpcs

#phpstan
docker compose run --rm dev composer phpstan

```

### Full CI workflow

[](#full-ci-workflow)

This command will run all checks load fixtures and run tests

```
docker compose run --rm dev composer ci

```

### Usage

[](#usage)

#### Snowflake

[](#snowflake-1)

ABS -> Snowflake `import/load`

```
use Keboola\Db\ImportExport\Backend\Snowflake\Importer;
use Keboola\Db\ImportExport\ImportOptions;
use Keboola\Db\ImportExport\Storage;

$absSourceFile = new Storage\ABS\SourceFile(...);
$snowflakeDestinationTable = new Storage\Snowflake\Table(...);
$importOptions = new ImportOptions(...);

(new Importer($snowflakeConnection))->importTable(
    $absSourceFile,
    $snowflakeDestinationTable,
    $importOptions
);
```

Snowflake -> Snowflake `copy`

```
use Keboola\Db\ImportExport\Backend\Snowflake\Importer;
use Keboola\Db\ImportExport\ImportOptions;
use Keboola\Db\ImportExport\Storage;

$snowflakeSourceTable = new Storage\Snowflake\Table(...);
$snowflakeDestinationTable = new Storage\Snowflake\Table(...);
$importOptions = new ImportOptions(...);

(new Importer($snowflakeConnection))->importTable(
    $snowflakeSourceTable,
    $snowflakeDestinationTable,
    $importOptions
);
```

Snowflake -> ABS `export/unload`

```
use Keboola\Db\ImportExport\Backend\Snowflake\Exporter;
use Keboola\Db\ImportExport\ExportOptions;
use Keboola\Db\ImportExport\Storage;

$snowflakeSourceTable = new Storage\Snowflake\Table(...);
$absDestinationFile = new Storage\ABS\DestinationFile(...);
$exportOptions = new ExportOptions(...);

(new Exporter($snowflakeConnection))->exportTable(
    $snowflakeSourceTable,
    $absDestinationFile,
    $exportOptions
);
```

### Internals/Extending

[](#internalsextending)

Library consists of few simple interfaces.

#### Create new backend

[](#create-new-backend)

Importer, Exporter Interface must be implemented in new Backed

```
Keboola\Db\ImportExport\Backend\ImporterInterface
Keboola\Db\ImportExport\Backend\ExporterInterface

```

For each backend there is corresponding adapter which supports own combination of SourceInterface and DestinationInterface. Custom adapters can be set with `setAdapters` method.

#### Create new storage

[](#create-new-storage)

Storage is now file storage ABS|S3 (in future) or table storage Snowflake. Storage can have `Source` and `Destination` which must implement `SourceInterface` or `DestinationInterface`. These interfaces are empty and it's up to adapter to support own combination. In general there is one Import/Export adapter per FileStorage <=> TableStorage combination.

Adapter must implement:

- `Keboola\Db\ImportExport\Backend\BackendImportAdapterInterface` for import
- `Keboola\Db\ImportExport\Backend\BackendExportAdapterInterface` for export

Backend can require own extended AdapterInterface.

License
-------

[](#license)

MIT licensed, see [LICENSE](./LICENSE) file.

###  Health Score

57

—

FairBetter than 98% of packages

Maintenance70

Regular maintenance activity

Popularity28

Limited adoption so far

Community26

Small or concentrated contributor base

Maturity92

Battle-tested with a long release history

 Bus Factor1

Top contributor holds 61.6% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Every ~16 days

Total

138

Last Release

52d ago

Major Versions

v0.24.0 → v1.0.02022-05-19

v1.21.2 → v2.0.02023-04-03

v2.11.0 → 3.0.02025-04-11

3.4.1 → 4.0.02025-12-29

PHP version history (8 changes)v0.1PHP ^7.1

v0.17PHP ^7.4

v1.9.0PHP >=7.4

v1.20.0PHP >=7.4|^8

v1.21.0PHP ^7.4|^8

v2.0.0PHP ^8.1

3.0.0PHP ^8.2

4.5.0PHP ^8.3

### Community

Maintainers

![](https://www.gravatar.com/avatar/101dbf2551a0709ddab522f97669f13a2c4cc2d0a1e8d009f3af6ba80accb1a9?d=identicon)[Keboola](/maintainers/Keboola)

---

Top Contributors

[![zajca](https://avatars.githubusercontent.com/u/180148?v=4)](https://github.com/zajca "zajca (771 commits)")[![jirkasemmler](https://avatars.githubusercontent.com/u/13363655?v=4)](https://github.com/jirkasemmler "jirkasemmler (300 commits)")[![romanbracinik](https://avatars.githubusercontent.com/u/6448364?v=4)](https://github.com/romanbracinik "romanbracinik (62 commits)")[![martinjunger](https://avatars.githubusercontent.com/u/84506613?v=4)](https://github.com/martinjunger "martinjunger (31 commits)")[![yustme](https://avatars.githubusercontent.com/u/5272798?v=4)](https://github.com/yustme "yustme (26 commits)")[![martinjandl](https://avatars.githubusercontent.com/u/18183860?v=4)](https://github.com/martinjandl "martinjandl (15 commits)")[![vojtabiberle](https://avatars.githubusercontent.com/u/528942?v=4)](https://github.com/vojtabiberle "vojtabiberle (15 commits)")[![tomasfejfar](https://avatars.githubusercontent.com/u/642928?v=4)](https://github.com/tomasfejfar "tomasfejfar (10 commits)")[![ErikZigo](https://avatars.githubusercontent.com/u/1726727?v=4)](https://github.com/ErikZigo "ErikZigo (9 commits)")[![martinsifra](https://avatars.githubusercontent.com/u/1210906?v=4)](https://github.com/martinsifra "martinsifra (8 commits)")[![themark147](https://avatars.githubusercontent.com/u/17779619?v=4)](https://github.com/themark147 "themark147 (2 commits)")[![ujovlado](https://avatars.githubusercontent.com/u/419849?v=4)](https://github.com/ujovlado "ujovlado (1 commits)")[![pepamartinec](https://avatars.githubusercontent.com/u/271753?v=4)](https://github.com/pepamartinec "pepamartinec (1 commits)")

###  Code Quality

TestsPHPUnit

Static AnalysisPHPStan

Type Coverage Yes

### Embed Badge

![Health badge](/badges/keboola-db-import-export/health.svg)

```
[![Health](https://phpackages.com/badges/keboola-db-import-export/health.svg)](https://phpackages.com/packages/keboola-db-import-export)
```

###  Alternatives

[spatie/db-dumper

Dump databases

1.2k25.9M69](/packages/spatie-db-dumper)[scienta/doctrine-json-functions

A set of extensions to Doctrine that add support for json query functions.

58723.9M36](/packages/scienta-doctrine-json-functions)[damienharper/auditor-bundle

Integrate auditor library in your Symfony projects.

4542.8M](/packages/damienharper-auditor-bundle)[sonata-project/entity-audit-bundle

Audit for Doctrine Entities

644989.8k1](/packages/sonata-project-entity-audit-bundle)[bartlett/php-compatinfo-db

Reference Database of all functions, constants, classes, interfaces on PHP standard distribution and about 110 extensions

1183.0k1](/packages/bartlett-php-compatinfo-db)[worksome/foggy

Foggy is a tool for making database dumps with some data removed/changed.

26571.7k1](/packages/worksome-foggy)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)