PHPackages                             keboola/storage-driver-bigquery - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Database & ORM](/categories/database)
4. /
5. keboola/storage-driver-bigquery

AbandonedArchivedProject[Database & ORM](/categories/database)

keboola/storage-driver-bigquery
===============================

Keboola BigQuery driver

v8.26.0(3mo ago)38.6k↓91.7%[5 PRs](https://github.com/keboola/php-storage-driver-bigquery/pulls)MITPHPPHP ^8.3CI failing

Since Sep 30Pushed 3mo ago13 watchersCompare

[ Source](https://github.com/keboola/php-storage-driver-bigquery)[ Packagist](https://packagist.org/packages/keboola/storage-driver-bigquery)[ RSS](/packages/keboola-storage-driver-bigquery/feed)WikiDiscussions main Synced yesterday

READMEChangelog (10)Dependencies (53)Versions (161)Used By (0)

Keboola Storage Driver Big Query
================================

[](#keboola-storage-driver-big-query)

Keboola high level storage backend driver for Big Query

Install dependencies.
---------------------

[](#install-dependencies)

```
docker compose run --rm dev composer install
```

Setup Big Query
---------------

[](#setup-big-query)

Install [Google Cloud client](https://cloud.google.com/sdk/docs/install-sdk) (via [Brew](https://formulae.brew.sh/cask/google-cloud-sdk#default)), initialize it and log in to [generate default credentials](https://cloud.google.com/docs/authentication/application-default-credentials#personal).

To prepare the backend use [Terraform template](./bq-storage-backend-init.tf). Create a sub folder in the **KBC Team Dev** (id: [431160969986](https://console.cloud.google.com/cloud-resource-manager?folder=431160969986)) folder and fill the folder into the terraform command.

1. get missing pieces (organization\_id and billing\_id) from [Connection repository](https://github.com/keboola/connection/blob/master/docs/DOCKER.md#bigquery).
2. (optional) move `bq-storage-backend-init.tf` out of project directory so new files would be out of git
3. Run `terraform init`
4. Run `terraform apply -var folder_id=[folder_id] -var billing_account_id=[billing_id] -var backend_prefix= -var file_storage_backend_region=` (e.g. us-central1 region)
    1. Optionally set `terraform.tfvars` file with variables there is predefined `terraform.tfvars.dist` file
5. New key files was created `principal_key.json` and `big_query_key.json`
6. Set envs from the `principal_key.json` and `big_query_key.json` by running `php SetBQVars.php` and stop. OR continue with following steps with manual extraction.
7. open `principal_key.json` set content of `private_key` as variable `BQ_SECRET` and remove (the whole entry) it from json file
    1. note: simply cut&paste it whole even with the quotes and new lines -> your .env will be like `BQ_SECRET="-----BEGIN PRIVATE KEY-----XXXXZQ==\n-----END PRIVATE KEY-----\n"`
8. remove line breaks from the rest of key file (without `private_key` entry) and set this string as variable `BQ_PRINCIPAL` to `.env`
    1. You can convert the key to string with `awk -v RS= '{$1=$1}1' principal_key.json`
9. remove line breaks from the `big_query_key.json` key file and set this string as variable `BQ_KEY_FILE` to `.env`
    1. You can convert the key to string with `awk -v RS= '{$1=$1}1' big_query_key.json`

At the end, your `.env` file should look like...

```
# the id is printed by terraform at the end and it is just the numbers after `folders/`

BQ_PRINCIPAL=
BQ_SECRET=

BQ_FOLDER_ID=
BQ_BUCKET_NAME=

# choose different BQ_STACK_PREFIX than you Terraform prefix otherwise project created by Terraform will be deleted . e.g. local :)
BQ_STACK_PREFIX=local

BQ_KEY_FILE=
```

All done. Now you can try `composer loadGcs` script and run tests.

```
docker compose run --rm dev composer loadGcs
```

Build docker images
-------------------

[](#build-docker-images)

```
docker compose build
```

Xdebug
------

[](#xdebug)

To run with xdebug use `dev-xdebug` container instead of `dev`

Tests
-----

[](#tests)

Run tests with following command.

```
# This will run all tests
docker compose run --rm dev composer tests
# This will run all tests in parallel
docker compose run --rm dev composer paratest
# This will run import tests in parallel
docker compose run --rm dev composer paratest-import
# This will run export tests in parallel
docker compose run --rm dev composer paratest-export
# This will run all tests in parallel excluding import and export
docker compose run --rm dev composer paratest-other
```

To disable retry copy `phpunit-retry.xml.dist`

```
cp phpunit-retry.xml.dist phpunit-retry.xml
```

Local development with libraries changes
----------------------------------------

[](#local-development-with-libraries-changes)

When doing local development and you want to use local changes in `php-storage-driver-common` or `php-table-backend-utils` packages, you can symlink them into the container.

```
# remove vendor folder
rm -r ./vendor/keboola/storage-driver-common
rm -r ./vendor/keboola/table-backend-utils
# symlink local proto folder
ln -s /home//keboola/storage-backend/packages/php-storage-driver-common ./vendor/keboola/storage-driver-common
ln -s /home//keboola/storage-backend/packages/php-table-backend-utils ./vendor/keboola/table-backend-utils
# create docker-compose.override.yml file with following content
# it is needed to mount local packages into the container
cat > docker-compose.override.yml