PHPackages nonsapiens/bigquery-model-sync - PHPackages - PHPackages [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register) 1. [Directory](/) 2. / 3. [Database & ORM](/categories/database) 4. / 5. nonsapiens/bigquery-model-sync ActiveLibrary[Database & ORM](/categories/database) nonsapiens/bigquery-model-sync ============================== A Laravel library for coordinating batch record-level syncs from the local database, to BigQuery v0.1.22(2mo ago)220MITPHPPHP ^8.3 Since Apr 3Pushed 2mo agoCompare [ Source](https://github.com/nonsapiens/bigquery-model-sync)[ Packagist](https://packagist.org/packages/nonsapiens/bigquery-model-sync)[ RSS](/packages/nonsapiens-bigquery-model-sync/feed)WikiDiscussions main Synced today READMEChangelogDependencies (6)Versions (25)Used By (0) BigQuery Model Sync =================== [](#bigquery-model-sync) A Laravel package to synchronize Eloquent models with Google BigQuery tables. Installation ------------ [](#installation) 1. Install the package via composer: ``` composer require nonsapiens/bigquery-model-sync ``` 2. Publish the configuration file and migrations: ``` php artisan vendor:publish --tag="bigquery-model-sync-config" php artisan vendor:publish --tag="bigquery-model-sync-migrations" ``` 3. Run the migrations: ``` php artisan migrate ``` Configuration ------------- [](#configuration) The configuration is located at `config/bigquery.php`. ### Environment Variables [](#environment-variables) - `GOOGLE_CLOUD_PROJECT`: Your Google Cloud Project ID. - `BIGQUERY_DATASET`: The BigQuery dataset where tables will be created. - `BIGQUERY_LOCATION`: The location for your BigQuery dataset (default: `africa-south1`). - `BIGQUERY_AUTOSYNC`: Boolean to enable/disable automatic scheduled syncing (default: `true`). - `BIGQUERY_LOGGING`: Boolean to enable/disable sync logging (default: `true`). - `BIGQUERY_CREDENTIALS`: Path to your Google Cloud service account JSON file. ### Localhost vs WIF (Workload Identity Federation) [](#localhost-vs-wif-workload-identity-federation) **Localhost:**Set `BIGQUERY_CREDENTIALS` in your `.env` file to the absolute path of your service account JSON key. ``` BIGQUERY_CREDENTIALS=/path/to/your/service-account-key.json ``` **Deployed (WIF):**When deployed on Google Cloud (GKE, Cloud Run, etc.) using Workload Identity Federation, you should NOT provide a service account key. The `BigQueryClient` will automatically use the environment's default credentials. Leave `BIGQUERY_CREDENTIALS` empty or unset in your production environment. Usage ----- [](#usage) ### 1. Prepare your Model [](#1-prepare-your-model) Add the `SyncsToBigQuery` trait to the models you want to synchronize. ``` namespace App\Models; use Illuminate\Database\Eloquent\Model; use Nonsapiens\BigqueryModelSync\Traits\SyncsToBigQuery; class User extends Model { use SyncsToBigQuery; } ``` ### 2. Configure Model Settings (Optional) [](#2-configure-model-settings-optional) You can customize the sync behavior by defining properties or methods in your model: Property / MethodDefaultDescription`$fieldsToSync``[]` (all fields)Array of columns to sync.`$syncStrategy``BATCH``BATCH`, `REPLACE`, or `ON_INSERT`.`$bigQueryTableName`Model table nameThe target BigQuery table name.`$syncSchedule``null`Cron expression for when the model should sync.`$batchSize``10000`Number of records per batch.`$batchField``sync_batch_uuid`Database field used to track sync batches.### 3. Create BigQuery Table [](#3-create-bigquery-table) Generate the SQL required to create the table in BigQuery: ``` php artisan make:bigquery-table --class="App\Models\User" ``` Copy the output and run it in the BigQuery console. ### 4. Prepare Database (if using BATCH or ON\_INSERT) [](#4-prepare-database-if-using-batch-or-on_insert) If you are using `BATCH` (default) or `ON_INSERT` strategies, you need a tracking column in your local database. Generate a migration: ``` php artisan make:bigquery-model-migration --class="App\Models\User" php artisan migrate ``` Syncing Strategies ------------------ [](#syncing-strategies) - **BATCH**: (Default) Synchronizes records in batches. It uses a UUID column to mark records that have been sent. Recommended for most use cases. - **REPLACE**: Truncates the BigQuery table and re-uploads all data. Good for small tables or when records are frequently updated. - **ON\_INSERT**: Dispatches a job to sync an individual record immediately after it is created in the local database. Geodata Mapping --------------- [](#geodata-mapping) If your model contains latitude and longitude information, the package can automatically map these into a single `GEOGRAPHY` field in BigQuery. ### Configuration [](#configuration-1) Add the following properties or methods to your model: Property / MethodDefaultDescription`$hasGeodata``false`Set to `true` to enable geodata mapping.`$geodataFields``['latitude', 'longitude']`The source fields in your local database.`$mappedGeographyField``geolocation`The name of the `GEOGRAPHY` column in BigQuery.### How it Works [](#how-it-works) 1. **Table Generation**: When running `php artisan make:bigquery-table`, the package will exclude the individual latitude and longitude fields from the generated SQL and instead include a single `GEOGRAPHY` field (e.g., `geolocation`). 2. **Data Transformation**: During synchronization, the package combines the latitude and longitude values into a [WKT (Well-Known Text)](https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry) `POINT` string: `POINT(longitude latitude)`. 3. **BigQuery Storage**: BigQuery stores this as a native `GEOGRAPHY` type, allowing you to use [BigQuery GIS functions](https://cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions) for spatial analysis. > **Note**: The interactive `bigquery:set-model` command will automatically detect if `latitude` and `longitude` fields are present and offer to configure this mapping for you. Requirements for Schedule and Queue ----------------------------------- [](#requirements-for-schedule-and-queue) ### Scheduling [](#scheduling) The package includes a scheduled task that runs every minute to check if any models are due for synchronization based on their `$syncSchedule`. To enable this, ensure the Laravel scheduler is running in your environment: ``` * * * * * cd /path-to-your-project && php artisan schedule:run >> /dev/null 2>&1 ``` You can disable automatic scheduling by setting `BIGQUERY_AUTOSYNC=false` in your `.env`. ### Queuing [](#queuing) The `ON_INSERT` strategy and the `bigquery:sync --queue` command rely on Laravel's queue system. Ensure you have a queue worker running: ``` php artisan queue:work ``` Commands -------- [](#commands) ### `bigquery:set-model` [](#bigqueryset-model) An interactive command to configure an Eloquent model for BigQuery synchronization. It guides you through selecting fields, configuring geodata mapping, and choosing a sync strategy. ``` php artisan bigquery:set-model ``` - **Interactive Steps:** - Select a model from your application. - Choose which fields should be synchronized. - Configure `GEOGRAPHY` field mapping if `latitude` and `longitude` are present. - Choose between immediate (`ON_INSERT`) or scheduled (`BATCH`) synchronization. - Generate the necessary database migration for the tracking column. ### `make:bigquery-table` [](#makebigquery-table) Generates the `CREATE TABLE` SQL required for BigQuery based on your Eloquent model's schema and configuration. ``` php artisan make:bigquery-table --class="App\Models\User" ``` - `--class`: (Required) The fully qualified class name of the model. ### `make:bigquery-model-migration` [](#makebigquery-model-migration) Creates a Laravel migration to add the required tracking column (default: `sync_batch_uuid`) to your model's database table. ``` php artisan make:bigquery-model-migration --class="App\Models\User" ``` - `--class`: (Required) The fully qualified class name of the model. ### `bigquery:sync` [](#bigquerysync) Synchronizes a specific model with BigQuery. This command is typically called by the scheduler but can be run manually. ``` php artisan bigquery:sync --class="App\Models\User" [options] ``` - `--class`: (Required) The fully qualified class name of the model. - `--force`: Run the sync even if it's not currently due according to the model's `$syncSchedule`. - `--queue`: Dispatch the sync job to the queue instead of running it immediately in the foreground. - `--schedule`: Override the cron expression defined in the model. ### `bigquery:sync-all` [](#bigquerysync-all) Discovers all models using the `SyncsToBigQuery` trait and runs their synchronization if they are due. ``` php artisan bigquery:sync-all [--force] ``` - `--force`: Run synchronization for all discovered models, regardless of their schedule. ### `bigquery:truncate` [](#bigquerytruncate) Truncates the BigQuery tables associated with your models. **Use with caution.** ``` php artisan bigquery:truncate {--all|--class="App\Models\User"} ``` - `--all`: Truncate tables for all models using the `SyncsToBigQuery` trait. - `--class`: A comma-separated list of model classes to truncate. - **Note**: This command requires manual confirmation and a randomly generated verification code to prevent accidental data loss. ### Health Score 40 — FairBetter than 86% of packages Maintenance86 Actively maintained with recent releases Popularity9 Limited adoption so far Community6 Small or concentrated contributor base Maturity49 Maturing project, gaining track record Bus Factor1 Top contributor holds 100% of commits — single point of failure How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window. **Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores. **Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement. **Maturity (30%)** — Project age, version count, PHP version support, and release stability. ### Release Activity Cadence Every ~1 days Total 23 Last Release 69d ago PHP version history (2 changes)v0.1.0PHP ^8.0 v0.1.3PHP ^8.3 ### Community Maintainers ![](https://www.gravatar.com/avatar/6d8b2540548fd4037017a3177a24b230506c67ca96d5ed870cd82153726e597f?d=identicon)[nonsapiens](/maintainers/nonsapiens) --- Top Contributors [![nonsapiens](https://avatars.githubusercontent.com/u/9744534?v=4)](https://github.com/nonsapiens "nonsapiens (1 commits)") ### Code Quality TestsPHPUnit ### Embed Badge ![Health badge](/badges/nonsapiens-bigquery-model-sync/health.svg) ``` [![Health](https://phpackages.com/badges/nonsapiens-bigquery-model-sync/health.svg)](https://phpackages.com/packages/nonsapiens-bigquery-model-sync) ``` ### Alternatives [anourvalar/eloquent-serialize Laravel Query Builder (Eloquent) serialization 11223.5M33](/packages/anourvalar-eloquent-serialize)[statamic-rad-pack/runway Eloquently manage your database models in Statamic. 135224.7k7](/packages/statamic-rad-pack-runway)[duncanmcclean/statamic-cargo Comprehensive e-commerce addon for Statamic. Build bespoke e-commerce sites without the complexity. 3417.0k](/packages/duncanmcclean-statamic-cargo) PHPackages © 2026 [Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)