PHPackages                             survos/jsonl-bundle - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Utility & Helpers](/categories/utility)
4. /
5. survos/jsonl-bundle

ActiveSymfony-bundle[Utility & Helpers](/categories/utility)

survos/jsonl-bundle
===================

Fast, concurrent, resumable, strictly-ordered JSONL ingestion utilities for Symfony 7.3 / PHP 8.4

2.10.19(2w ago)11.5k↓27.9%17MITPHPPHP ^8.5CI passing

Since Oct 13Pushed 1w agoCompare

[ Source](https://github.com/survos/jsonl-bundle)[ Packagist](https://packagist.org/packages/survos/jsonl-bundle)[ GitHub Sponsors](https://github.com/kbond)[ RSS](/packages/survos-jsonl-bundle/feed)WikiDiscussions main Synced 4d ago

READMEChangelogDependencies (98)Versions (260)Used By (17)

Survos JsonlBundle
==================

[](#survos-jsonlbundle)

**Streaming JSONL read/write utilities for Symfony**, with first-class support for **resumable writes**, **sidecar progress tracking**, **transparent gzip compression**, and **CLI-level inspection and querying**.

JsonlBundle is intentionally small at the API surface, but *feature-rich under the hood*. Many of its most powerful capabilities are currently “hidden” because they require no extra code once you adopt the bundle’s reader/writer abstractions.

This README makes those capabilities explicit, with concrete, copy-pasteable examples.

---

Why JSONL?
----------

[](#why-jsonl)

JSON Lines (`.jsonl`) is the ideal format for:

- Large datasets
- Streaming ingestion pipelines
- Fault-tolerant ETL
- CLI-driven workflows
- Append-only logs
- Partial resume after failure

JsonlBundle embraces these properties rather than fighting them.

---

Installation
------------

[](#installation)

```
composer require survos/jsonl-bundle
```

Symfony Flex will register the bundle automatically.

---

Core Concepts
-------------

[](#core-concepts)

### Files

[](#files)

- One JSON object per line
- Append-only
- Safe for streaming

### Sidecar

[](#sidecar)

Every JSONL file written with `JsonlWriter` has an optional **sidecar file** that tracks:

- Total records written
- Byte offset
- Completion status
- Timestamps
- Resume metadata

Sidecars enable **resume**, **progress inspection**, and **CLI tooling**.

### Compression

[](#compression)

If the filename ends in `.gz`, compression is automatic.

No flags. No config. Just naming.

---

Writing JSONL
-------------

[](#writing-jsonl)

### Basic Write (Overwrite by Default)

[](#basic-write-overwrite-by-default)

By default, `JsonlWriter::open()` uses **overwrite mode (`'w'`)**, matching PHP’s `fopen()` semantics.

This means:

- Existing files are truncated
- Sidecar state is reset
- You are explicitly starting a new dataset

```
use Survos\JsonlBundle\IO\JsonlWriter;

$writer = JsonlWriter::open('data/products.jsonl'); // mode: 'w'

$writer->write([
    'id' => 1,
    'name' => 'Widget',
    'price' => 9.99,
]);

$writer->close();
```

This immediately gives you:

- File locking
- Atomic writes
- Sidecar tracking

---

### Append vs Overwrite

[](#append-vs-overwrite)

JsonlBundle makes **dataset intent explicit** via write modes.

- `'w'` — overwrite (default): start a fresh dataset
- `'a'` — append: continue an existing dataset

#### Append to an Existing Dataset

[](#append-to-an-existing-dataset)

Use append mode when resuming or extending a JSONL file:

```
use Survos\JsonlBundle\IO\JsonlWriter;

$writer = JsonlWriter::open(
    'data/products.jsonl',
    'a', // append
);

foreach ($products as $product) {
    $writer->write($product);
}

$writer->close();
```

Append mode:

- Preserves existing data
- Resumes safely using the sidecar
- Is safe to re-run after partial failure

---

#### Overwrite an Existing Dataset

[](#overwrite-an-existing-dataset)

Overwrite mode is intentional and destructive:

```
use Survos\JsonlBundle\IO\JsonlWriter;

$writer = JsonlWriter::open(
    'data/products.jsonl',
    'w', // overwrite
);

$writer->write(['id' => 1, 'name' => 'Widget']);
$writer->close();
```

Overwrite mode:

- Truncates the file
- Resets sidecar state
- Signals a brand-new dataset

No buffering required. Memory-safe for large datasets.

---

### Resume a Partial Write

[](#resume-a-partial-write)

If the process crashes midway, reopen the file in **append mode**:

```
use Survos\JsonlBundle\IO\JsonlWriter;

$writer = JsonlWriter::open(
    'data/products.jsonl',
    'a', // resume
);

foreach ($remainingProducts as $product) {
    $writer->write($product);
}

$writer->close();
```

The writer will:

- Detect the sidecar
- Resume safely
- Avoid corrupt output

---

### Writing with Gzip Compression

[](#writing-with-gzip-compression)

Just use `.gz` — mode semantics are unchanged:

```
use Survos\JsonlBundle\IO\JsonlWriter;

$writer = JsonlWriter::open(
    'data/products.jsonl.gz',
    'w', // overwrite (default)
);

$writer->write([
    'id' => 1,
    'name' => 'Compressed Widget',
]);
```

Features retained:

- Streaming
- Resume
- Sidecar
- Locking

---

### Writer Options

[](#writer-options)

Advanced behavior is controlled via `JsonlWriterOptions`, passed as the **third argument** to `JsonlWriter::open()`.

Options configure **policy**, not mechanics. Defaults are safe and explicit.

#### Example: Overwrite and Ensure Directory Exists

[](#example-overwrite-and-ensure-directory-exists)

```
use Survos\JsonlBundle\IO\JsonlWriter;
use Survos\JsonlBundle\IO\JsonlWriterOptions;

$writer = JsonlWriter::open(
    'var/data/products.jsonl',
    'w',
    new JsonlWriterOptions(
        ensureDir: true,
    ),
);

$writer->write($row);
$writer->close();
```

In this example:

- `'w'` signals a fresh dataset
- `ensureDir: true` allows directory creation explicitly

Avoid overwrite mode (`'w'`) for resumable pipelines. Append mode with sidecars is the intended default for long-running jobs.

For the full list of writer options and their semantics, see:

- `doc/basic.md`
- `doc/advanced.md`

---

Reading JSONL
-------------

[](#reading-jsonl)

### Basic Read

[](#basic-read)

```
use Survos\JsonlBundle\IO\JsonlReader;

$reader = new JsonlReader('data/products.jsonl');

foreach ($reader as $row) {
    echo $row['name'] . PHP_EOL;
}
```

The reader:

- Streams line-by-line
- Uses constant memory
- Works for `.jsonl` and `.jsonl.gz`

---

### Read with Offset (Resume Processing)

[](#read-with-offset-resume-processing)

```
$reader = new JsonlReader('data/products.jsonl');
foreach ($reader as $row) {
    // Continue processing after record 5000
}
```

Ideal for batch pipelines and workers.

---

Sidecar Files (The Hidden Superpower)
-------------------------------------

[](#sidecar-files-the-hidden-superpower)

For a file:

```
data/products.jsonl

```

JsonlBundle maintains:

```
data/products.jsonl.sidecar.json

```

### What’s Inside the Sidecar?

[](#whats-inside-the-sidecar)

Typical contents:

```
{
  "rows": 12450,
  "bytes": 9876543,
  "completed": false,
  "started_at": "2026-01-05T10:21:11Z",
  "updated_at": "2026-01-05T10:25:02Z"
}
```

You do **not** manage this manually.

---

### Query Progress Programmatically

[](#query-progress-programmatically)

```
use Survos\JsonlBundle\Model\JsonlSidecar;

$repo  = new JsonlStateRepository();
$state = $repo->load('data/products.jsonl');

$stats = $state->getStats();

$rows      = $stats->getRows();
$bytes     = $stats->getBytes();
$completed = $stats->isCompleted();

if (!$state->isFresh()) {
    // JSONL changed outside of JsonlWriter
    // decide whether to trust, refresh, or regenerate
}
```

---

### Mark Completion Explicitly

[](#mark-completion-explicitly)

```
$writer = JsonlWriter::open('data/products.jsonl');

// write rows...

$writer->markComplete();
$writer->close();
```

This is especially useful in multi-stage pipelines.

---

CLI Utilities
-------------

[](#cli-utilities)

JsonlBundle ships with CLI tooling to inspect files *without writing code*.

### Inspect a File

[](#inspect-a-file)

```
php bin/console jsonl:info data/products.jsonl
```

Example output:

```
File: data/products.jsonl
Rows: 12450
Bytes: 9.4 MB
Compressed: no
Completed: false

```

---

### Inspect a Gzipped File

[](#inspect-a-gzipped-file)

```
php bin/console jsonl:info data/products.jsonl.gz
```

Works exactly the same.

---

### Peek at Records

[](#peek-at-records)

```
php bin/console jsonl:head data/products.jsonl --limit=5
```

---

### Resume-Aware Counting

[](#resume-aware-counting)

Unlike `wc -l`, JsonlBundle uses the sidecar:

```
php bin/console jsonl:count data/products.jsonl
```

Fast and accurate even for huge files.

---

Patterns Enabled by JsonlBundle
-------------------------------

[](#patterns-enabled-by-jsonlbundle)

### ETL Pipelines

[](#etl-pipelines)

```
download → normalize → enrich → translate → index

```

Each step emits JSONL, resumes safely, and records progress.

---

### Doctrine Import Pipelines

[](#doctrine-import-pipelines)

- Stream JSONL
- Batch insert entities
- Resume on failure
- Track progress via sidecar

---

### Translation Caches

[](#translation-caches)

- `source.jsonl`
- `target.jsonl`
- De-duplication
- Resume after API failure

---

### Search Index Feeds

[](#search-index-feeds)

- Emit JSONL once
- Replay into Meilisearch / OpenSearch
- Deterministic re-runs

---

When *Not* to Use JsonlBundle
-----------------------------

[](#when-not-to-use-jsonlbundle)

- Small datasets fully loaded into memory
- Interactive CRUD forms
- Request/response APIs

JsonlBundle shines in **pipelines**, **CLI tools**, and **long-running jobs**.

---

Design Philosophy
-----------------

[](#design-philosophy)

- Zero configuration
- Filename-driven behavior
- Streaming first
- Fail-safe defaults
- Sidecar over database state
- CLI-friendly

---

Summary
-------

[](#summary)

JsonlBundle is more than a reader/writer:

- ✔ Resumable writes
- ✔ Sidecar progress tracking
- ✔ Transparent gzip compression
- ✔ CLI inspection & querying
- ✔ Streaming everywhere

Most of this works automatically once you adopt `JsonlWriter` and `JsonlReader`.

---

Example: Caching a Remote API as JSONL (DummyJSON Products)
-----------------------------------------------------------

[](#example-caching-a-remote-api-as-jsonl-dummyjson-products)

This example demonstrates how JsonlBundle can be used as a **simple, durable cache** for a remote HTTP API, without introducing a separate caching abstraction.

Goal:

- Command: `bin/console products:list`
- If `products.jsonl` does **not** exist:
    - Fetch `https://dummyjson.com/products`
    - Write each product as a JSONL record
- Then:
    - Read `products.jsonl`
    - Display the **first product**

This turns a one-shot HTTP call into a **replayable, CLI-friendly dataset**.

---

### The Command (Minimal, Accurate API Usage)

[](#the-command-minimal-accurate-api-usage)

```