PHPackages                             survos/multi-fetch-bundle - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [HTTP & Networking](/categories/http)
4. /
5. survos/multi-fetch-bundle

Abandoned → [survos/fetch-bundle](/?search=survos%2Ffetch-bundle)Symfony-bundle[HTTP & Networking](/categories/http)

survos/multi-fetch-bundle
=========================

HTTP fetch utilities for Symfony: resumable chunk downloads, caching, retry/backoff, and concurrent fetch into JSONL

2.10.19(2w ago)156MITPHPPHP ^8.5

Since Mar 19Pushed 1mo agoCompare

[ Source](https://github.com/survos/fetch-bundle)[ Packagist](https://packagist.org/packages/survos/multi-fetch-bundle)[ GitHub Sponsors](https://github.com/kbond)[ RSS](/packages/survos-multi-fetch-bundle/feed)WikiDiscussions main Synced today

READMEChangelog (1)Dependencies (54)Versions (138)Used By (0)

Fetch Bundle
============

[](#fetch-bundle)

Reusable HTTP fetch utilities for Symfony applications that harvest remote data into JSONL.

This bundle started as `multi-fetch-bundle`, an experiment around parallel API fetching. The useful core is broader than concurrency: most dataset fetchers need the same small set of primitives:

- bounded-concurrency HTTP requests;
- retry and backoff for transient failures;
- page planning for offset, page-number, and cursor APIs;
- resumable downloads for large files;
- JSONL output through `survos/jsonl-bundle`;
- optional HTTP cache helpers for source APIs that do not send useful cache headers.

The package is `survos/fetch-bundle`. "Multi" (concurrent fetching) is just one execution mode; the resumable `ChunkDownloader` and cache helpers stand on their own.

Current Services
----------------

[](#current-services)

`SymfonyConcurrentFetcher` implements bounded concurrent HTTP fetching using Symfony HttpClient streaming. It accepts keyed request metadata and yields keyed response arrays as requests complete.

`ExponentialBackoffRetry` provides a simple retry policy for transport errors, HTTP 429, and 5xx responses.

`ChunkDownloader` downloads large files to `*.part`, supports HTTP Range resume when the source honors it, retries transient failures, and reports byte progress.

`multi:fetch` is an experimental CLI for Solr/JSON/JSON-LD style sources. It writes rows with `Survos\JsonlBundle\IO\JsonlWriter`.

Dependencies
------------

[](#dependencies)

`survos/jsonl-bundle` is a hard dependency because JSONL is the canonical output format for Survos dataset harvesting. This bundle should depend on jsonl-bundle, not copy classes from it.

Example
-------

[](#example)

```
use Survos\FetchBundle\Contract\ConcurrentFetcherInterface;
use Survos\FetchBundle\Contract\DTO\FetchOptions;

final class DatasetFetcher
{
    public function __construct(
        private readonly ConcurrentFetcherInterface $fetcher,
    ) {}

    public function fetch(array $urls): iterable
    {
        $requests = [];
        foreach ($urls as $i => $url) {
            $requests[$i] = ['url' => $url];
        }

        yield from $this->fetcher->fetchMany($requests, new FetchOptions(
            concurrency: 8,
            timeout: 60.0,
            defaultHeaders: ['Accept' => 'application/json'],
        ));
    }
}
```

Harvest References
------------------

[](#harvest-references)

The next design pass should extract repetition from Harvest dataset commands such as:

- `dataset:fetch:belvedere`: page-number API, XML parse, stop on empty page;
- `dataset:fetch:victoria`: page-number API, JSON parse, sidecar/count based resume;
- `dataset:fetch:aust`: offset/limit API, multiple raw output cores;
- `dataset:fetch:walters`: large archive download and local CSV-to-JSONL conversion.

Source-specific parsing and row normalization should stay in applications. Pagination, retry, resume, cache behavior, and JSONL output targets belong here.

TUI Progress
------------

[](#tui-progress)

Symfony 8.1's TUI component is a good fit for visualizing concurrent fetches, but it should be an optional presentation layer over the fetch engine.

The core fetch service should emit structured progress events such as `planned`, `started`, `bytes`, `pageComplete`, `retry`, `failed`, and `merged`. A TUI renderer can show one row per active page/download, plus aggregate totals and a log pane. Non-interactive runs should use the same events for normal console progress output.

For precomputed page ranges, concurrent downloads can be displayed naturally: page number, URL/key, status, retries, bytes, rows, and elapsed time. For cursor or `nextPage` APIs, concurrency is usually limited because the next URL is discovered only after reading the current response; the TUI still helps by showing cursor progress, row counts, retries, and merge state.

A future TUI implementation should follow the `tui-monitor` pattern: keep the engine independent of TUI classes, then put dashboard/widgets in a separate namespace that is only registered when `Symfony\Component\Tui\Tui` exists.

Generic Pagination Flow
-----------------------

[](#generic-pagination-flow)

The target generic flow is:

1. Build a fetch plan from endpoint configuration, auth headers/query params, and a pagination strategy.
2. Fetch pages to a temporary directory as page-local JSON/JSONL files.
3. Extract rows from each page using a source-specific selector/extractor.
4. Merge page files in stable order into the final JSONL output with `JsonlWriter`.
5. Write sidecar state so interrupted runs can resume or skip completed pages.

For page-number and offset/limit APIs, the plan can often be known up front and fetched concurrently. For `nextPage`/cursor APIs, planning and fetching are interleaved unless the API also exposes all cursors or a total count.

MVP Scope
---------

[](#mvp-scope)

The first useful extraction should be sequential and resumable, not concurrent.

A practical v1 should cover the common Harvest loop:

- read existing JSONL sidecar/count state;
- resume from the correct page or offset;
- fetch one page at a time with retry/backoff and optional delay;
- extract rows from JSON;
- append rows with `JsonlWriter`;
- stop on empty page, missing `nextPage`, or explicit limit.

Victoria is the best first consumer. Belvedere is a good second consumer. Multi-output fetchers such as Aust and archive converters such as Walters should wait until the small sequential API is stable.

###  Health Score

49

—

FairBetter than 94% of packages

Maintenance95

Actively maintained with recent releases

Popularity11

Limited adoption so far

Community6

Small or concentrated contributor base

Maturity69

Established project with proven stability

 Bus Factor1

Top contributor holds 100% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Every ~1 days

Total

137

Last Release

17d ago

PHP version history (2 changes)2.0.193PHP ^8.4

2.8.0PHP ^8.5

### Community

Maintainers

![](https://www.gravatar.com/avatar/21b39551f92ed4143772c622f9e571589c5a72c96ab3c53fe67489ce0d83e806?d=identicon)[tacman1123](/maintainers/tacman1123)

---

Top Contributors

[![tacman](https://avatars.githubusercontent.com/u/619585?v=4)](https://github.com/tacman "tacman (8 commits)")

### Embed Badge

![Health badge](/badges/survos-multi-fetch-bundle/health.svg)

```
[![Health](https://phpackages.com/badges/survos-multi-fetch-bundle/health.svg)](https://phpackages.com/packages/survos-multi-fetch-bundle)
```

###  Alternatives

[shopware/core

Shopware platform is the core for all Shopware ecommerce products.

585.6M577](/packages/shopware-core)[sylius/sylius

E-Commerce platform for PHP, based on Symfony framework.

8.5k5.9M738](/packages/sylius-sylius)[matomo/matomo

Matomo is the leading Free/Libre open analytics platform

21.7k38.9k](/packages/matomo-matomo)[shopware/platform

The Shopware e-commerce core

3.4k1.5M3](/packages/shopware-platform)[sulu/sulu

Core framework that implements the functionality of the Sulu content management system

1.3k1.4M204](/packages/sulu-sulu)[contao/core-bundle

Contao Open Source CMS

1231.6M2.8k](/packages/contao-core-bundle)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)