PHPackages                             nerou/large-array-buffer - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Utility & Helpers](/categories/utility)
4. /
5. nerou/large-array-buffer

ActiveLibrary[Utility & Helpers](/categories/utility)

nerou/large-array-buffer
========================

Handle arrays with huge cardinality but small items with ease

v1.3.0(7mo ago)12.4k—3.3%MITPHPPHP >=8.0CI passing

Since Nov 20Pushed 2mo ago1 watchersCompare

[ Source](https://github.com/nerou42/LargeArrayBuffer)[ Packagist](https://packagist.org/packages/nerou/large-array-buffer)[ RSS](/packages/nerou-large-array-buffer/feed)WikiDiscussions master Synced yesterday

READMEChangelog (10)Dependencies (2)Versions (11)Used By (0)

LargeArrayBuffer
================

[](#largearraybuffer)

[![License](https://camo.githubusercontent.com/cc20eb72b8f3bde45c4afec0d500ac85b33198c23dbaaade9005b06ab89172fe/68747470733a2f2f706f7365722e707567782e6f72672f6e65726f752f6c617267652d61727261792d6275666665722f6c6963656e7365)](https://packagist.org/packages/nerou/large-array-buffer)[![PHP Version Require](https://camo.githubusercontent.com/833e3d4b90708aa909df4f3a47e39a91613f9e878ccc57fa52d12c25809ebc1f/68747470733a2f2f706f7365722e707567782e6f72672f6e65726f752f6c617267652d61727261792d6275666665722f726571756972652f706870)](https://packagist.org/packages/nerou/large-array-buffer)[![Version](https://camo.githubusercontent.com/b8f8da22a185621bde58ecad9ea4d9e81c009323c86bc1b09f2f1122177fe6f2/68747470733a2f2f706f7365722e707567782e6f72672f6e65726f752f6c617267652d61727261792d6275666665722f76657273696f6e)](https://packagist.org/packages/nerou/large-array-buffer)[![Psalm Type Coverage](https://camo.githubusercontent.com/46ae35133faa13a41c245a70d0ee18f791fb500ae836231a93678e9b76bcfe11/68747470733a2f2f73686570686572642e6465762f6769746875622f6e65726f7534322f4c6172676541727261794275666665722f636f7665726167652e737667)](https://packagist.org/packages/nerou/large-array-buffer)

This is for you, if...
----------------------

[](#this-is-for-you-if)

...you are working with pretty large arrays that conflict with close to every memory limit you can set. The array elements on the other hand should not be that big, such that you can keep a single element in memory quite easily. If the array gets too big for memory, it will be moved to disk temporarily and transparently. You can still iterate over it as if it was a normal array.

One typical use case would be to load a lot of datasets from a database at once. (There are reasons to prefer this over running multiple queries.) See *Usage* below for an example for this use case using this library.

In general, you might want to try the following PHP-builtin solutions first:

- [SplFixedArray](https://www.php.net/manual/class.splfixedarray.php): By being of fixed length, this saves some memory compared to traditional `array`s. It behaves pretty much like `array`s do. Although the savings are not that huge compared to other solutions, it is probably the easiest to adopt.
- [Generator](https://www.php.net/manual/language.generators.php): Instead of returning an `array` all at once, this allows to return one item after the other. Thereby a "consumer" is able to process one item after the other accordingly. One can also terminate the "generation" of further items. One of the downsides is, that a Generator can only be iterated over once. The memory consumption can be as good as constant, but this approach is quite different and therefore rather hard to adopt and not appropriate in every scenario.

It should be noted, that a `LargeArrayBuffer` can be converted to both of these using `toFixedArray()` and `toGenerator()` respectively.

Install
-------

[](#install)

Note: This library requires PHP 8.0+!

Use composer to install this library:

`composer require nerou/large-array-buffer`

There are pretty much no dependencies with some exceptions:

- If you want to use the `toJSONFile()` method, you need to install `ext-json` (PHP's PECL JSON extension) as well.
- If you want to use the igbinary serializer, `ext-igbinary` is required. See [php-ext-igbinary](https://github.com/igbinary/igbinary).
- If you want to use the msgpack serializer, `ext-msgpack` is required. See [php-ext-msgpack](https://github.com/msgpack/msgpack-php).
- If you want to use LZ4 compression, `ext-lz4` is required. See [php-ext-lz4](https://github.com/kjdev/php-ext-lz4).

Usage
-----

[](#usage)

```
$pdo = new PDO(/* your database credentials */);
$stmt = $pdo->query('SELECT * FROM SomeDatabaseTable', PDO::FETCH_ASSOC);

$buffer = new LargeArrayBuffer();       // explicit use of LargeArrayBuffer
$buffer = new ArrayBuffer(1000);        // wrapper using `array` until given threshold (item count) is reached,
                                        // then switching to LargeArrayBuffer

while(($dataset = $stmt->fetch()) !== false){  // load one dataset at a time
    $buffer->push($dataset);    // push this dataset to the buffer
}

// ...

foreach($buffer as $item){
    // work with your data here
}
```

### Options

[](#options)

The constructor of `LargeArrayBuffer` provides some options:

1. You can set the threshold when to move the data to disk. When pushing data to the buffer, it is stored in memory until it gets too large. E.g.: `new LargeArrayBuffer(512);` to set a 512 MiB threshold.
2. You can choose either the PHP serializer, the [igbinary](https://github.com/igbinary/igbinary) serializer or the [msgpack](https://github.com/msgpack/msgpack-php) serializer (PHP serializer is default). E.g.: `new LargeArrayBuffer(serializer: LargeArrayBuffer::SERIALIZER_IGBINARY);`
3. You can enable GZIP or LZ4 compression for the serialized items. Although this is recommended only if your items are pretty big like > 1 KiB each. E.g.: `new LargeArrayBuffer(compression: LargeArrayBuffer::COMPRESSION_GZIP);`. Note, that LZ4 compression requires [ext-lz4](https://github.com/kjdev/php-ext-lz4) to be loaded.

### Read from the buffer

[](#read-from-the-buffer)

There are several options to read the data:

1. Iterate: As you might have seen in the example above, you can iterate over the buffer just like over an array with `foreach`. `foreach($buffer as $item){ /* work with your data here */ }`
2. `toArray()`: If you have a set of buffers which do not fit in memory all together but one at a time, you can use `$buffer->toArray()` to get an array to work with.
3. `toJSONFile()`: If you want to write the array in JSON format to some file or `resource`, use this method. It supports all the `json_encode()` options and flags.

### Buffer stats

[](#buffer-stats)

There are some stats you can obtain:

1. `count()`: The number of items in the buffer.
2. `getSize()`: The number of bytes of the serialized (!) data.

How it works
------------

[](#how-it-works)

To put it in one sentence: This library uses [php://temp](https://www.php.net/manual/en/wrappers.php.php) as well as PHP's [serialize](https://www.php.net/manual/en/function.serialize.php)/[unserialize](https://www.php.net/manual/en/function.unserialize.php) functions to store an array on disk if it gets too large.

Limitations and concerns
------------------------

[](#limitations-and-concerns)

- associative arrays are not supported
- the item type needs to be compatible with PHP's [serialize](https://www.php.net/manual/en/function.serialize.php)/[unserialize](https://www.php.net/manual/en/function.unserialize.php) functions
- since storage drives (even PCIe SSDs) are a lot slower than memory and de-/serialization needs to be done, you trade hard memory overflows for performance losses

### Benchmark

[](#benchmark)

A benchmark with 1 million measurements (consisting of DateTimeImmutable, int and float) using PHP 8.1 with 10 iterations comparing a normal array with the LargeArrayBuffer gave the following results (LargeArrayBuffer was configured with a memory limit of 128 MiB):

ActionSerializerCompressionConsumed timeConsumed memoryBuffer sizeFill arraynonenone1.57 s490.0 MiBNAIterate over arraynonenone0.27 s492.0 MiBNAFill bufferPHPnone4.27 s0 B378.7 MiBIterate over bufferPHPnone2.85 s0 B378.7 MiBFill bufferPHPGZIP24.76 s0 B192.5 MiBIterate over bufferPHPGZIP6.79 s0 B192.5 MiBFill bufferPHPLZ44.89 s0 B241.0 MiBIterate over bufferPHPLZ42.93 s0 B241.0 MiBFill bufferigbinarynone4.26 s0 B319.1 MiBIterate over bufferigbinarynone3.41 s0 B319.1 MiBFill bufferigbinaryGZIP21.50 s0 B173.2 MiBIterate over bufferigbinaryGZIP4.80 s0 B173.2 MiBFill bufferigbinaryLZ44.38 s0 B195.1 MiBIterate over bufferigbinaryLZ43.17 s0 B195.1 MiBNote:

- The peak memory usage using the buffer is about its memory limit. The table shows the memory usage after the specified action.
- PHP seems to cache the array once it is created for the first time, although `unset` is used. That is why I have not put the average value in the table for this specific value but the maximum (first run).
- The serialized data is smaller than the binary data in memory. I have absolutly no idea why.

To reproduce call bench/benchmark.php.

License
-------

[](#license)

This library is licensed under the MIT License. Please see [LICENSE](LICENSE) for more information.

###  Health Score

43

—

FairBetter than 89% of packages

Maintenance75

Regular maintenance activity

Popularity22

Limited adoption so far

Community7

Small or concentrated contributor base

Maturity54

Maturing project, gaining track record

 Bus Factor1

Top contributor holds 100% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Every ~79 days

Recently: every ~152 days

Total

10

Last Release

239d ago

Major Versions

0.4 → 1.0.02024-03-06

PHP version history (2 changes)0.1PHP >=8.0 <8.4

1.0.0PHP >=8.0

### Community

Maintainers

![](https://avatars.githubusercontent.com/u/28577589?v=4)[cracksalad](/maintainers/cracksalad)[@cracksalad](https://github.com/cracksalad)

---

Top Contributors

[![cracksalad](https://avatars.githubusercontent.com/u/28577589?v=4)](https://github.com/cracksalad "cracksalad (44 commits)")

---

Tags

arraybufferbig-dataphp

###  Code Quality

Static AnalysisRector

Code StylePHP\_CodeSniffer

### Embed Badge

![Health badge](/badges/nerou-large-array-buffer/health.svg)

```
[![Health](https://phpackages.com/badges/nerou-large-array-buffer/health.svg)](https://phpackages.com/packages/nerou-large-array-buffer)
```

###  Alternatives

[raiym/instagram-php-scraper

Instagram PHP Scraper. Get account information, photos and videos without any authorization

3.3k1.2M6](/packages/raiym-instagram-php-scraper)[michaeljennings/carpenter

A PHP package to create HTML tables from a data store that can be sorted and paginated.

191.7k](/packages/michaeljennings-carpenter)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)