PHPackages                             nuglif/chunkytown - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Utility & Helpers](/categories/utility)
4. /
5. nuglif/chunkytown

ActiveLibrary[Utility & Helpers](/categories/utility)

nuglif/chunkytown
=================

French paragraph/list/sentence-aware text chunker for RAG pipelines.

1.0.0(1mo ago)010MITPHPPHP >=8.1

Since Jul 3Pushed 1mo agoCompare

[ Source](https://github.com/Nuglif/chunkytown)[ Packagist](https://packagist.org/packages/nuglif/chunkytown)[ RSS](/packages/nuglif-chunkytown/feed)WikiDiscussions master Synced 1w ago

READMEChangelogDependencies (1)Versions (2)Used By (0)

ChunkyTown (PHP)
================

[](#chunkytown-php)

Text chunker for **French** RAG pipelines. Splits a document into overlapping spans that respect natural boundaries (paragraphs, lists, sentences, words) instead of cutting in the middle of a sentence — and never inside a composed emoji or accented grapheme.

Built on PHP's `intl` extension (ICU), with a French‑tuned abbreviation filter so common shortcuts (`M. Dupont`, `St. Patrick`, `cf.`, `etc.`) are not mistaken for sentence ends.

Installation
------------

[](#installation)

```
composer require nuglif/chunkytown
```

Requires PHP 8.1+ with the `intl` and `mbstring` extensions.

Quick start
-----------

[](#quick-start)

```
use ChunkyTown\Chunker;
use ChunkyTown\ChunkerConfig;

$text = file_get_contents('article.txt');

$chunker = new Chunker();
$cfg = new ChunkerConfig(
    chunkSize: 1500,   // target graphemes per chunk
    window: 175,       // ± snapping window around the chunk end
    overlap: 275,      // desired overlap between consecutive chunks
    overlapWindow: 150 // ± snapping window when looking for the overlap start
);

$spans = $chunker->chunk($text, $cfg);
// $spans is a list of [start, end) tuples in grapheme offsets.

foreach ($chunker->render($text, $spans) as $i => $chunk) {
    echo "=== CHUNK " . ($i + 1) . " ===\n$chunk\n\n";
}
```

### Reusable analysis (multi‑config runs)

[](#reusable-analysis-multiconfig-runs)

If you chunk the same text under several configs (tuning, A/B sweeps), pre‑compute the boundary detection once with `analyze()`:

```
$analysis = $chunker->analyze($text);

$spansA = $chunker->chunk($analysis, new ChunkerConfig(chunkSize: 1000));
$spansB = $chunker->chunk($analysis, new ChunkerConfig(chunkSize: 2000));
// boundary detection runs once, snapping runs twice.
```

### Configuration

[](#configuration)

`ChunkerConfig` is an immutable value object:

OptionDefaultMeaning`chunkSize``300`Target chunk length in **grapheme clusters** (user‑perceived characters).`window``50`± window around the target end to snap to a boundary.`preferOrder``['paragraph', 'list', 'sentence', 'word']`Boundary kinds tried in order when snapping.`overlap``0`Desired overlap (graphemes) between successive chunks.`overlapWindow``null` (falls back to `window`)± window used when locating the start of the next chunk.`normalize``false`Normalize the input to Unicode NFC before analysis (use for mixed NFC/NFD).### External boundaries

[](#external-boundaries)

If you already have sentence/paragraph offsets from another parser, merge them in (offsets are in graphemes):

```
$spans = $chunker->chunk($text, $cfg, externalBoundaries: [
    'sentence' => [120, 305, 480],
    'paragraph' => [0, 480, 1100],
]);
```

### Offsets and slicing

[](#offsets-and-slicing)

All offsets returned by the chunker are **grapheme cluster** offsets (emojis with ZWJ, regional‑indicator flags, skin‑tone modifiers and NFD‑decomposed accents all count as a single unit). Slice safely with `Chunker::render()`, which preserves grapheme integrity for you.

What's special
--------------

[](#whats-special)

- **ICU sentence detection** with a built‑in French abbreviation filter — `M. Dupont`, `St. Patrick`, `cf.`, `etc.` are not mistaken for sentence ends.
- **ICU word detection** — handles apostrophes (`l'arbre`), hyphens (`peut‑être`), and the typical French punctuation cleanly.
- **Grapheme‑safe**: a chunk boundary will never land inside a composed emoji or a decomposed accent.
- **Reusable analysis**: pay boundary detection once, re‑chunk many times.

Testing
-------

[](#testing)

```
composer install
composer test
```

The suite covers the grapheme mapper, every boundary detector, `ChunkerConfig`, `TextAnalysis`, and end‑to‑end chunking — including parity with the upstream Python implementation on a French fixture.

License
-------

[](#license)

MIT

###  Health Score

38

—

LowBetter than 83% of packages

Maintenance90

Actively maintained with recent releases

Popularity6

Limited adoption so far

Community6

Small or concentrated contributor base

Maturity42

Maturing project, gaining track record

 Bus Factor1

Top contributor holds 100% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Unknown

Total

1

Last Release

48d ago

### Community

Maintainers

![](https://www.gravatar.com/avatar/35607eb323a445cbcb16f311282bedd405d3f839e498592e8c77e94b8f7cb1fd?d=identicon)[adoy](/maintainers/adoy)

---

Top Contributors

[![adoy](https://avatars.githubusercontent.com/u/80873?v=4)](https://github.com/adoy "adoy (1 commits)")

---

Tags

textnlpragchunkingsplitter

###  Code Quality

TestsPHPUnit

### Embed Badge

![Health badge](/badges/nuglif-chunkytown/health.svg)

```
[![Health](https://phpackages.com/badges/nuglif-chunkytown/health.svg)](https://phpackages.com/packages/nuglif-chunkytown)
```

###  Alternatives

[rubix/ml

A high-level machine learning and deep learning library for the PHP language.

2.2k1.6M30](/packages/rubix-ml)[ckeditor/ckeditor

JavaScript WYSIWYG web text editor.

5224.4M81](/packages/ckeditor-ckeditor)[nlp-tools/nlp-tools

NlpTools is a set of php 5.3+ classes for beginner to semi advanced natural language processing work.

776679.6k5](/packages/nlp-tools-nlp-tools)[tinymce/tinymce

Web based JavaScript HTML WYSIWYG editor control.

1708.2M126](/packages/tinymce-tinymce)[codewithkyrian/transformers

State-of-the-art Machine Learning for PHP. Run Transformers in PHP

759280.6k11](/packages/codewithkyrian-transformers)[yooper/php-text-analysis

PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language

541429.7k4](/packages/yooper-php-text-analysis)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)