PHPackages                             neuron-core/tokenizer - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Utility & Helpers](/categories/utility)
4. /
5. neuron-core/tokenizer

ActiveLibrary[Utility & Helpers](/categories/utility)

neuron-core/tokenizer
=====================

High-Performance Tokenizer implementation in PHP

1.0.0(8mo ago)6219↓33.3%1MITPHPPHP ^8.1

Since Sep 16Pushed 6mo agoCompare

[ Source](https://github.com/neuron-core/tokenizer)[ Packagist](https://packagist.org/packages/neuron-core/tokenizer)[ RSS](/packages/neuron-core-tokenizer/feed)WikiDiscussions main Synced 1mo ago

READMEChangelog (1)Dependencies (5)Versions (3)Used By (0)

High-Performance Tokenizer implementation in PHP
================================================

[](#high-performance-tokenizer-implementation-in-php)

```
composer require neuron-core/tokeinzer

```

BPE (Byte-Pair Encoding) Tokenizer
----------------------------------

[](#bpe-byte-pair-encoding-tokenizer)

This implementation requires an already trained vocabulary and merges files. It's focused on performance with cache strategy and priority queue.

```
$tokenizer = new BPETokenizer();

// Initialize pre-trained tokenizer
$tokenizer->loadFrom(
    __DIR__.'/../dataset/bpe-vocabulary.json',
    __DIR__.'/../dataset/bpe-merges.txt'
);

// Tokenize text
$tokens = $tokenizer->encode('Hello world!');

var_dump($tokens);
```

Benchmarks
----------

[](#benchmarks)

The benchmarks are done with [PHPBench](https://github.com/phpbench/phpbench), and they show a sub-millisecond performance encoding the example text below for a thousand times:

```
Contrary to popular belief, Lorem Ipsum is not simply random text. It has roots in a piece of classical Latin literature from 45 BC, making it over 2000 years old. Richard McClintock, a Latin professor at Hampden-Sydney College in Virginia, looked up one of the more obscure Latin words, consectetur, from a Lorem Ipsum passage, and going through the cites of the word in classical literature, discovered the undoubtable source. Lorem Ipsum comes from sections 1.10.32 and 1.10.33 of "de Finibus Bonorum et Malorum" (The Extremes of Good and Evil) by Cicero, written in 45 BC. This book is a treatise on the theory of ethics, very popular during the Renaissance. The first line of Lorem Ipsum, "Lorem ipsum dolor sit amet..", comes from a line in section 1.10.32.

The standard chunk of Lorem Ipsum used since the 1500s is reproduced below for those interested. Sections 1.10.32 and 1.10.33 from "de Finibus Bonorum et Malorum" by Cicero are also reproduced in their exact original form, accompanied by English versions from the 1914 translation by H. Rackham.

```

With the already trained GPT-2 datasets:

- Vocabulary:
- Merges:

[![PHP Tokenizer](./doc/tokenizer-bench.png)](./doc/tokenizer-bench.png)

###  Health Score

37

—

LowBetter than 83% of packages

Maintenance63

Regular maintenance activity

Popularity21

Limited adoption so far

Community7

Small or concentrated contributor base

Maturity45

Maturing project, gaining track record

 Bus Factor1

Top contributor holds 100% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Unknown

Total

1

Last Release

244d ago

### Community

Maintainers

![](https://www.gravatar.com/avatar/320345fbe48c7cff2b3a25992c8207e959ae7817fb2fb97bb176dc6e559aacf5?d=identicon)[valerione](/maintainers/valerione)

---

Top Contributors

[![ilvalerione](https://avatars.githubusercontent.com/u/13559278?v=4)](https://github.com/ilvalerione "ilvalerione (10 commits)")

---

Tags

agentic-frameworkagentic-workflowaiai-agentsai-frameworkbpe-tokenizerllmphptokenizer

###  Code Quality

TestsPHPUnit

Static AnalysisPHPStan, Rector

Code StylePHP CS Fixer

Type Coverage Yes

### Embed Badge

![Health badge](/badges/neuron-core-tokenizer/health.svg)

```
[![Health](https://phpackages.com/badges/neuron-core-tokenizer/health.svg)](https://phpackages.com/packages/neuron-core-tokenizer)
```

###  Alternatives

[hanson/laravel-work-exception

133.8k](/packages/hanson-laravel-work-exception)[wujunze/money-wrapper

MoneyPHP Wrapper

113.8k](/packages/wujunze-money-wrapper)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)