PHPackages                             lalbert/tokenizer-engine - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Parsing & Serialization](/categories/parsing)
4. /
5. lalbert/tokenizer-engine

ActiveLibrary[Parsing & Serialization](/categories/parsing)

lalbert/tokenizer-engine
========================

Tokenize string like Elasticsearch

0.0.1(9y ago)118MITPHP >=5.4.0

Since Apr 7Compare

[ Source](https://github.com/lalbert/tokenizer-engine)[ Packagist](https://packagist.org/packages/lalbert/tokenizer-engine)[ Docs](https://github.com/lalbert/TokenizerEngine)[ RSS](/packages/lalbert-tokenizer-engine/feed)WikiDiscussions Synced 2d ago

READMEChangelogDependencies (2)Versions (3)Used By (0)

TokenizerEngine
===============

[](#tokenizerengine)

[![Codacy Badge](https://camo.githubusercontent.com/9568012030f11b0e84c62bff67caddbc1c3d5f80be6ee6720f4dcc44c73a90e3/68747470733a2f2f6170692e636f646163792e636f6d2f70726f6a6563742f62616467652f47726164652f6465386362366161303636613433626239356637636361613964323834376335)](https://www.codacy.com/app/lalbert/tokenizer-engine?utm_source=github.com&utm_medium=referral&utm_content=lalbert/tokenizer-engine&utm_campaign=badger)

Tokenize string like Elasticsearch

###  Health Score

22

—

LowBetter than 21% of packages

Maintenance20

Infrequent updates — may be unmaintained

Popularity8

Limited adoption so far

Community2

Small or concentrated contributor base

Maturity49

Maturing project, gaining track record

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Unknown

Total

1

Last Release

3373d ago

### Community

Maintainers

![](https://avatars.githubusercontent.com/u/6500272?v=4)[lalbert](/maintainers/lalbert)[@lalbert](https://github.com/lalbert)

---

Tags

tokenizer

###  Code Quality

TestsPHPUnit

### Embed Badge

![Health badge](/badges/lalbert-tokenizer-engine/health.svg)

```
[![Health](https://phpackages.com/badges/lalbert-tokenizer-engine/health.svg)](https://phpackages.com/packages/lalbert-tokenizer-engine)
```

###  Alternatives

[yethee/tiktoken

PHP version of tiktoken

1624.0M29](/packages/yethee-tiktoken)[gioni06/gpt3-tokenizer

PHP package for Byte Pair Encoding (BPE) used by GPT-3.

84591.1k11](/packages/gioni06-gpt3-tokenizer)[rajentrivedi/tokenizer-x

TokenizerX calculates required tokens for given prompt

92252.4k3](/packages/rajentrivedi-tokenizer-x)[nicoswd/php-rule-parser

Rule Engine - Rule Parser & Evaluator

13080.4k7](/packages/nicoswd-php-rule-parser)[jeremeamia/functionparser

Function parser for PHP functions, methods, and closures

48174.6k6](/packages/jeremeamia-functionparser)[danny50610/bpe-tokeniser

PHP port for openai/tiktoken (most)

10457.3k2](/packages/danny50610-bpe-tokeniser)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)