PHPackages                             webrek/tokenizers - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Parsing & Serialization](/categories/parsing)
4. /
5. webrek/tokenizers

ActivePhp-ext[Parsing & Serialization](/categories/parsing)

webrek/tokenizers
=================

Native byte-level BPE (tiktoken-compatible: cl100k\_base/o200k\_base) plus WordPiece and Unigram tokenizers for PHP 8.3+, with a process-shared vocab cache. Loads HuggingFace tokenizer.json models and counts Claude/Gemini tokens via their APIs.

v0.1.0(today)00Apache-2.0PHP >=8.3

Since Jun 29Compare

[ Source](https://github.com/webrek/tokenizers)[ Packagist](https://packagist.org/packages/webrek/tokenizers)[ RSS](/packages/webrek-tokenizers/feed)WikiDiscussions Synced today

READMEChangelogDependenciesVersions (2)Used By (0)

### README not available

The README for this package hasn't been synced yet. View it on [GitHub](https://github.com/webrek/tokenizers).

###  Health Score

37

—

LowBetter than 81% of packages

Maintenance100

Actively maintained with recent releases

Popularity0

Limited adoption so far

Community2

Small or concentrated contributor base

Maturity38

Early-stage or recently created project

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Unknown

Total

1

Last Release

0d ago

### Community

Maintainers

![](https://www.gravatar.com/avatar/7d8deca81629993819087597b5ad7695976b02e3d014f038e26e985f35f569de?d=identicon)[webrek](/maintainers/webrek)

---

Tags

tokenstokenizerextensionllmbpetiktokenwordpieceunigram

### Embed Badge

![Health badge](/badges/webrek-tokenizers/health.svg)

```
[![Health](https://phpackages.com/badges/webrek-tokenizers/health.svg)](https://phpackages.com/packages/webrek-tokenizers)
```

###  Alternatives

[yethee/tiktoken

PHP version of tiktoken

1623.7M28](/packages/yethee-tiktoken)[danny50610/bpe-tokeniser

PHP port for openai/tiktoken (most)

10442.7k2](/packages/danny50610-bpe-tokeniser)[jeremeamia/functionparser

Function parser for PHP functions, methods, and closures

48172.5k6](/packages/jeremeamia-functionparser)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)