PHPackages                             yuubit/tokenizer - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Parsing & Serialization](/categories/parsing)
4. /
5. yuubit/tokenizer

ActiveLibrary[Parsing & Serialization](/categories/parsing)

yuubit/tokenizer
================

PHP Library to tokenize various inputs.

1.0.0(7y ago)04031MITPHP >=7.0

Since Jan 3Compare

[ Source](https://github.com/Yuubit/Tokenizer)[ Packagist](https://packagist.org/packages/yuubit/tokenizer)[ RSS](/packages/yuubit-tokenizer/feed)WikiDiscussions Synced 3d ago

READMEChangelogDependencies (1)Versions (3)Used By (1)

### README not available

The README for this package hasn't been synced yet. View it on [GitHub](https://github.com/Yuubit/Tokenizer).

###  Health Score

26

—

LowBetter than 43% of packages

Maintenance20

Infrequent updates — may be unmaintained

Popularity12

Limited adoption so far

Community5

Small or concentrated contributor base

Maturity57

Maturing project, gaining track record

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Unknown

Total

1

Last Release

2653d ago

### Community

Maintainers

![](https://www.gravatar.com/avatar/ae540d6b696897ad9f2391f0a46a23079a06a257ad4ffbe61b0221057a0b7c94?d=identicon)[jozzan](/maintainers/jozzan)

---

Tags

lexertokenizeroop-librarylexical analasys

###  Code Quality

TestsPHPUnit

### Embed Badge

![Health badge](/badges/yuubit-tokenizer/health.svg)

```
[![Health](https://phpackages.com/badges/yuubit-tokenizer/health.svg)](https://phpackages.com/packages/yuubit-tokenizer)
```

###  Alternatives

[doctrine/lexer

PHP Doctrine Lexer parser library that can be used in Top-Down, Recursive Descent Parsers.

11.2k910.8M118](/packages/doctrine-lexer)[cerbero/json-parser

Zero-dependencies pull parser to read large JSON from any source in a memory-efficient way.

803474.6k5](/packages/cerbero-json-parser)[yethee/tiktoken

PHP version of tiktoken

1583.1M15](/packages/yethee-tiktoken)[creof/geo-parser

Parser for geography coordinate strings

624.4M15](/packages/creof-geo-parser)[nikic/phlexy

Lexing experiments in PHP

162570.9k13](/packages/nikic-phlexy)[creof/wkt-parser

Parser for well-known text (WKT) object strings

554.8M16](/packages/creof-wkt-parser)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)