PHPackages                             yuubit/tokenizer - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Parsing & Serialization](/categories/parsing)
4. /
5. yuubit/tokenizer

ActiveLibrary[Parsing & Serialization](/categories/parsing)

yuubit/tokenizer
================

PHP Library to tokenize various inputs.

1.0.0(7y ago)04031MITPHPPHP >=7.0

Since Jan 3Pushed 7y ago1 watchersCompare

[ Source](https://github.com/Yuubit/Tokenizer)[ Packagist](https://packagist.org/packages/yuubit/tokenizer)[ RSS](/packages/yuubit-tokenizer/feed)WikiDiscussions master Synced 2w ago

READMEChangelog (1)Dependencies (1)Versions (3)Used By (1)

Tokenizer
=========

[](#tokenizer)

[![Build Status](https://camo.githubusercontent.com/b40a926193b3c2b79b8838f5d29fe1b032d6bacb514313cf172da19db3daa505/68747470733a2f2f7472617669732d63692e6f72672f5975756269742f546f6b656e697a65722e7376673f6272616e63683d312e302e302d534e415053484f54)](https://travis-ci.org/Yuubit/Tokenizer)

###  Health Score

27

—

LowBetter than 47% of packages

Maintenance20

Infrequent updates — may be unmaintained

Popularity12

Limited adoption so far

Community9

Small or concentrated contributor base

Maturity57

Maturing project, gaining track record

 Bus Factor1

Top contributor holds 100% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Unknown

Total

1

Last Release

2701d ago

### Community

Maintainers

![](https://avatars.githubusercontent.com/u/66067242?v=4)[jozzan](/maintainers/jozzan)[@Jozzan](https://github.com/Jozzan)

---

Top Contributors

[![reenigneEsrever92](https://avatars.githubusercontent.com/u/9577952?v=4)](https://github.com/reenigneEsrever92 "reenigneEsrever92 (2 commits)")

---

Tags

lexertokenizeroop-librarylexical analasys

###  Code Quality

TestsPHPUnit

### Embed Badge

![Health badge](/badges/yuubit-tokenizer/health.svg)

```
[![Health](https://phpackages.com/badges/yuubit-tokenizer/health.svg)](https://phpackages.com/packages/yuubit-tokenizer)
```

###  Alternatives

[doctrine/lexer

PHP Doctrine Lexer parser library that can be used in Top-Down, Recursive Descent Parsers.

11.2k942.7M159](/packages/doctrine-lexer)[cerbero/json-parser

Zero-dependencies pull parser to read large JSON from any source in a memory-efficient way.

806588.3k5](/packages/cerbero-json-parser)[yethee/tiktoken

PHP version of tiktoken

1624.0M29](/packages/yethee-tiktoken)[creof/geo-parser

Parser for geography coordinate strings

624.6M15](/packages/creof-geo-parser)[nikic/phlexy

Lexing experiments in PHP

162599.0k14](/packages/nikic-phlexy)[creof/wkt-parser

Parser for well-known text (WKT) object strings

545.0M16](/packages/creof-wkt-parser)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)