PHPackages                             sterzik/tokenizer - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Utility & Helpers](/categories/utility)
4. /
5. sterzik/tokenizer

ActiveLibrary[Utility & Helpers](/categories/utility)

sterzik/tokenizer
=================

Library for creating an universal tokenizer

01PHP

Since Oct 14Pushed 7mo agoCompare

[ Source](https://github.com/marek-sterzik/tokenizer)[ Packagist](https://packagist.org/packages/sterzik/tokenizer)[ RSS](/packages/sterzik-tokenizer/feed)WikiDiscussions master Synced 1mo ago

READMEChangelogDependenciesVersions (1)Used By (0)

The `Sterzik\Tokenizer` library
===============================

[](#the-sterziktokenizer-library)

This library is currently in the very first stage of implementation and therefore not really useful.

But maybe in the future, there will be an univerzal tokenizer library.

###  Health Score

16

—

LowBetter than 5% of packages

Maintenance45

Moderate activity, may be stable

Popularity1

Limited adoption so far

Community6

Small or concentrated contributor base

Maturity13

Early-stage or recently created project

 Bus Factor1

Top contributor holds 100% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

### Community

Maintainers

![](https://www.gravatar.com/avatar/9647fc2c8d6e7d851c6830c5d5d69bac65765fab19502759cfe5e12674db10f9?d=identicon)[marek.sterzik](/maintainers/marek.sterzik)

---

Top Contributors

[![marek-sterzik](https://avatars.githubusercontent.com/u/26008248?v=4)](https://github.com/marek-sterzik "marek-sterzik (1 commits)")

### Embed Badge

![Health badge](/badges/sterzik-tokenizer/health.svg)

```
[![Health](https://phpackages.com/badges/sterzik-tokenizer/health.svg)](https://phpackages.com/packages/sterzik-tokenizer)
```

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)