PHPackages                             xp-framework/tokenize - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Utility & Helpers](/categories/utility)
4. /
5. xp-framework/tokenize

ActiveLibrary[Utility & Helpers](/categories/utility)

xp-framework/tokenize
=====================

Text tokenizing

v9.1.0(2y ago)0177.0k↑23.5%7BSD-3-ClausePHPPHP >=7.0.0

Since Dec 8Pushed 1y ago1 watchersCompare

[ Source](https://github.com/xp-framework/tokenize)[ Packagist](https://packagist.org/packages/xp-framework/tokenize)[ Docs](http://xp-framework.net/)[ RSS](/packages/xp-framework-tokenize/feed)WikiDiscussions master Synced 1mo ago

READMEChangelog (10)Dependencies (2)Versions (12)Used By (7)

Tokenize
========

[](#tokenize)

[![Build status on GitHub](https://github.com/xp-framework/tokenize/workflows/Tests/badge.svg)](https://github.com/xp-framework/tokenize/actions)[![XP Framework Module](https://raw.githubusercontent.com/xp-framework/web/master/static/xp-framework-badge.png)](https://github.com/xp-framework/core)[![BSD Licence](https://raw.githubusercontent.com/xp-framework/web/master/static/licence-bsd.png)](https://github.com/xp-framework/core/blob/master/LICENCE.md)[![Requires PHP 7.0+](https://raw.githubusercontent.com/xp-framework/web/master/static/php-7_0plus.svg)](http://php.net/)[![Supports PHP 8.0+](https://raw.githubusercontent.com/xp-framework/web/master/static/php-8_0plus.svg)](http://php.net/)[![Latest Stable Version](https://camo.githubusercontent.com/cf1e99903d1907bf570595dc45ae1a57bb52528a4559302fa27afc84d81fad5d/68747470733a2f2f706f7365722e707567782e6f72672f78702d6672616d65776f726b2f746f6b656e697a652f76657273696f6e2e737667)](https://packagist.org/packages/xp-framework/tokenize)

Tokenizing text

```
use text\{StringTokenizer, StreamTokenizer};
use io\File;

// Supports strings and streams
$tokens= new StringTokenizer('He asked: Can you parse this?', ' .?!,;:', true);
$tokens= new StreamTokenizer((new File('parse-me.txt'))->in(), ' .?!,;:', true);

// Can iterate using foreach...
foreach ($tokens as $token) {
  Console::writeLine($token);
}

// ...or with an iterator API
while ($tokens->hasMoreTokens()) {
  Console::writeLine($tokens->nextToken());
}

// Returns: ["He", " ", "asked", ":", " ", "Can", " ", "you", " ", "parse", " ", "this", "?"]
```

###  Health Score

38

—

LowBetter than 85% of packages

Maintenance29

Infrequent updates — may be unmaintained

Popularity30

Limited adoption so far

Community14

Small or concentrated contributor base

Maturity65

Established project with proven stability

 Bus Factor1

Top contributor holds 100% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Every ~302 days

Recently: every ~394 days

Total

11

Last Release

785d ago

Major Versions

v6.6.0 → v7.0.02016-02-21

v7.1.0 → v8.0.02017-06-03

v8.1.1 → v9.0.02019-11-30

PHP version history (3 changes)v6.6.0PHP >=5.5.0

v8.0.0PHP >=5.6.0

v9.0.0PHP >=7.0.0

### Community

Maintainers

![](https://www.gravatar.com/avatar/07d18d882c8b4aaf3466432f64018214f2771eda333202175431ee7233795376?d=identicon)[thekid](/maintainers/thekid)

---

Top Contributors

[![thekid](https://avatars.githubusercontent.com/u/696742?v=4)](https://github.com/thekid "thekid (43 commits)")

---

Tags

phpstreamtokenizerxp-frameworkmodulexp

### Embed Badge

![Health badge](/badges/xp-framework-tokenize/health.svg)

```
[![Health](https://phpackages.com/badges/xp-framework-tokenize/health.svg)](https://phpackages.com/packages/xp-framework-tokenize)
```

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)