PHPackages                             org\_heigl/tokenizer - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Utility & Helpers](/categories/utility)
4. /
5. org\_heigl/tokenizer

ActiveLibrary[Utility & Helpers](/categories/utility)

org\_heigl/tokenizer
====================

Provides a way of tokenizing strings

12.3k

Since Jan 7Pushed 12y ago1 watchersCompare

[ Source](https://github.com/heiglandreas/Tokenizer)[ Packagist](https://packagist.org/packages/org_heigl/tokenizer)[ RSS](/packages/org-heigl-tokenizer/feed)WikiDiscussions master Synced 3d ago

READMEChangelogDependenciesVersions (2)Used By (0)

Tokenizer
=========

[](#tokenizer)

Provides ways to split strings into smaller entities depending on the used tokenizers.

You can chain different tokenizers to a tokenizer-chain to get the results you want.

Currently this library provides these Tokenizers:

- WhitespaceTokenizer to split strings on every whitespace. Can be used to split a sentence into single words.
- CamelCaseTokenizer to split CamelCased-Strings into separate tokens.

[![Build Status](https://camo.githubusercontent.com/0b98dd37509a87a3ddd8d986829cae3ef9e482b1165747fe50d972c595d5ab05/68747470733a2f2f7472617669732d63692e6f72672f686569676c616e64726561732f546f6b656e697a65722e706e67)](https://travis-ci.org/heiglandreas/Tokenizer)

Installation
------------

[](#installation)

Install using `composer` by adding the following line to your `composer.conf`-files `require`-Section:

```
"org_heigl/tokenizer" : "dev-master"

```

Usage
-----

[](#usage)

Usage is rather simple:

```
use Org_Heigl\Tokenizer\TokenizerQueue;
use Org_Heigl\Tokenizer\Tokenizers;
// Create a new Tokenizer-Queue
$tokenizer = new TokenizerQueue();

// Add single tokenizers to the queue
// First a Whitespace tokenizer
$tokenizer->addTokenizer(new Tokenizers\WhitespaceTokenizer());
// Then a CamelCase-Tokenizer
$tokenizer->addTokenizer(new Tokenizers\CamelCaseTokenizer());

// Finally tokenize a given string
$tokenList = $tokenizer->tokenize('A String with WhiteSpace');

var_dump((array) $tokenList);

// This will print the following:
/*
array(8) {
  [0] =>
  class Org_Heigl\Tokenizer\Token#216 (3) {
    protected $token =>
    string(1) "A"
    protected $offset =>
    int(0)
    protected $type =>
    string(6) "string"
  }
  [1] =>
  class Org_Heigl\Tokenizer\Token#215 (3) {
    protected $token =>
    string(1) " "
    protected $offset =>
    int(1)
    protected $type =>
    string(10) "whitespace"
  }
  [2] =>
  class Org_Heigl\Tokenizer\Token#214 (3) {
    protected $token =>
    string(6) "String"
    protected $offset =>
    int(2)
    protected $type =>
    string(6) "string"
  }
  [3] =>
  class Org_Heigl\Tokenizer\Token#213 (3) {
    protected $token =>
    string(1) " "
    protected $offset =>
    int(8)
    protected $type =>
    string(10) "whitespace"
  }
  [4] =>
  class Org_Heigl\Tokenizer\Token#212 (3) {
    protected $token =>
    string(4) "with"
    protected $offset =>
    int(9)
    protected $type =>
    string(6) "string"
  }
  [5] =>
  class Org_Heigl\Tokenizer\Token#211 (3) {
    protected $token =>
    string(1) " "
    protected $offset =>
    int(13)
    protected $type =>
    string(10) "whitespace"
  }
  [6] =>
  class Org_Heigl\Tokenizer\Token#209 (3) {
    protected $token =>
    string(5) "White"
    protected $offset =>
    int(14)
    protected $type =>
    string(6) "string"
  }
  [7] =>
  class Org_Heigl\Tokenizer\Token#208 (3) {
    protected $token =>
    string(5) "Space"
    protected $offset =>
    int(19)
    protected $type =>
    string(6) "string"
  }
}
*/

```

###  Health Score

24

—

LowBetter than 32% of packages

Maintenance20

Infrequent updates — may be unmaintained

Popularity17

Limited adoption so far

Community7

Small or concentrated contributor base

Maturity43

Maturing project, gaining track record

 Bus Factor1

Top contributor holds 100% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

### Community

Maintainers

![](https://www.gravatar.com/avatar/5ae5183aaad2bc7453230704bd6991dc6ccbcd6e775c6a29efdc94350a69f247?d=identicon)[heiglandreas](/maintainers/heiglandreas)

---

Top Contributors

[![heiglandreas](https://avatars.githubusercontent.com/u/91998?v=4)](https://github.com/heiglandreas "heiglandreas (19 commits)")

### Embed Badge

![Health badge](/badges/org-heigl-tokenizer/health.svg)

```
[![Health](https://phpackages.com/badges/org-heigl-tokenizer/health.svg)](https://phpackages.com/packages/org-heigl-tokenizer)
```

###  Alternatives

[vivait/string-generator-bundle

Generate random strings for IDs or keys using property annotations

1759.4k](/packages/vivait-string-generator-bundle)[withinboredom/time

Converting time to units

152.7k1](/packages/withinboredom-time)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)