PHPackages                             thruster/tokenizer - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Parsing & Serialization](/categories/parsing)
4. /
5. thruster/tokenizer

ActiveLibrary[Parsing & Serialization](/categories/parsing)

thruster/tokenizer
==================

Thruster Tokenizer Component

1.0.0(9y ago)06MITPHPPHP >=7.1

Since Jan 31Pushed 9y ago1 watchersCompare

[ Source](https://github.com/ThrusterIO/tokenizer)[ Packagist](https://packagist.org/packages/thruster/tokenizer)[ Docs](https://thruster.io)[ RSS](/packages/thruster-tokenizer/feed)WikiDiscussions master Synced 2mo ago

READMEChangelog (1)Dependencies (1)Versions (2)Used By (0)

Tokenizer Component
===================

[](#tokenizer-component)

[![Latest Version](https://camo.githubusercontent.com/2a4e012aa3da734be668279a5242bbac5a4cf52c678c909ced32a847d6e0494a/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f72656c656173652f5468727573746572494f2f746f6b656e697a65722e7376673f7374796c653d666c61742d737175617265)](https://github.com/ThrusterIO/tokenizer/releases)[![Software License](https://camo.githubusercontent.com/55c0218c8f8009f06ad4ddae837ddd05301481fcf0dff8e0ed9dadda8780713e/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d4d49542d627269676874677265656e2e7376673f7374796c653d666c61742d737175617265)](LICENSE)[![Build Status](https://camo.githubusercontent.com/15832c8fb5809f66b96b10e254154dc3eaa95903407ae9af23b7d25b7851e465/68747470733a2f2f696d672e736869656c64732e696f2f7472617669732f5468727573746572494f2f746f6b656e697a65722e7376673f7374796c653d666c61742d737175617265)](https://travis-ci.org/ThrusterIO/tokenizer)[![Code Coverage](https://camo.githubusercontent.com/45fe5f3799c31956b6c658ab2541fc10740f49013750996d1e1fbc33ef864a50/68747470733a2f2f696d672e736869656c64732e696f2f7363727574696e697a65722f636f7665726167652f672f5468727573746572494f2f746f6b656e697a65722e7376673f7374796c653d666c61742d737175617265)](https://scrutinizer-ci.com/g/ThrusterIO/tokenizer)[![Quality Score](https://camo.githubusercontent.com/a1066f675b9b81919905af7dd6dae2ccde411ab1b42b543509007b38d96bbf96/68747470733a2f2f696d672e736869656c64732e696f2f7363727574696e697a65722f672f5468727573746572494f2f746f6b656e697a65722e7376673f7374796c653d666c61742d737175617265)](https://scrutinizer-ci.com/g/ThrusterIO/tokenizer)[![Total Downloads](https://camo.githubusercontent.com/e8a0fce7495cddd9a219660f0e2f9c45dcbbb816c40a61ebfc2454a26aecbdae/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f64742f74687275737465722f746f6b656e697a65722e7376673f7374796c653d666c61742d737175617265)](https://packagist.org/packages/thruster/tokenizer)

[![Email](https://camo.githubusercontent.com/e3e4a46ec999fd7bad8871d0ce2a60123a3b8a79f04c305bfb2b976482bbcfbb/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f656d61696c2d617572696d6173406e69656b69732e6c742d626c75652e7376673f7374796c653d666c61742d737175617265)](mailto:aurimas@niekis.lt)

The Thruster Tokenizer Component.

Install
-------

[](#install)

Via Composer

```
$ composer require thruster/tokenizer
```

Usage
-----

[](#usage)

```
use Thruster\Component\Tokenizer\Parser;

$parser = new Parser();

$parser->parse(''); // => Token[]
$parser->parseBlock('function fooBar(){...};'); // => Token[]
$parser->parseFile('/some/file.php'); // => Token[]
```

Testing
-------

[](#testing)

```
$ composer test
```

Contributing
------------

[](#contributing)

Please see [CONTRIBUTING](CONTRIBUTING.md) and [CONDUCT](CONDUCT.md) for details.

License
-------

[](#license)

Please see [License File](LICENSE) for more information.

###  Health Score

25

—

LowBetter than 37% of packages

Maintenance20

Infrequent updates — may be unmaintained

Popularity4

Limited adoption so far

Community7

Small or concentrated contributor base

Maturity58

Maturing project, gaining track record

 Bus Factor1

Top contributor holds 100% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Unknown

Total

1

Last Release

3391d ago

### Community

Maintainers

![](https://www.gravatar.com/avatar/770bae1f46ac2b4536db820b8642491c25e2e2fd44e761110269e4722679a80d?d=identicon)[aurimasniekis](/maintainers/aurimasniekis)

---

Top Contributors

[![aurimasniekis](https://avatars.githubusercontent.com/u/15481?v=4)](https://github.com/aurimasniekis "aurimasniekis (2 commits)")

---

Tags

componentphpthrusterthrusteriotokenizertokenstokenizerthruster

###  Code Quality

TestsPHPUnit

### Embed Badge

![Health badge](/badges/thruster-tokenizer/health.svg)

```
[![Health](https://phpackages.com/badges/thruster-tokenizer/health.svg)](https://phpackages.com/packages/thruster-tokenizer)
```

###  Alternatives

[yethee/tiktoken

PHP version of tiktoken

1583.1M15](/packages/yethee-tiktoken)[gioni06/gpt3-tokenizer

PHP package for Byte Pair Encoding (BPE) used by GPT-3.

85537.5k8](/packages/gioni06-gpt3-tokenizer)[rajentrivedi/tokenizer-x

TokenizerX calculates required tokens for given prompt

91214.0k3](/packages/rajentrivedi-tokenizer-x)[nicoswd/php-rule-parser

Rule Engine - Rule Parser & Evaluator

13078.6k7](/packages/nicoswd-php-rule-parser)[jeremeamia/functionparser

Function parser for PHP functions, methods, and closures

48169.7k6](/packages/jeremeamia-functionparser)[danny50610/bpe-tokeniser

PHP port for openai/tiktoken (most)

10422.0k1](/packages/danny50610-bpe-tokeniser)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)