PHPackages                             phootwork/tokenizer - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Parsing & Serialization](/categories/parsing)
4. /
5. phootwork/tokenizer

ActiveLibrary[Parsing & Serialization](/categories/parsing)

phootwork/tokenizer
===================

Tokenizer for PHP code

v3.2.3(1y ago)2212.8k—0%12MITPHPPHP >=8.0

Since May 12Pushed 1y ago2 watchersCompare

[ Source](https://github.com/phootwork/tokenizer)[ Packagist](https://packagist.org/packages/phootwork/tokenizer)[ RSS](/packages/phootwork-tokenizer/feed)WikiDiscussions master Synced 1mo ago

READMEChangelog (10)Dependencies (1)Versions (17)Used By (2)

PHP Tokenizer Library
=====================

[](#php-tokenizer-library)

[![Tests](https://github.com/phootwork/phootwork/workflows/Tests/badge.svg)](https://github.com/phootwork/phootwork/workflows/Tests/badge.svg)[![Scrutinizer Code Quality](https://camo.githubusercontent.com/69bd7e143cbdfb19d74ce75708a9f883ef5e8c3e12e93af074e908c564d06c52/68747470733a2f2f7363727574696e697a65722d63692e636f6d2f672f70686f6f74776f726b2f70686f6f74776f726b2f6261646765732f7175616c6974792d73636f72652e706e673f623d6d6173746572)](https://scrutinizer-ci.com/g/phootwork/phootwork/?branch=master)[![Code Coverage](https://camo.githubusercontent.com/826ef921280a14b18e896b455eda961bf8c53758c493844143b31e66791a06d5/68747470733a2f2f7363727574696e697a65722d63692e636f6d2f672f70686f6f74776f726b2f70686f6f74776f726b2f6261646765732f636f7665726167652e706e673f623d6d6173746572)](https://scrutinizer-ci.com/g/phootwork/phootwork/?branch=master)[![License](https://camo.githubusercontent.com/e61c02ff13136c356270233f9cd7b8deb4f0d2723daecbbbe62d154bae3f59e3/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f6c6963656e73652f70686f6f74776f726b2f746f6b656e697a65722e7376673f7374796c653d666c61742d737175617265)](https://packagist.org/packages/phootwork/tokenizer)[![Latest Stable Version](https://camo.githubusercontent.com/baff462b0763de2fc80830583eec40ec66304abec53166de68b961bc3ffe3dc0/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f762f70686f6f74776f726b2f746f6b656e697a65722e7376673f7374796c653d666c61742d737175617265)](https://packagist.org/packages/phootwork/tokenizer)[![Total Downloads](https://camo.githubusercontent.com/91aec4475bc0d2831c790ebbc63fd1846d93f5b31e42a055a04a62764a44d14e/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f64742f70686f6f74776f726b2f746f6b656e697a65722e7376673f7374796c653d666c61742d73717561726526636f6c6f72423d303037656336)](https://packagist.org/packages/phootwork/tokenizer)

Tokenizer for PHP code.

Installation
------------

[](#installation)

Install via Composer:

```
composer require phootwork/tokenizer

```

Running tests
-------------

[](#running-tests)

This package is a part of the Phootwork library. In order to run the test suite, you have to download the full library.

```
git clone https://github.com/phootwork/phootwork

```

Then install the dependencies via composer:

```
composer install

```

Now, run the *tokenizer* test suite:

```
vendor/bin/phpunit --testsuite tokenizer

```

If you want to run the whole library tests, simply run:

```
vendor/bin/phpunit

```

Contact
-------

[](#contact)

Report issues at the github [Issue Tracker](https://github.com/phootwork/phootwork/issues).

Changelog
---------

[](#changelog)

Refer to [Releases](https://github.com/phootwork/phootwork/releases)

###  Health Score

45

—

FairBetter than 93% of packages

Maintenance36

Infrequent updates — may be unmaintained

Popularity34

Limited adoption so far

Community18

Small or concentrated contributor base

Maturity76

Established project with proven stability

 Bus Factor1

Top contributor holds 50% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Every ~229 days

Recently: every ~326 days

Total

16

Last Release

592d ago

Major Versions

v0.3 → v2.0-alpha2019-09-08

v2.1.2 → v3.02021-03-08

PHP version history (3 changes)v0.3PHP >=5.6

v2.0-alphaPHP >=7.2

v3.0PHP >=8.0

### Community

Maintainers

![](https://www.gravatar.com/avatar/0b1ca3beb32834f0f6a4b66d7a695ecc01397aa92e9a97bd0ba5dd8aa14e0c6e?d=identicon)[gossi](/maintainers/gossi)

![](https://www.gravatar.com/avatar/fb5ce96caf1e78aeefa09e75b49f21cc5214729ac6a9a06b852cf5de1269e058?d=identicon)[cristianoc72](/maintainers/cristianoc72)

---

Top Contributors

[![gossi](https://avatars.githubusercontent.com/u/283700?v=4)](https://github.com/gossi "gossi (12 commits)")[![cristianoc72](https://avatars.githubusercontent.com/u/865437?v=4)](https://github.com/cristianoc72 "cristianoc72 (10 commits)")[![andypost](https://avatars.githubusercontent.com/u/73713?v=4)](https://github.com/andypost "andypost (1 commits)")[![peter279k](https://avatars.githubusercontent.com/u/9021747?v=4)](https://github.com/peter279k "peter279k (1 commits)")

---

Tags

phptokenizertokenizer

### Embed Badge

![Health badge](/badges/phootwork-tokenizer/health.svg)

```
[![Health](https://phpackages.com/badges/phootwork-tokenizer/health.svg)](https://phpackages.com/packages/phootwork-tokenizer)
```

###  Alternatives

[yethee/tiktoken

PHP version of tiktoken

1583.1M15](/packages/yethee-tiktoken)[gioni06/gpt3-tokenizer

PHP package for Byte Pair Encoding (BPE) used by GPT-3.

85537.5k8](/packages/gioni06-gpt3-tokenizer)[rajentrivedi/tokenizer-x

TokenizerX calculates required tokens for given prompt

91214.0k3](/packages/rajentrivedi-tokenizer-x)[nicoswd/php-rule-parser

Rule Engine - Rule Parser & Evaluator

13078.6k7](/packages/nicoswd-php-rule-parser)[jeremeamia/functionparser

Function parser for PHP functions, methods, and closures

48169.7k6](/packages/jeremeamia-functionparser)[danny50610/bpe-tokeniser

PHP port for openai/tiktoken (most)

10422.0k1](/packages/danny50610-bpe-tokeniser)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)