PHPackages                             hippophp/tokenizer - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Utility & Helpers](/categories/utility)
4. /
5. hippophp/tokenizer

AbandonedArchivedLibrary[Utility & Helpers](/categories/utility)

hippophp/tokenizer
==================

Wrapper around token\_get\_all

91.3k[2 issues](https://github.com/HippoPHP/Tokenizer/issues)1PHP

Since Nov 8Pushed 8y ago2 watchersCompare

[ Source](https://github.com/HippoPHP/Tokenizer)[ Packagist](https://packagist.org/packages/hippophp/tokenizer)[ RSS](/packages/hippophp-tokenizer/feed)WikiDiscussions master Synced 1w ago

READMEChangelogDependenciesVersions (3)Used By (1)

Tokenizer
=========

[](#tokenizer)

[![Build Status](https://camo.githubusercontent.com/21255e7b3318306c527ecd536062ab2b1abead0b49295ed169b0658b84e5288d/687474703a2f2f696d672e736869656c64732e696f2f7472617669732f486970706f5048502f546f6b656e697a65722e7376673f7374796c653d666c61742d737175617265)](https://travis-ci.org/HippoPHP/Tokenizer)[![Code Climate](https://camo.githubusercontent.com/e4af97f32767c4b8381963bff8156adb25273c8d66198864ffca987f40e86233/687474703a2f2f696d672e736869656c64732e696f2f636f6465636c696d6174652f6769746875622f486970706f5048502f546f6b656e697a65722e7376673f7374796c653d666c61742d737175617265)](https://codeclimate.com/github/HippoPHP/Tokenizer)[![Test Coverage](https://camo.githubusercontent.com/cfc4fa82e75b40c80ab61b32cc8e7b793a0ead5360c5f902707dd72cc5c0cafa/687474703a2f2f696d672e736869656c64732e696f2f636f6465636c696d6174652f636f7665726167652f6769746875622f486970706f5048502f546f6b656e697a65722e7376673f7374796c653d666c61742d737175617265)](https://codeclimate.com/github/HippoPHP/Tokenizer)[![Dependencies](https://camo.githubusercontent.com/cd50cd70d9afc273deedecacd47596e32a17068ce0c12447722b52ddbc2b5b27/687474703a2f2f7777772e76657273696f6e6579652e636f6d2f757365722f70726f6a656374732f3534356466306564656238646632323733333030303033652f62616467652e7376673f7374796c653d666c61742d737175617265)](http://www.versioneye.com/user/projects/545df0edeb8df2273300003e)

What is Tokenizer?
------------------

[](#what-is-tokenizer)

Tokenizer is a wrapper around the `token_get_all` function. It's built up of three classes:

1. `Token` - Each token returned from `token_get_all` is wrapped in this.
2. `TokenListIterator` - Implements `SeekableIterator` and `Countable` with extra methods to allow easy moving through tokens.
3. `Tokenizer` - Calls `token_get_all` on the source, turns each result into a `Token` representation and returns a `TokenListIterator` object.

Installation
------------

[](#installation)

You can install Tokenizer by adding `hippophp/tokenizer` to your `composer.json` file.

License
-------

[](#license)

HippoPHP Tokenizer is licensed under [The MIT License (MIT)](/LICENSE)

###  Health Score

26

—

LowBetter than 43% of packages

Maintenance16

Infrequent updates — may be unmaintained

Popularity23

Limited adoption so far

Community12

Small or concentrated contributor base

Maturity44

Maturing project, gaining track record

 Bus Factor1

Top contributor holds 94.8% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

### Community

Maintainers

![](https://www.gravatar.com/avatar/9e4934b15ff7d8cffabece955994bed29b31429f4c2e8f13fa0868c753bb1c30?d=identicon)[jbrooksuk](/maintainers/jbrooksuk)

---

Top Contributors

[![jbrooksuk](https://avatars.githubusercontent.com/u/246103?v=4)](https://github.com/jbrooksuk "jbrooksuk (128 commits)")[![GrahamCampbell](https://avatars.githubusercontent.com/u/2829600?v=4)](https://github.com/GrahamCampbell "GrahamCampbell (7 commits)")

### Embed Badge

![Health badge](/badges/hippophp-tokenizer/health.svg)

```
[![Health](https://phpackages.com/badges/hippophp-tokenizer/health.svg)](https://phpackages.com/packages/hippophp-tokenizer)
```

###  Alternatives

[asantibanez/livewire-resource-time-grid

Laravel Livewire resource time grid component

2298.1k](/packages/asantibanez-livewire-resource-time-grid)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)