PHPackages                             hexydec/tokenise - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Parsing & Serialization](/categories/parsing)
4. /
5. hexydec/tokenise

ActiveLibrary[Parsing & Serialization](/categories/parsing)

hexydec/tokenise
================

A program for performing lexical analysis, written in PHP

1.0.4(2mo ago)312.1k↓46.4%13MITPHPPHP >=8.4CI passing

Since May 15Pushed 2mo ago1 watchersCompare

[ Source](https://github.com/hexydec/tokenise)[ Packagist](https://packagist.org/packages/hexydec/tokenise)[ Docs](https://github.com/hexydec/tokenise)[ RSS](/packages/hexydec-tokenise/feed)WikiDiscussions master Synced 2d ago

READMEChangelog (9)Dependencies (4)Versions (14)Used By (3)

Tokenise: A Regexp Based PHP Tokeniser
======================================

[](#tokenise-a-regexp-based-php-tokeniser)

A program for performing lexical analysis, written in PHP. Designed to supply tokens to a parser in order to analyse the syntax of programming languages.

[![Licence](https://camo.githubusercontent.com/1be08aa7893603196bc3465a2235df43f54fab679741e5cdb07b9f513c5dd5d7/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4c6963656e63652d4d49542d6c69676874677265792e737667)](https://camo.githubusercontent.com/1be08aa7893603196bc3465a2235df43f54fab679741e5cdb07b9f513c5dd5d7/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4c6963656e63652d4d49542d6c69676874677265792e737667)[![Status: Stable](https://camo.githubusercontent.com/af83a8dcabf200b08620cd9f1e69904403bdef64b5962d6ff3126c88446bf9e2/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f5374617475732d537461626c652d477265656e2e737667)](https://camo.githubusercontent.com/af83a8dcabf200b08620cd9f1e69904403bdef64b5962d6ff3126c88446bf9e2/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f5374617475732d537461626c652d477265656e2e737667)[![Tests Status](https://github.com/hexydec/tokenise/actions/workflows/tests.yml/badge.svg)](https://github.com/hexydec/tokenise/actions/workflows/tests.yml)[![Code Coverage](https://camo.githubusercontent.com/6a5bee23060745424386da3dd2e748260ccc9921581a0847e98711ff8ad438c2/68747470733a2f2f636f6465636f762e696f2f67682f686578796465632f746f6b656e6973652f6272616e63682f6d61737465722f67726170682f62616467652e737667)](https://app.codecov.io/gh/hexydec/tokenise)

Description
-----------

[](#description)

Using an array of regular expressions, tokens are extracted from the input code in sequence and upon request. Previous tokens can also be re-requested, and custom regular expressions can also be used to retrieve the next token.

Currently used in my other projects:

- [HTMLdoc: PHP HTML Document Parser and Minifier](http://github.com/hexydec/htmldoc)
- [CSSdoc: PHP CSS Document Parser and Minifier](http://github.com/hexydec/cssdoc)
- [JSlite: PHP Javascript Minifier](http://github.com/hexydec/jslite)

Usage
-----

[](#usage)

Basic usage within a program:

```
$obj = new \hexydec\tokens\tokenise($tokens, $value);

while (($token = $obj->next()) !== null) {
	// parse each token
}
```

API
---

[](#api)

For more detailed usage documentation, [read the API document](docs/api.md).

Support
-------

[](#support)

Tokenise supports PHP version 8.1+.

Contributing
------------

[](#contributing)

If you find an issue with tokenise, please create an issue in the tracker.

If you wish to fix an issue yourself, please fork the code, fix the issue, then create a pull request, and I will evaluate your submission.

Licence
-------

[](#licence)

The MIT License (MIT). Please see [License File](LICENCE) for more information.

###  Health Score

56

—

FairBetter than 97% of packages

Maintenance88

Actively maintained with recent releases

Popularity29

Limited adoption so far

Community16

Small or concentrated contributor base

Maturity77

Established project with proven stability

 Bus Factor1

Top contributor holds 100% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Every ~201 days

Recently: every ~331 days

Total

10

Last Release

60d ago

Major Versions

0.4.2 → 1.0.02022-03-04

PHP version history (4 changes)0.3.0PHP >=7.3

1.0.0PHP >=7.4

1.0.2PHP >=8.1

1.0.4PHP >=8.4

### Community

Maintainers

![](https://avatars.githubusercontent.com/u/743478?v=4)[Will Earp / Hexydec](/maintainers/hexydec)[@hexydec](https://github.com/hexydec)

---

Top Contributors

[![hexydec](https://avatars.githubusercontent.com/u/743478?v=4)](https://github.com/hexydec "hexydec (55 commits)")

---

Tags

parsertokenizerlexingtokenizationlexicaltokenisertokenisation

###  Code Quality

TestsPHPUnit

Static AnalysisPHPStan

Type Coverage Yes

### Embed Badge

![Health badge](/badges/hexydec-tokenise/health.svg)

```
[![Health](https://phpackages.com/badges/hexydec-tokenise/health.svg)](https://phpackages.com/packages/hexydec-tokenise)
```

###  Alternatives

[nikic/php-parser

A PHP parser written in PHP

17.4k954.1M2.5k](/packages/nikic-php-parser)[doctrine/lexer

PHP Doctrine Lexer parser library that can be used in Top-Down, Recursive Descent Parsers.

11.2k959.8M160](/packages/doctrine-lexer)[erusev/parsedown

Parser for Markdown.

15.1k156.8M868](/packages/erusev-parsedown)[masterminds/html5

An HTML5 parser and serializer.

1.8k269.7M322](/packages/masterminds-html5)[matomo/device-detector

The Universal Device Detection library, that parses User Agents and detects devices (desktop, tablet, mobile, tv, cars, console, etc.), clients (browsers, media players, mobile apps, feed readers, libraries, etc), operating systems, devices, brands and models.

3.5k26.4M182](/packages/matomo-device-detector)[jakubledl/dissect

Lexing and parsing in pure PHP

2234.7M11](/packages/jakubledl-dissect)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)