PHPackages                             hexydec/tokenise - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Parsing & Serialization](/categories/parsing)
4. /
5. hexydec/tokenise

ActiveLibrary[Parsing & Serialization](/categories/parsing)

hexydec/tokenise
================

A program for performing lexical analysis, written in PHP

1.0.3(10mo ago)311.2k↓31.3%1[1 PRs](https://github.com/hexydec/tokenise/pulls)3MITPHPPHP >=8.1CI passing

Since May 15Pushed 3mo ago1 watchersCompare

[ Source](https://github.com/hexydec/tokenise)[ Packagist](https://packagist.org/packages/hexydec/tokenise)[ Docs](https://github.com/hexydec/tokenise)[ RSS](/packages/hexydec-tokenise/feed)WikiDiscussions master Synced 1mo ago

READMEChangelog (8)Dependencies (2)Versions (10)Used By (3)

Tokenise: A Regexp Based PHP Tokeniser
======================================

[](#tokenise-a-regexp-based-php-tokeniser)

A program for performing lexical analysis, written in PHP. Designed to supply tokens to a parser in order to analyse the syntax of programming languages.

[![Licence](https://camo.githubusercontent.com/1be08aa7893603196bc3465a2235df43f54fab679741e5cdb07b9f513c5dd5d7/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4c6963656e63652d4d49542d6c69676874677265792e737667)](https://camo.githubusercontent.com/1be08aa7893603196bc3465a2235df43f54fab679741e5cdb07b9f513c5dd5d7/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4c6963656e63652d4d49542d6c69676874677265792e737667)[![Status: Stable](https://camo.githubusercontent.com/af83a8dcabf200b08620cd9f1e69904403bdef64b5962d6ff3126c88446bf9e2/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f5374617475732d537461626c652d477265656e2e737667)](https://camo.githubusercontent.com/af83a8dcabf200b08620cd9f1e69904403bdef64b5962d6ff3126c88446bf9e2/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f5374617475732d537461626c652d477265656e2e737667)[![Tests Status](https://github.com/hexydec/tokenise/actions/workflows/tests.yml/badge.svg)](https://github.com/hexydec/tokenise/actions/workflows/tests.yml)[![Code Coverage](https://camo.githubusercontent.com/6a5bee23060745424386da3dd2e748260ccc9921581a0847e98711ff8ad438c2/68747470733a2f2f636f6465636f762e696f2f67682f686578796465632f746f6b656e6973652f6272616e63682f6d61737465722f67726170682f62616467652e737667)](https://app.codecov.io/gh/hexydec/tokenise)

Description
-----------

[](#description)

Using an array of regular expressions, tokens are extracted from the input code in sequence and upon request. Previous tokens can also be re-requested, and custom regular expressions can also be used to retrieve the next token.

Currently used in my other projects:

- [HTMLdoc: PHP HTML Document Parser and Minifier](http://github.com/hexydec/htmldoc)
- [CSSdoc: PHP CSS Document Parser and Minifier](http://github.com/hexydec/cssdoc)
- [JSlite: PHP Javascript Minifier](http://github.com/hexydec/jslite)

Usage
-----

[](#usage)

Basic usage within a program:

```
$obj = new \hexydec\tokens\tokenise($tokens, $value);

while (($token = $obj->next()) !== null) {
	// parse each token
}
```

API
---

[](#api)

For more detailed usage documentation, [read the API document](docs/api.md).

Support
-------

[](#support)

Tokenise supports PHP version 8.1+.

Contributing
------------

[](#contributing)

If you find an issue with tokenise, please create an issue in the tracker.

If you wish to fix an issue yourself, please fork the code, fix the issue, then create a pull request, and I will evaluate your submission.

Licence
-------

[](#licence)

The MIT License (MIT). Please see [License File](LICENCE) for more information.

###  Health Score

48

—

FairBetter than 95% of packages

Maintenance68

Regular maintenance activity

Popularity29

Limited adoption so far

Community15

Small or concentrated contributor base

Maturity66

Established project with proven stability

 Bus Factor1

Top contributor holds 100% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Every ~217 days

Recently: every ~355 days

Total

8

Last Release

307d ago

Major Versions

0.4.2 → 1.0.02022-03-04

PHP version history (3 changes)0.3.0PHP >=7.3

1.0.0PHP >=7.4

1.0.2PHP >=8.1

### Community

Maintainers

![](https://www.gravatar.com/avatar/5257ffcb9e543c03368e1090d353092b40b1491d15cd3126d14d3d32975b6384?d=identicon)[hexydec](/maintainers/hexydec)

---

Top Contributors

[![hexydec](https://avatars.githubusercontent.com/u/743478?v=4)](https://github.com/hexydec "hexydec (45 commits)")

---

Tags

parsertokenizerlexingtokenizationlexicaltokenisertokenisation

###  Code Quality

TestsPHPUnit

Static AnalysisPHPStan

Type Coverage Yes

### Embed Badge

![Health badge](/badges/hexydec-tokenise/health.svg)

```
[![Health](https://phpackages.com/badges/hexydec-tokenise/health.svg)](https://phpackages.com/packages/hexydec-tokenise)
```

###  Alternatives

[nikic/php-parser

A PHP parser written in PHP

17.4k902.6M1.8k](/packages/nikic-php-parser)[doctrine/lexer

PHP Doctrine Lexer parser library that can be used in Top-Down, Recursive Descent Parsers.

11.2k910.8M118](/packages/doctrine-lexer)[erusev/parsedown

Parser for Markdown.

15.0k151.8M732](/packages/erusev-parsedown)[league/commonmark

Highly-extensible PHP Markdown parser which fully supports the CommonMark spec and GitHub-Flavored Markdown (GFM)

3.0k404.0M702](/packages/league-commonmark)[masterminds/html5

An HTML5 parser and serializer.

1.8k242.8M229](/packages/masterminds-html5)[sabberworm/php-css-parser

Parser for CSS Files written in PHP

1.8k191.2M65](/packages/sabberworm-php-css-parser)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)