PHPackages                             tanuel/tokenizer - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Parsing & Serialization](/categories/parsing)
4. /
5. tanuel/tokenizer

ActiveLibrary[Parsing & Serialization](/categories/parsing)

tanuel/tokenizer
================

A tokenizer for PHP

v0.1.1(6y ago)515[1 issues](https://github.com/Tanuel/php-tokenizer/issues)[1 PRs](https://github.com/Tanuel/php-tokenizer/pulls)GPL-3.0PHPPHP >=7.1CI failing

Since Oct 5Pushed 6y ago1 watchersCompare

[ Source](https://github.com/Tanuel/php-tokenizer)[ Packagist](https://packagist.org/packages/tanuel/tokenizer)[ Docs](https://github.com/Tanuel/php-tokenizer)[ RSS](/packages/tanuel-tokenizer/feed)WikiDiscussions master Synced 3d ago

READMEChangelog (2)Dependencies (2)Versions (4)Used By (0)

tanuel/tokenizer
================

[](#tanueltokenizer)

### Lightweight Zero-Dependency Tokenizer for PHP

[](#lightweight-zero-dependency-tokenizer-for-php)

[![PHP from Packagist](https://camo.githubusercontent.com/7c55a9e6a9ae44e59783d202bdc92ed1c1713f24ce4cb7760f11e771ef0e191d/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f7068702d762f74616e75656c2f746f6b656e697a6572)](https://camo.githubusercontent.com/7c55a9e6a9ae44e59783d202bdc92ed1c1713f24ce4cb7760f11e771ef0e191d/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f7068702d762f74616e75656c2f746f6b656e697a6572)[![Packagist Version](https://camo.githubusercontent.com/e4a6804fb2d46621cc17337c88245f175d77b9b8c9e24640a3ffa58fd39fd44b/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f762f74616e75656c2f746f6b656e697a6572)](https://packagist.org/packages/tanuel/tokenizer)[![Packagist](https://camo.githubusercontent.com/42a86b48ba75e9674feef4625e47df632d1459d0eb76c506eb63a1a78ae5c196/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f64742f74616e75656c2f746f6b656e697a6572)](https://packagist.org/packages/tanuel/tokenizer)[![License: GPL-3.0](https://camo.githubusercontent.com/1610da3ec5848e67c9b226ba9a44e0825a8a4b13ca1edb3a752f7a35bbb72f58/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f6c2f74616e75656c2f746f6b656e697a6572)](https://www.gnu.org/licenses/gpl-3.0.en.html)

This is a simple but powerful tokenizer written in php where you can pass your own token definitions.

Setup
-----

[](#setup)

```
composer install tanuel/tokenizer

```

Usage
-----

[](#usage)

**Tip:** Take a look at the [unit tests](./test/TokenizerTest.php) for examples

### 1. Create a token definition with regex patterns

[](#1-create-a-token-definition-with-regex-patterns)

```