PHPackages                             affinity4/tokenizer - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Utility & Helpers](/categories/utility)
4. /
5. affinity4/tokenizer

ActiveLibrary[Utility & Helpers](/categories/utility)

affinity4/tokenizer
===================

A zero-dedpendency tokenizer written in PHP. Returns an easily navigatable Stream object of Token objects with public type, value, offset and length properties

0.0.4(3y ago)071MITPHPPHP >=8.0

Since May 25Pushed 3y ago1 watchersCompare

[ Source](https://github.com/affinity4/tokenizer)[ Packagist](https://packagist.org/packages/affinity4/tokenizer)[ RSS](/packages/affinity4-tokenizer/feed)WikiDiscussions main Synced 1mo ago

READMEChangelog (4)Dependencies (1)Versions (5)Used By (0)

Tokenizer
=========

[](#tokenizer)

[![Affinity4](https://camo.githubusercontent.com/a7aeeac3c091902c692c002168e378ef2702bdcd17a3c4815be74edd59a66d16/68747470733a2f2f636972636c6563692e636f6d2f67682f616666696e697479342f746f6b656e697a65722e7376673f7374796c653d737667)](https://circleci.com/gh/affinity4/tokenizer)

A zero-dedpendency tokenizer written in PHP. Returns an easily navigatable Stream object of Token objects with public type, value, offset and length properties

Simply pass an associative array \[match\_pattern => type\] (`'\s+' => 'T_WHITESPACE', '[a-zA-Z]\w+' => 'T_STRING'`), and the Tokenizer will return all matches as an array of Token objects

Installation
------------

[](#installation)

### Composer

[](#composer)

`composer require affinity4/tokenizer`

Basic Example
-------------

[](#basic-example)

Let's assume we want to create a DSL (Domain Specific Language) for a template engine language that looks more like code, instead of markup

Example template snippet:

```
$template =