PHPackages                             robinthehood/tokenizer - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Parsing & Serialization](/categories/parsing)
4. /
5. robinthehood/tokenizer

ActiveLibrary[Parsing & Serialization](/categories/parsing)

robinthehood/tokenizer
======================

With Tokenizer you can easily convert a string to a stream or array of tokens.

0.1.3(7y ago)09052MITPHP

Since Nov 8Pushed 7y ago1 watchersCompare

[ Source](https://github.com/RobinTheHood/Tokenizer)[ Packagist](https://packagist.org/packages/robinthehood/tokenizer)[ RSS](/packages/robinthehood-tokenizer/feed)WikiDiscussions master Synced 3d ago

READMEChangelogDependenciesVersions (4)Used By (2)

Tokenizer
=========

[](#tokenizer)

[![alt text](https://raw.githubusercontent.com/RobinTheHood/Tokenizer/master/docs/Example-Image.png)](https://raw.githubusercontent.com/RobinTheHood/Tokenizer/master/docs/Example-Image.png)

About
-----

[](#about)

With Tokenizer you can easily convert a string to a stream or array of tokens. You can use it for syntax highlighting or a syntax parser. If you wish more features feel free to contribute.

Installation
------------

[](#installation)

If Composer is installed globally, run

```
composer require robinthehood/tokenizer
```

How to use
----------

[](#how-to-use)

### Code

[](#code)

```
require 'vendor/autoload.php'; // if you are using Composer

$tokenizer = new RobinTheHood\Tokenizer\Tokenizer(__FILE__, 'token_php.php');
$tokens = $tokenizer->getAllTokens();
print_r($tokens);
```

### Result

[](#result)

You get an array of token objects:

```
Array
(
    [0] => Tokenizer\Token Object
        (
            [type] => T_UNKNOWN
            [value] =>