PHPackages alexandrebrach/tokentree - PHPackages - PHPackages [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register) 1. [Directory](/) 2. / 3. [Parsing & Serialization](/categories/parsing) 4. / 5. alexandrebrach/tokentree ActiveLibrary[Parsing & Serialization](/categories/parsing) alexandrebrach/tokentree ======================== Tool to parse file. It provide results in a "tree of token" manner 1.0.1(7y ago)030PHP Since Aug 10Pushed 7y ago1 watchersCompare [ Source](https://github.com/AlexandreBrach/tokentree)[ Packagist](https://packagist.org/packages/alexandrebrach/tokentree)[ RSS](/packages/alexandrebrach-tokentree/feed)WikiDiscussions master Synced today READMEChangelogDependencies (1)Versions (4)Used By (0) PHP Parser Library ================== [](#php-parser-library) This library parse your text and convert it to tokens based on rules you have define. You can define tokens on multiple layers to achieve complex parsing, resulting on an array of array of tokens. (or in an n-level array of tokens, depending on the depth of the rules you have defined) Install libraries ================= [](#install-libraries) ``` composer require alexandrebrach/tokentree ``` Run the unit tests ================== [](#run-the-unit-tests) ``` phpunit tests ``` How to use it ============= [](#how-to-use-it) Define token extraction rules ----------------------------- [](#define-token-extraction-rules) To extract each tokens, rules must be defined in classes extending `Tokentree\Lexic\Lexic`. Each rule can be : - a simple string (a comma or a keyword like `class`) - a regular expression (to extract a `$dollarBaseNamedVariable`) - a callback function to handle critical things (think about `SET_DELIMITER` SQL instruction) Exemple with SQL Parsing (this is for illustration purpose and may not be functionnal. A complete SQL tokenizer is provided in the code) The following explain how to parse a SQL `CREATE TABLE` statement using 3 layers of tokens. The first class is able to extract any word or character that may appear in a SQL `CREATE TABLE` statement. They will be converted to tokens. Each token will have a token-type (`CREATE_TABLE`, `NAME`, etc), and a value (the value of the string or the resulting regular expresion matches) ``` class Token extends Lexic { protected function _init() { $this ->_addRule( 'CREATE_TABLE', 'regexp', '|^(CREATE\s*TABLE(\s*IF\s*NOT\s*EXISTS)?)|' ) ->_addRule( 'NAME', 'regexp', '|^`([^`]*)`|' ) ->_addRule( 'DATATYPE', 'regexp', '/^(\b(int|tinyint|float|decimal|text|varchar|datetime|date|timestamp|time|point|char)\b(\s*\(\s*(\d*(\s*\,\d*)?)\s*\))?)/' ) ->_addRule( 'CHARSET', 'regexp', '|^CHARACTER\s*SET\s*(\S*)|' ) ->_addRule( 'UNSIGNED', 'string', 'UNSIGNED' ) ->_addRule( 'NOT_NULL', 'string', 'NOT NULL' ) ->_addRule( 'NULL', 'string', 'NULL' ) ->_addRule( 'DEFAULT', 'string', 'DEFAULT' ) ->_addRule( 'STRING', 'regexp', '|^\'(.*)\'|' ) ->_addRule( 'STRING', 'regexp' ,'|^\"(.*)\"|' ) ->_addRule( 'EXPRESSION', 'regexp', '|^(CURRENT_TIMESTAMP)|' ) ->_addRule( 'VALUE', 'regexp', '|^(DEFAULT\s*CHARSET\s*)|') ->_addRule( 'VALUE', 'regexp', '|^([a-zA-Z_0-9]+)|') ->_addRule( 'ON', 'string', 'ON' ) ->_addRule( 'ACTION', 'regexp', '/^(INSERT|UPDATE|DELETE)/' ) ->_addRule( 'AUTO_INCREMENT', 'string', 'AUTO_INCREMENT' ) ->_addRule( 'SQLCOMMENT', 'string', 'COMMENT' ) ->_addRule( 'KEY', 'regexp', '/^(PRIMARY|UNIQUE)?\s*KEY/' ) ->_addRule( 'USING', 'string', 'USING' ) ->_addRule( 'CONSTRAINT', 'string', 'CONSTRAINT' ) ->_addRule( 'FOREIGN_KEY', 'regexp', '|^(FOREIGN\s*KEY)|' ) ->_addRule( 'REFERENCES', 'string', 'REFERENCES' ) ->_addRule( 'TRIGGER_ACTION', 'regexp', '/^(CASCADE|RESTRICT|SET\s*NULL|NO\s*ACTION)/' ) ->_addRule( 'ALIAS', 'regexp', '|^@+([a-zA-Z_0-9]+)|') ; } } ``` Once finished, the parser will concatenate every token-type value found in the first layer. The resulting string will be parsed by the second layer in the same manner than previously. So, it provide a upper abstraction-level using the previous tokens : it is able to reconize the begining of a `CREATE TABLE` statement, a colum definition, etc. ``` class Expression extends Lexic { protected function _init() { $this ->_addRule( 'CREATE_TABLE', 'regexp', '/^CREATE_TABLE\s*NAME/' ) ->_addRule( 'COLUMN', 'regexp', '/^(NAME\s*DATATYPE\s*(CHARSET)?\s*(UNSIGNED)?\s*(NULL|NOT_NULL)?\s*(DEFAULT)?\s*(STRING|NULL|EXPRESSION|VALUE)?\s*(ON ACTION)?\s*(STRING|NULL|EXPRESSION)?\s*(AUTO_INCREMENT)?\s*(SQLCOMMENT\s*STRING)?)\s*/' ) ->_addRule( 'KEY', 'regexp', '/^(KEY\s*(NAME)?\s*\(\s*(NAME(\s*\,\s*NAME\s*)*)\s*\)\s*(USING\s*VALUE)?)/' ) ->_addRule( 'FOREIGN_KEY', 'regexp', '/^(CONSTRAINT\s*NAME\s*FOREIGN_KEY\s*\(\s*NAME\s*\)\s*REFERENCES\s*NAME\s*\(\s*NAME\s*\)\s*(ON\s*ACTION\s*TRIGGER_ACTION\s*)*)/' ) ->_addRule( 'ASSIGNMENT', 'regexp', '/^((VALUE|ALIAS|AUTO_INCREMENT) \= (VALUE|ALIAS|STRING))/' ) ->_addRule( 'ASSIGNMENT', 'regexp', '/^((NAMES) (VALUE))/' ) ; } } ``` Finaly, in the same manner than we go from layer 1 to layer 2, we concatenate the value of each token of layer 2 and provide the resulting string to the layer 3. This last layer is able to reconize the structure of any `CREATE TABLE` statement using the tokens of the second layer. An unique rule is enough to achieve this : ``` class Statement extends Lexic { protected function _init() { $this ->_addRule( 'CREATE_TABLE', 'regexp', '/^(CREATE_TABLE\s*\(\s*COLUMN(\s*\,\s*(COLUMN|KEY|FOREIGN_KEY))*\s*\)(\s*ASSIGNMENT)*)/' ) } } ``` Use the layers to parse strings ------------------------------- [](#use-the-layers-to-parse-strings) Use the tree previous classes (in the correct order) to define the multilayer parsing engine : ``` $tokenTree = new \Tokentree\Tokentree(); $tokenTree->setLexics( [ new Token(), new Expression(), new Statement() ] ); ``` Parse the string : ``` $tokenizer->tokenize( $sql ); ``` Get the resulting tokens : ``` $tokens = $tokenizer->shiftAllStack(); ``` ### Health Score 28 — LowBetter than 52% of packages Maintenance20 Infrequent updates — may be unmaintained Popularity7 Limited adoption so far Community7 Small or concentrated contributor base Maturity65 Established project with proven stability Bus Factor1 Top contributor holds 100% of commits — single point of failure How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window. **Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores. **Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement. **Maturity (30%)** — Project age, version count, PHP version support, and release stability. ### Release Activity Cadence Every ~8 days Total 2 Last Release 2872d ago ### Community Maintainers ![](https://www.gravatar.com/avatar/e82fdedcec9a42023dc7a01ddfcb0aa90d35a8edf7e0d64a788aa6d5e67ac816?d=identicon)[alexandre.brach](/maintainers/alexandre.brach) --- Top Contributors [![AlexandreBrach](https://avatars.githubusercontent.com/u/594612?v=4)](https://github.com/AlexandreBrach "AlexandreBrach (2 commits)") ### Code Quality TestsPHPUnit ### Embed Badge ![Health badge](/badges/alexandrebrach-tokentree/health.svg) ``` [![Health](https://phpackages.com/badges/alexandrebrach-tokentree/health.svg)](https://phpackages.com/packages/alexandrebrach-tokentree) ``` ### Alternatives [mck89/peast Peast is PHP library that generates AST for JavaScript code 19037.7M41](/packages/mck89-peast)[sauladam/shipment-tracker Parses tracking information for several carriers, like UPS, USPS, DHL and GLS by simply scraping the data. No need for any kind of API access. 9642.0k](/packages/sauladam-shipment-tracker)[jstewmc/rtf Read and write Rich Text Format (RTF) documents with PHP 46143.1k6](/packages/jstewmc-rtf)[moonshine/layouts-field Field for repeating groups of fields for MoonShine 107.9k](/packages/moonshine-layouts-field)[tcds-io/php-jackson A lightweight, flexible object serializer for PHP, inspired by FasterXML/jackson 112.9k10](/packages/tcds-io-php-jackson) PHPackages © 2026 [Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)