PHPackages                             tmilos/lexer - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Parsing & Serialization](/categories/parsing)
4. /
5. tmilos/lexer

ActiveLibrary[Parsing & Serialization](/categories/parsing)

tmilos/lexer
============

Lexical analyzer with individual token definition with regular expressions

1.0.2(9y ago)211.7M—8.7%3[1 issues](https://github.com/tmilos/lexer/issues)2MITPHPPHP >=5.5

Since Dec 19Pushed 7y ago2 watchersCompare

[ Source](https://github.com/tmilos/lexer)[ Packagist](https://packagist.org/packages/tmilos/lexer)[ RSS](/packages/tmilos-lexer/feed)WikiDiscussions master Synced 1mo ago

READMEChangelog (3)Dependencies (2)Versions (4)Used By (2)

PHP lexical analyzer
====================

[](#php-lexical-analyzer)

PHP implementation of Lexical Analyzer.

[![Author](https://camo.githubusercontent.com/c66133f19d9fe3288aee0e4be5c2e4b63190edb4d032c77a58d62f29b4ca0c66/687474703a2f2f696d672e736869656c64732e696f2f62616467652f617574686f722d40746d696c6f732d626c75652e7376673f7374796c653d666c61742d737175617265)](https://twitter.com/tmilos77)[![Build Status](https://camo.githubusercontent.com/b85250beb413817793a4047057e375848abb62c9340dd16c27818060fee80d16/68747470733a2f2f7472617669732d63692e6f72672f746d696c6f732f6c657865722e7376673f6272616e63683d6d6173746572)](https://travis-ci.org/tmilos/lexer)[![Coverage Status](https://camo.githubusercontent.com/396b2cd4916fd5cba3cb7f3f5334b1e049da2418b78961cbc8d8f42c433fb4ac/68747470733a2f2f636f766572616c6c732e696f2f7265706f732f6769746875622f746d696c6f732f6c657865722f62616467652e7376673f6272616e63683d6d6173746572)](https://coveralls.io/github/tmilos/lexer?branch=master)[![License](https://camo.githubusercontent.com/03c41bc6f23f9315c1537403bb748892b22a88370da1800c0ecde37d4f98b26f/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f6c2f746d696c6f732f6c657865722e737667)](https://packagist.org/packages/tmilos/lexer)[![SensioLabsInsight](https://camo.githubusercontent.com/0b687aaa0b7e30c99dfde451d49c0d46f04b200284f58bd5dc4e173dfef100e3/68747470733a2f2f696e73696768742e73656e73696f6c6162732e636f6d2f70726f6a656374732f32303237663239622d623935302d343563612d613138352d3534613165333337396266372f736d616c6c2e706e67)](https://insight.sensiolabs.com/projects/2027f29b-b950-45ca-a185-54a1e3379bf7)

> **Warning**This is not a GENERATOR like classical lex is. It does not produce any php code. It's a simple plain scanner of the given input string and tokenizer into given set of tokens by matching regular expressions. Thus, at runtime you can change the token definition and use one same code for any token set.

Token Definition
================

[](#token-definition)

Tokens are defined with `TokenDefinition` class that holds token name and regular expression. Token name can be empty, and in that case, lexer will ignore/skip such tokens.

Lexer Configuration
===================

[](#lexer-configuration)

The lexer configuration holds a list of all token definitions. With `LexerArrayConfig` it can be easily created from an array where keys are regular expressions and values are names of tokens.

Full scan
=========

[](#full-scan)

Lexer's static method `scan($config, $input)` can be used to scan given input string and return an array of tokens.

Lexer with state
================

[](#lexer-with-state)

Instance of the `Lexer` class be used to walk trough scanned tokens with single look-ahead token.

It's similar in API to [doctrine/lexer](https://github.com/doctrine/lexer), just tokens are defined and scanned differently, w/out the need for recognizing the token type/name from the tokenized value - rather the token type/name is given by the same `TokenDefn`that gave the regex to recognize the token.

Examples
========

[](#examples)

```