PHPackages                             hike/hike - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Parsing & Serialization](/categories/parsing)
4. /
5. hike/hike

ActiveLibrary[Parsing & Serialization](/categories/parsing)

hike/hike
=========

A consistent interface to tokenize, analyse and modify any text format. Programming languages, natural languages, markup languages

v0.1.1(1y ago)15BSD-2-ClausePHP

Since Apr 6Pushed 1y ago1 watchersCompare

[ Source](https://github.com/hike-php/hike)[ Packagist](https://packagist.org/packages/hike/hike)[ Docs](http://github.com/hike-php/hike)[ RSS](/packages/hike-hike/feed)WikiDiscussions master Synced 1mo ago

READMEChangelog (2)Dependencies (4)Versions (3)Used By (0)

Hike - Tokenize, navigate around and modify any text format
===========================================================

[](#hike---tokenize-navigate-around-and-modify-any-text-format)

A consistent interface to analyse and modify any text format. Programming languages, natural languages or any text format.

There are inbuilt (and extensible) navigation tools for moving between tokens and editing/inserting text into the tokeninzed object.

Includes a very minimalistic reference implementation for words/spaces which is shown below.

This is for demonstration purposes as Hike is intended as a foundation for building customised tokenizers with their own lexers.

1. Navigating
-------------

[](#1-navigating)

```
use Hike\Tokenizer\Word\Tokenizer;
use Hike\Tokenizer\Word\Token;
use Hike\TokenList\Tokens;
use Hike\TokenList\Navigator\Next;

$string = 'This is a string containing some words and spaces';

$tokens = Tokens::create(new Tokenizer($string));

echo $tokens->current()->value; // prints "This"

// Move to next word
$tokens = $tokens->navigate(new Next(Token::Word));

echo $tokens->current()->value; // prints "is"

// Move to next occurrenc of "some"
$tokens = $tokens->navigate(new Next('some'));

// Move to the next occurrent of either "words" or "spaces", whichever is first
$tokens = $tokens->navigate(
    new Next('words', 'spaces')
);
```

2. Modifying/outputting
-----------------------

[](#2-modifyingoutputting)

```
use Hike\Tokenizer\Word\Tokenizer;
use Hike\Tokenizer\Word\Token;
use Hike\TokenList\Tokens;
use Hike\TokenList\Navigator\Next;
use Hike\TokenList\Output\Serialize;
use Hike\TokenList\Modifier\Remove;

$string = 'This is a string containing some words and spaces';

$tokens = Tokens::create(new Tokenizer($string));

// Move to 'containing'
// remove the token
// Print the resulting tokens
// Prints 'This is a string some words and spaces'
echo $tokens = $tokens->navigate(new Next('containing'))
    ->modify(new Remove())
    ->output(new Serialize())
;
```

3. Selection
------------

[](#3-selection)

```
use Hike\Tokenizer\Word\Tokenizer;
use Hike\Tokenizer\Word\Token;
use Hike\TokenList\Tokens;
use Hike\TokenList\Navigator\Next;
use Hike\TokenList\Selection;

$string = 'This is a string containing some words and spaces';

$tokens = Tokens::create(new Tokenizer($string));

$tokens = $tokens->navigate(new Next('string'));

// Select everything from the current token to the next occurrence of 'words'
// and copy it
$selection = Selection::from($tokens)
    ->to(new Next('words'))
    ->copy()
;

// Prints 'string containing some words'
echo $selection->output(new Serialize());

// Select everything from the current token to the next occurrence of 'words'
// and delete it from the original token list
$selection = Selection::from($tokens)
    ->to(new Next('words'))
    ->delete()
;

// Prints 'This is a  and spaces'
echo $selection->output(new Serialize());
```

4. Custom tokenizer with configured lexers
------------------------------------------

[](#4-custom-tokenizer-with-configured-lexers)

Hike contains a `Words` tokenizer by default, below is an example implementing a more complex implementation that supports the following tokens:

1. Words
2. Spaces
3. Numbers
4. Punctuation

Firstly, define an enum of the possible tokens:

```
namespace My\Tokenizer;

enum Token
{
    case Word;
    case Space;
    case Number;
    case Punctuation;
}
```

Secondly define a tokenizer. There is an existing `CharacterList` lexer which matches one or more consecutive chracters

```
namespace My\Tokenizer;

use Hike\Tokenizer\Tokenizer as HikeTokenizer;
use Hike\Tokenizer\Lexer\Lexer\CharacterList;
use Hike\Tokenizer\Lexer\Tokenizer as LexerTokenizer;

final readonly class Tokenizer implements HikeTokenizer
{
    public function __construct(
        private string $string,
    ) {}

    public function getTokens(): array
    {
        // use inbuilt LexerTokenizer which iterates over the string and extracts tokens using lexers
        // n.b. LezerTokenizer is a Hike tokenizer implementation which tokenizes a string into characters
        $tokenizer = LexerTokenizer::fromString($this->string);

        $tokenizer = $tokenizer
            // If a continuous string of characters A-z are matched it will be represented by the token Word
            ->addLexer(new CharacterList(Token::Word, range('A', 'z'))
            // One or more spaces will be Token::Space
            ->addLexer(new CharacterList(Token::Space, [ ' ', "\t", "\n", "\r" ])
            // And additional character sets can be mapeed to other tokens
            ->addLexer(new CharacterList(Token::Number, range(0, 9))
            ->addLexer(new CharacterList(Token::Punctuation, ['.', ',', ':', '?', ';'])
        ;

        return $tokenizer->getTokens();
    }
}
```

Other Lexers can match Keywords or other substrings of characters.

Finally, use your configured tokenizer:

```
use My\Tokenizer\Tokenizer;
use My\Tokenizer\Token;
use Hike\TokenList\Tokens;
use Hike\TokenList\Navigator\Next;
use Hike\TokenList\Selection;

$string = 'Is it working? Who knows?';

$tokens = Tokens::create(new Tokenizer($string));

// Move to the first punctuation character
$tokens->navigate(new Next(Token::Punctuation));

// Returns Token::Punctuation
$type = $tokens->current()->name;

// Prints '?'
echo $tokens->current()->value;

$tokens = $tokens->navigate(new Next());

// Returns Token::Space
$type = $tokens->current()->name;

// Prints ' '
echo $tokens->current()->value;
```

###  Health Score

22

—

LowBetter than 22% of packages

Maintenance46

Moderate activity, may be stable

Popularity5

Limited adoption so far

Community7

Small or concentrated contributor base

Maturity28

Early-stage or recently created project

 Bus Factor1

Top contributor holds 100% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Every ~0 days

Total

2

Last Release

407d ago

### Community

Maintainers

![](https://www.gravatar.com/avatar/caa6bf0e9aa1359e0d477690cfdad5388d1cde567520c797797f48cc27f34fc2?d=identicon)[TomButler](/maintainers/TomButler)

---

Top Contributors

[![TRPB](https://avatars.githubusercontent.com/u/2454540?v=4)](https://github.com/TRPB "TRPB (6 commits)")

---

Tags

tokensparserlexertokenizerstatic anaylsis

###  Code Quality

TestsPHPUnit

Static AnalysisPHPStan

Code StylePHP CS Fixer

Type Coverage Yes

### Embed Badge

![Health badge](/badges/hike-hike/health.svg)

```
[![Health](https://phpackages.com/badges/hike-hike/health.svg)](https://phpackages.com/packages/hike-hike)
```

###  Alternatives

[doctrine/lexer

PHP Doctrine Lexer parser library that can be used in Top-Down, Recursive Descent Parsers.

11.2k910.8M118](/packages/doctrine-lexer)[cerbero/json-parser

Zero-dependencies pull parser to read large JSON from any source in a memory-efficient way.

803474.6k5](/packages/cerbero-json-parser)[creof/geo-parser

Parser for geography coordinate strings

624.4M15](/packages/creof-geo-parser)[creof/wkt-parser

Parser for well-known text (WKT) object strings

554.8M16](/packages/creof-wkt-parser)[nicoswd/php-rule-parser

Rule Engine - Rule Parser & Evaluator

13078.6k7](/packages/nicoswd-php-rule-parser)[tmilos/lexer

Lexical analyzer with individual token definition with regular expressions

211.7M2](/packages/tmilos-lexer)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)