PHPackages                             danny50610/bpe-tokeniser - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Parsing & Serialization](/categories/parsing)
4. /
5. danny50610/bpe-tokeniser

ActiveLibrary[Parsing & Serialization](/categories/parsing)

danny50610/bpe-tokeniser
========================

PHP port for openai/tiktoken (most)

0.3.0(10mo ago)10457.3k↓54.9%2MITPHPPHP ^8.1CI passing

Since Aug 21Pushed 8mo ago1 watchersCompare

[ Source](https://github.com/danny50610/bpe-tokeniser)[ Packagist](https://packagist.org/packages/danny50610/bpe-tokeniser)[ RSS](/packages/danny50610-bpe-tokeniser/feed)WikiDiscussions main Synced 2d ago

READMEChangelog (3)Dependencies (1)Versions (4)Used By (2)

bpe-tokeniser
=============

[](#bpe-tokeniser)

[![PHP Test](https://github.com/danny50610/bpe-tokeniser/actions/workflows/php.yml/badge.svg)](https://github.com/danny50610/bpe-tokeniser/actions)[![codecov](https://camo.githubusercontent.com/7a26d8721ecc6e146ea408b4e1676affd0f8135a5dfc4c9db9e7d796e0370d02/68747470733a2f2f636f6465636f762e696f2f67682f64616e6e7935303631302f6270652d746f6b656e697365722f67726170682f62616467652e7376673f746f6b656e3d43474f52525131503657)](https://codecov.io/gh/danny50610/bpe-tokeniser)[![Latest Stable Version](https://camo.githubusercontent.com/0d48908cf608f7ee36444ecd6a8f57fd373aa125bb041aa7d7fb424d32419a14/68747470733a2f2f706f7365722e707567782e6f72672f64616e6e7935303631302f6270652d746f6b656e697365722f76)](https://packagist.org/packages/danny50610/bpe-tokeniser)[![Total Downloads](https://camo.githubusercontent.com/c2927f24cc9465fe9df027f551ff97218f032958d733fbad37ade320d7108ba8/68747470733a2f2f706f7365722e707567782e6f72672f64616e6e7935303631302f6270652d746f6b656e697365722f646f776e6c6f616473)](https://packagist.org/packages/danny50610/bpe-tokeniser)[![License](https://camo.githubusercontent.com/c39dab201ca515f291ae9602b67dc1d40caa3f3782f7317a1b13b0b7b371d19e/68747470733a2f2f706f7365722e707567782e6f72672f64616e6e7935303631302f6270652d746f6b656e697365722f6c6963656e7365)](https://packagist.org/packages/danny50610/bpe-tokeniser)

PHP port for [openai/tiktoken](https://github.com/openai/tiktoken) (most)

Supported encodings
-------------------

[](#supported-encodings)

- gpt-3.5-turbo
- gpt-4
- gpt-4o
- gpt-oss
- gpt-5
- more ...

For available encodings, see `src/EncodingFactory.php`

Installation
------------

[](#installation)

```
composer require danny50610/bpe-tokeniser
```

Example
-------

[](#example)

### GPT-5 (o200k\_base)

[](#gpt-5-o200k_base)

```
use Danny50610\BpeTokeniser\EncodingFactory;

$enc = EncodingFactory::createByModelName('gpt-5-2025-08-07');

var_dump($enc->encode("hello world"));
/**
 * output:
 * array(2) {
 *  [0]=>
 *  int(24912)
 *  [1]=>
 *  int(2375)
 * }
 */

var_dump($enc->decode($enc->encode("hello world")));
// output: string(11) "hello world"
```

### GPT-4 / GPT-3.5-Turbo (cl100k\_base)

[](#gpt-4--gpt-35-turbo-cl100k_base)

```
use Danny50610\BpeTokeniser\EncodingFactory;

$enc = EncodingFactory::createByEncodingName('cl100k_base');

var_dump($enc->encode("hello world"));
/**
 * output:
 * array(2) {
 *  [0]=>
 *  int(15339)
 *  [1]=>
 *  int(1917)
 * }
 */

var_dump($enc->decode($enc->encode("hello world")));
// output: string(11) "hello world"
```

```
use Danny50610\BpeTokeniser\EncodingFactory;

$enc = EncodingFactory::createByModelName('gpt-3.5-turbo');

var_dump($enc->decode($enc->encode("hello world")));
// output: string(11) "hello world"
```

###  Health Score

42

—

FairBetter than 88% of packages

Maintenance57

Moderate activity, may be stable

Popularity42

Moderate usage in the ecosystem

Community11

Small or concentrated contributor base

Maturity45

Maturing project, gaining track record

 Bus Factor1

Top contributor holds 100% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Every ~359 days

Total

3

Last Release

329d ago

### Community

Maintainers

![](https://www.gravatar.com/avatar/56a0b13f5c10263516e4e96bdb4743e5e10fa2a9e5e3977dd23222fcb1de63ad?d=identicon)[danny50610](/maintainers/danny50610)

---

Top Contributors

[![danny50610](https://avatars.githubusercontent.com/u/3949245?v=4)](https://github.com/danny50610 "danny50610 (63 commits)")

---

Tags

cl100kbasegpt-4gpt-5gpt-osstiktokentokenizertokenizertiktokengpt-4gpt-3.5cl100k\_base

###  Code Quality

TestsPHPUnit

### Embed Badge

![Health badge](/badges/danny50610-bpe-tokeniser/health.svg)

```
[![Health](https://phpackages.com/badges/danny50610-bpe-tokeniser/health.svg)](https://phpackages.com/packages/danny50610-bpe-tokeniser)
```

###  Alternatives

[yethee/tiktoken

PHP version of tiktoken

1624.0M30](/packages/yethee-tiktoken)[rajentrivedi/tokenizer-x

TokenizerX calculates required tokens for given prompt

92252.4k3](/packages/rajentrivedi-tokenizer-x)[gioni06/gpt3-tokenizer

PHP package for Byte Pair Encoding (BPE) used by GPT-3.

84591.1k11](/packages/gioni06-gpt3-tokenizer)[nicoswd/php-rule-parser

Rule Engine - Rule Parser & Evaluator

13181.4k7](/packages/nicoswd-php-rule-parser)[jeremeamia/functionparser

Function parser for PHP functions, methods, and closures

48174.6k6](/packages/jeremeamia-functionparser)[hybridlogic/classifier

A Naive Bayesian classification library for PHP with support for different tokenizers to optimize string classification.

801.4k](/packages/hybridlogic-classifier)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)