PHPackages                             danny50610/bpe-tokeniser - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Parsing & Serialization](/categories/parsing)
4. /
5. danny50610/bpe-tokeniser

ActiveLibrary[Parsing & Serialization](/categories/parsing)

danny50610/bpe-tokeniser
========================

PHP port for openai/tiktoken (most)

0.3.0(9mo ago)10422.0k↓31.5%1MITPHPPHP ^8.1CI passing

Since Aug 21Pushed 6mo ago1 watchersCompare

[ Source](https://github.com/danny50610/bpe-tokeniser)[ Packagist](https://packagist.org/packages/danny50610/bpe-tokeniser)[ RSS](/packages/danny50610-bpe-tokeniser/feed)WikiDiscussions main Synced 1mo ago

READMEChangelog (3)Dependencies (1)Versions (4)Used By (1)

bpe-tokeniser
=============

[](#bpe-tokeniser)

[![PHP Test](https://github.com/danny50610/bpe-tokeniser/actions/workflows/php.yml/badge.svg)](https://github.com/danny50610/bpe-tokeniser/actions)[![codecov](https://camo.githubusercontent.com/7a26d8721ecc6e146ea408b4e1676affd0f8135a5dfc4c9db9e7d796e0370d02/68747470733a2f2f636f6465636f762e696f2f67682f64616e6e7935303631302f6270652d746f6b656e697365722f67726170682f62616467652e7376673f746f6b656e3d43474f52525131503657)](https://codecov.io/gh/danny50610/bpe-tokeniser)[![Latest Stable Version](https://camo.githubusercontent.com/0d48908cf608f7ee36444ecd6a8f57fd373aa125bb041aa7d7fb424d32419a14/68747470733a2f2f706f7365722e707567782e6f72672f64616e6e7935303631302f6270652d746f6b656e697365722f76)](https://packagist.org/packages/danny50610/bpe-tokeniser)[![Total Downloads](https://camo.githubusercontent.com/c2927f24cc9465fe9df027f551ff97218f032958d733fbad37ade320d7108ba8/68747470733a2f2f706f7365722e707567782e6f72672f64616e6e7935303631302f6270652d746f6b656e697365722f646f776e6c6f616473)](https://packagist.org/packages/danny50610/bpe-tokeniser)[![License](https://camo.githubusercontent.com/c39dab201ca515f291ae9602b67dc1d40caa3f3782f7317a1b13b0b7b371d19e/68747470733a2f2f706f7365722e707567782e6f72672f64616e6e7935303631302f6270652d746f6b656e697365722f6c6963656e7365)](https://packagist.org/packages/danny50610/bpe-tokeniser)

PHP port for [openai/tiktoken](https://github.com/openai/tiktoken) (most)

Supported encodings
-------------------

[](#supported-encodings)

- gpt-3.5-turbo
- gpt-4
- gpt-4o
- gpt-oss
- gpt-5
- more ...

For available encodings, see `src/EncodingFactory.php`

Installation
------------

[](#installation)

```
composer require danny50610/bpe-tokeniser
```

Example
-------

[](#example)

### GPT-5 (o200k\_base)

[](#gpt-5-o200k_base)

```
use Danny50610\BpeTokeniser\EncodingFactory;

$enc = EncodingFactory::createByModelName('gpt-5-2025-08-07');

var_dump($enc->encode("hello world"));
/**
 * output:
 * array(2) {
 *  [0]=>
 *  int(24912)
 *  [1]=>
 *  int(2375)
 * }
 */

var_dump($enc->decode($enc->encode("hello world")));
// output: string(11) "hello world"
```

### GPT-4 / GPT-3.5-Turbo (cl100k\_base)

[](#gpt-4--gpt-35-turbo-cl100k_base)

```
use Danny50610\BpeTokeniser\EncodingFactory;

$enc = EncodingFactory::createByEncodingName('cl100k_base');

var_dump($enc->encode("hello world"));
/**
 * output:
 * array(2) {
 *  [0]=>
 *  int(15339)
 *  [1]=>
 *  int(1917)
 * }
 */

var_dump($enc->decode($enc->encode("hello world")));
// output: string(11) "hello world"
```

```
use Danny50610\BpeTokeniser\EncodingFactory;

$enc = EncodingFactory::createByModelName('gpt-3.5-turbo');

var_dump($enc->decode($enc->encode("hello world")));
// output: string(11) "hello world"
```

###  Health Score

43

—

FairBetter than 91% of packages

Maintenance62

Regular maintenance activity

Popularity42

Moderate usage in the ecosystem

Community9

Small or concentrated contributor base

Maturity44

Maturing project, gaining track record

 Bus Factor1

Top contributor holds 100% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Every ~359 days

Total

3

Last Release

283d ago

### Community

Maintainers

![](https://www.gravatar.com/avatar/56a0b13f5c10263516e4e96bdb4743e5e10fa2a9e5e3977dd23222fcb1de63ad?d=identicon)[danny50610](/maintainers/danny50610)

---

Top Contributors

[![danny50610](https://avatars.githubusercontent.com/u/3949245?v=4)](https://github.com/danny50610 "danny50610 (63 commits)")

---

Tags

cl100kbasegpt-4gpt-5gpt-osstiktokentokenizertokenizertiktokengpt-4gpt-3.5cl100k\_base

###  Code Quality

TestsPHPUnit

### Embed Badge

![Health badge](/badges/danny50610-bpe-tokeniser/health.svg)

```
[![Health](https://phpackages.com/badges/danny50610-bpe-tokeniser/health.svg)](https://phpackages.com/packages/danny50610-bpe-tokeniser)
```

###  Alternatives

[yethee/tiktoken

PHP version of tiktoken

1583.1M15](/packages/yethee-tiktoken)[rajentrivedi/tokenizer-x

TokenizerX calculates required tokens for given prompt

91214.0k3](/packages/rajentrivedi-tokenizer-x)[gioni06/gpt3-tokenizer

PHP package for Byte Pair Encoding (BPE) used by GPT-3.

85537.5k8](/packages/gioni06-gpt3-tokenizer)[nicoswd/php-rule-parser

Rule Engine - Rule Parser & Evaluator

13078.6k7](/packages/nicoswd-php-rule-parser)[jeremeamia/functionparser

Function parser for PHP functions, methods, and closures

48169.7k6](/packages/jeremeamia-functionparser)[hybridlogic/classifier

A Naive Bayesian classification library for PHP with support for different tokenizers to optimize string classification.

801.4k](/packages/hybridlogic-classifier)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)