PHPackages                             aisk/mistral-tokenizer - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Utility & Helpers](/categories/utility)
4. /
5. aisk/mistral-tokenizer

ActiveLibrary[Utility & Helpers](/categories/utility)

aisk/mistral-tokenizer
======================

PHP implementation of Mistral's tokenizers

0.2.0(1y ago)111.9k—1%[1 PRs](https://github.com/GetAisk/mistral-php-tokenizer/pulls)MITPHPPHP ^8.0

Since Apr 22Pushed 1y agoCompare

[ Source](https://github.com/GetAisk/mistral-php-tokenizer)[ Packagist](https://packagist.org/packages/aisk/mistral-tokenizer)[ RSS](/packages/aisk-mistral-tokenizer/feed)WikiDiscussions main Synced 1mo ago

READMEChangelog (2)Dependencies (1)Versions (5)Used By (0)

Mistral PHP Tokenizer
=====================

[](#mistral-php-tokenizer)

A PHP implementation of the Mistral tokenizers, focusing on the Tekken tokenizer.

Developed by [Aisk](https://getaisk.com).

Installation
------------

[](#installation)

```
composer require aisk/mistral-tokenizer
```

Features
--------

[](#features)

- Text tokenization and detokenization
- Token counting
- Batch processing
- Special token handling
- Unicode character support

Usage
-----

[](#usage)

### Basic Usage

[](#basic-usage)

```