PHPackages codewithkyrian/tokenizers - PHPackages - PHPackages [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register) 1. [Directory](/) 2. / 3. [Parsing & Serialization](/categories/parsing) 4. / 5. codewithkyrian/tokenizers ActiveLibrary[Parsing & Serialization](/categories/parsing) codewithkyrian/tokenizers ========================= Fast, pure-PHP tokenizer library compatible with Hugging Face tokenizers for encoding and decoding text 2.1.0(3mo ago)1813MITPHPPHP ^8.2CI passing Since Dec 10Pushed 3mo agoCompare [ Source](https://github.com/CodeWithKyrian/tokenizers-php)[ Packagist](https://packagist.org/packages/codewithkyrian/tokenizers)[ RSS](/packages/codewithkyrian-tokenizers/feed)WikiDiscussions main Synced 1mo ago READMEChangelog (3)Dependencies (6)Versions (8)Used By (0) Tokenizers PHP ============== [](#tokenizers-php) [![GitHub Workflow Status (main)](https://camo.githubusercontent.com/0c22fa469b041efd75886a25847e635bec4731be6cde989c919c05c3bb5235bd/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f616374696f6e732f776f726b666c6f772f7374617475732f636f6465776974686b797269616e2f746f6b656e697a6572732d7068702f74657374732e796d6c3f6272616e63683d6d61696e266c6162656c3d7465737473267374796c653d666c61742d737175617265)](https://github.com/codewithkyrian/tokenizers-php/actions) [![Total Downloads](https://camo.githubusercontent.com/3502edd50fac770056ab64ab0f475cc654b2165c09b3dccfd2ca18820ed894f3/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f64742f636f6465776974686b797269616e2f746f6b656e697a6572733f7374796c653d666c61742d737175617265)](https://packagist.org/packages/codewithkyrian/tokenizers) [![Latest Version](https://camo.githubusercontent.com/26be4497a06addf4e986d8ba2dc822f0d59ee5e3ea0740b71769cef67cfde9a0/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f762f636f6465776974686b797269616e2f746f6b656e697a6572733f7374796c653d666c61742d737175617265)](https://packagist.org/packages/codewithkyrian/tokenizers) [![License](https://camo.githubusercontent.com/bbb887f61df5b851490d828cd8ca984d4dd9dff7b58e25cc28266b2a174bdf07/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f6c6963656e73652f636f6465776974686b797269616e2f746f6b656e697a6572732d7068703f7374796c653d666c61742d737175617265)](https://packagist.org/packages/codewithkyrian/tokenizers) --- **Tokenizers PHP** is a lightweight, dependency-free PHP library for tokenizing text using the same tokenizers powering models on the Hugging Face Hub. Whether you're building LLM applications, search systems, or text processing pipelines, this library provides fast, accurate tokenization that matches the original model implementations. Highlights ---------- [](#highlights) - **Pure PHP** — No FFI, no external binaries, no compiled extensions. Works everywhere PHP runs. - **Hub Integration** — Load tokenizers from Hugging Face Hub with smart caching and manifest-based file checks. - **Flexible Loading** — Load from local files, config arrays, or build custom tokenizers with the builder API. - **Fully Tested** — Validated against BERT, GPT-2, Llama, Gemma, Qwen, RoBERTa, ALBERT, and more. - **Modern PHP** — Built for PHP 8.2+ with strict types, readonly classes, and clean interfaces. Installation ------------ [](#installation) Install via Composer: ``` composer require codewithkyrian/tokenizers ``` ### HTTP Client (for Hub loading) [](#http-client-for-hub-loading) Loading tokenizers from the Hugging Face Hub requires an HTTP client. We recommend Guzzle: ``` composer require guzzlehttp/guzzle ``` Quick Start ----------- [](#quick-start) ``` use Codewithkyrian\Tokenizers\Tokenizer; // Load a tokenizer from Hugging Face Hub $tokenizer = Tokenizer::fromHub('bert-base-uncased'); // Encode text to token IDs $encoding = $tokenizer->encode('Hello, how are you?'); echo implode(', ', $encoding->ids); // 101, 7592, 1010, 2129, 2024, 2017, 1029, 102 echo implode(', ', $encoding->tokens); // [CLS], hello, ,, how, are, you, ?, [SEP] // Decode token IDs back to text $text = $tokenizer->decode($encoding->ids); echo $text; // "[CLS] hello, how are you? [SEP]" ``` Loading Tokenizers ------------------ [](#loading-tokenizers) Tokenizers PHP provides multiple ways to load tokenizers depending on your use case. ### From Hugging Face Hub [](#from-hugging-face-hub) Load any tokenizer from the Hugging Face Hub by providing the model ID: ``` use Codewithkyrian\Tokenizers\Tokenizer; // Load a popular model $tokenizer = Tokenizer::fromHub('bert-base-uncased'); // Load a model from an organization $tokenizer = Tokenizer::fromHub('meta-llama/Llama-3.1-8B-Instruct'); // With options $tokenizer = Tokenizer::fromHub( modelId: 'openai/gpt-oss-20b', cacheDir: '/path/to/cache', // Custom cache directory revision: 'main', // Branch, tag, or commit hash token: 'hf_...' // Auth token for private models ); ``` #### Parameters [](#parameters) ParameterTypeDefaultDescription`modelId``string`—The model identifier on Hugging Face Hub (e.g., `bert-base-uncased` or `org/model-name`)`cacheDir``?string``null`Custom directory for caching downloaded files. Defaults to system cache directory`revision``?string``'main'`Specific version to load—can be a branch name, tag, or commit hash`token``?string``null`Hugging Face authentication token for accessing private or gated models#### Cache Directory Resolution [](#cache-directory-resolution) When `cacheDir` is not specified, the library automatically resolves the cache location: 1. **HF\_HUB\_CACHE** — if set, used directly 2. **HF\_HOME** — if set, `$HF_HOME/hub` 3. **macOS** — `~/Library/Caches/huggingface/hub` 4. **Linux** — `$XDG_CACHE_HOME/huggingface/hub` or `~/.cache/huggingface/hub` 5. **Windows** — `%LOCALAPPDATA%\huggingface\hub` Pass `cacheDir` to use a custom directory. ### From Local Files [](#from-local-files) Load tokenizers from local JSON files: ``` use Codewithkyrian\Tokenizers\Tokenizer; // Single file (tokenizer.json with all config merged) $tokenizer = Tokenizer::fromFile('/path/to/tokenizer.json'); // Multiple files (configs are merged, later files override earlier ones) $tokenizer = Tokenizer::fromFile( '/path/to/tokenizer.json', '/path/to/tokenizer_config.json' ); ``` This is useful when you've downloaded model files manually or are working in an offline environment. ### From Configuration Array [](#from-configuration-array) Build a tokenizer from a raw configuration array: ``` use Codewithkyrian\Tokenizers\Tokenizer; $config = json_decode(file_get_contents('tokenizer.json'), true); $tokenizer = Tokenizer::fromConfig($config); ``` ### Universal Loader [](#universal-loader) The `load()` method provides a convenient unified interface: ``` use Codewithkyrian\Tokenizers\Tokenizer; // Automatically detects the source type $tokenizer = Tokenizer::load('bert-base-uncased'); // From Hub $tokenizer = Tokenizer::load('/path/to/tokenizer.json'); // From file $tokenizer = Tokenizer::load($configArray); // From array ``` ### Accessing Configuration [](#accessing-configuration) The tokenizer stores its configuration and provides access via `getConfig()`: ``` $tokenizer = Tokenizer::fromHub('bert-base-uncased'); // Get a specific config value $maxLength = $tokenizer->getConfig('model_max_length'); // 512 $cleanup = $tokenizer->getConfig('clean_up_tokenization_spaces'); // true $custom = $tokenizer->getConfig('unknown_key', 'default'); // 'default' // Get all configuration (pass null or no arguments) $allConfig = $tokenizer->getConfig(); ``` Common configuration keys: - `model_max_length` — Maximum sequence length - `remove_space` — Whether to remove leading/trailing spaces - `do_lowercase_and_remove_accent` — Whether to lowercase and strip accents - `clean_up_tokenization_spaces` — Whether to clean up spaces during decoding Encoding Text ------------- [](#encoding-text) The `encode()` method tokenizes text and returns an `Encoding` object containing the token IDs, tokens, and type IDs. ``` $encoding = $tokenizer->encode('The quick brown fox jumps over the lazy dog.'); ``` ### The Encoding Object [](#the-encoding-object) ``` $encoding->ids; // int[] - Token IDs: [101, 1996, 4248, 2829, 4419, ...] $encoding->tokens; // string[] - Tokens: ['[CLS]', 'the', 'quick', 'brown', ...] $encoding->typeIds; // int[] - Segment IDs for sentence pairs: [0, 0, 0, ...] ``` ### Encoding Options [](#encoding-options) ``` $encoding = $tokenizer->encode( text: 'First sentence.', textPair: 'Second sentence.', // Optional second text for pair encoding addSpecialTokens: true // Whether to add [CLS], [SEP], etc. (default: true) ); ``` #### Parameters [](#parameters-1) ParameterTypeDefaultDescription`text``string`—The primary text to tokenize`textPair``?string``null`Optional second text for sequence pair tasks (e.g., question-answering)`addSpecialTokens``bool``true`Whether to add model-specific special tokens (like `[CLS]`, `[SEP]`)### Sentence Pairs [](#sentence-pairs) For tasks involving two text sequences (like question-answering or natural language inference), pass both texts: ``` $encoding = $tokenizer->encode( text: 'What is the capital of France?', textPair: 'Paris is the capital of France.' ); // tokens: ['[CLS]', 'what', 'is', 'the', 'capital', 'of', 'france', '?', '[SEP]', // 'paris', 'is', 'the', 'capital', 'of', 'france', '.', '[SEP]'] // typeIds: [0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1] ``` The `typeIds` distinguish between the first sequence (0) and the second sequence (1), which many models use during attention computation. Decoding Tokens --------------- [](#decoding-tokens) Convert token IDs back to human-readable text: ``` $text = $tokenizer->decode([101, 7592, 1010, 2129, 2024, 2017, 1029, 102]); // "hello, how are you?" ``` ### Decoding Options [](#decoding-options) ``` $text = $tokenizer->decode( ids: $encoding->ids, skipSpecialTokens: true, // Remove [CLS], [SEP], etc. (default: true) cleanup: null // Override cleanup behavior (default: use model config) ); ``` #### Parameters [](#parameters-2) ParameterTypeDefaultDescription`ids``int[]`—Array of token IDs to decode`skipSpecialTokens``bool``true`Whether to exclude special tokens from the output`cleanup``?bool``null`Whether to clean up tokenization artifacts (extra spaces). Uses model's config when `null`### Cleanup Behavior [](#cleanup-behavior) The `cleanup` parameter controls whether tokenization artifacts are cleaned: ``` // With cleanup (default when model config says so) $tokenizer->decode($ids, cleanup: true); // "hello, how are you?" // Without cleanup $tokenizer->decode($ids, cleanup: false); // "hello , how are you ?" ``` When `cleanup` is `null`, the library respects the `clean_up_tokenization_spaces` setting from the model's configuration. Custom Tokenizers with the Builder ---------------------------------- [](#custom-tokenizers-with-the-builder) For advanced use cases, build tokenizers from scratch using the fluent builder API: ``` use Codewithkyrian\Tokenizers\Tokenizer; use Codewithkyrian\Tokenizers\Models\WordPieceModel; use Codewithkyrian\Tokenizers\Normalizers\LowercaseNormalizer; use Codewithkyrian\Tokenizers\PreTokenizers\WhitespacePreTokenizer; use Codewithkyrian\Tokenizers\PostProcessors\BertPostProcessor; use Codewithkyrian\Tokenizers\Decoders\WordPieceDecoder; $vocab = ['[UNK]' => 0, '[CLS]' => 1, '[SEP]' => 2, 'hello' => 3, 'world' => 4, ...]; $tokenizer = Tokenizer::builder() ->withModel(new WordPieceModel($vocab, '[UNK]')) ->withNormalizer(new LowercaseNormalizer()) ->withPreTokenizer(new WhitespacePreTokenizer()) ->withPostProcessor(new BertPostProcessor('[CLS]', '[SEP]')) ->withDecoder(new WordPieceDecoder()) ->withSpecialTokens(['[UNK]', '[CLS]', '[SEP]', '[PAD]', '[MASK]']) ->withConfig('model_max_length', 512) ->withConfig('clean_up_tokenization_spaces', true) ->build(); ``` ### Builder Methods [](#builder-methods) MethodDescription`withModel(ModelInterface $model)`**Required.** Set the tokenization model (BPE, WordPiece, Unigram)`withNormalizer(NormalizerInterface $normalizer)`Set text normalizer. Defaults to `PassThroughNormalizer``withPreTokenizer(PreTokenizerInterface $preTokenizer)`Set pre-tokenizer. Defaults to `IdentityPreTokenizer``withPostProcessor(PostProcessorInterface $postProcessor)`Set post-processor. Defaults to `DefaultPostProcessor``withDecoder(DecoderInterface $decoder)`Set decoder. Defaults to `FuseDecoder``withAddedTokens(array $tokens)`Add extra tokens to the vocabulary`withSpecialTokens(array $tokens)`Define special tokens (skipped during decode by default)`withConfig(string $key, mixed $value)`Set a configuration value (see common keys below)`build()`Build and return the `Tokenizer` instanceCommon config keys for `withConfig()`: - `'model_max_length'` — Maximum sequence length - `'remove_space'` — Remove leading/trailing spaces before normalization - `'do_lowercase_and_remove_accent'` — Lowercase and strip accents - `'clean_up_tokenization_spaces'` — Clean up spaces during decoding The Tokenization Pipeline ------------------------- [](#the-tokenization-pipeline) Understanding the tokenization pipeline helps when debugging or customizing behavior. Each input text passes through these stages: ``` ┌─────────────────────────────────────────────────────────────────────┐ │ Input Text │ │ "Hello, how are you doing?" │ └──────────────────────────┬──────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────┐ │ 1. Normalization │ │ • Unicode normalization (NFC, NFKC, NFD, NFKD) │ │ • Lowercase transformation │ │ • Accent stripping │ │ • Control character removal │ │ │ │ → "hello, how are you doing?" │ └──────────────────────────┬──────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────┐ │ 2. Pre-tokenization │ │ • Split on whitespace and/or punctuation │ │ • Identify word boundaries │ │ │ │ → ["hello", ",", "how", "are", "you", "doing", "?"] │ └──────────────────────────┬──────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────┐ │ 3. Model Tokenization │ │ • BPE: Byte-Pair Encoding merges │ │ • WordPiece: Greedy longest-match-first │ │ • Unigram: Probabilistic subword selection │ │ │ │ → ["hello", ",", "how", "are", "you", "do", "##ing", │ │ "?"] │ └──────────────────────────┬──────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────┐ │ 4. Post-processing │ │ • Add special tokens ([CLS], [SEP], , , etc.) │ │ • Generate token type IDs for sentence pairs │ │ │ │ → ["[CLS]", "hello", ",", "how", "are", "you", "do", │ │ "##ing", "?", "[SEP]"] │ └──────────────────────────┬──────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────┐ │ 5. ID Mapping │ │ • Convert tokens to numerical IDs using vocabulary │ │ │ │ → [101, 7592, 1010, 2129, 2024, 2017, 2079, 2075, │ │ 1029, 102] │ └─────────────────────────────────────────────────────────────────────┘ ``` Components Reference -------------------- [](#components-reference) ### Normalizers [](#normalizers) Normalizers clean and standardize input text before tokenization. NormalizerDescription`BertNormalizer`BERT-style: clean text, handle Chinese chars, lowercase, strip accents`LowercaseNormalizer`Convert all characters to lowercase`NFCNormalizer`Unicode NFC normalization`NFKCNormalizer`Unicode NFKC normalization`NFKDNormalizer`Unicode NFKD normalization`StripNormalizer`Strip leading/trailing whitespace`StripAccentsNormalizer`Remove accent marks from characters`ReplaceNormalizer`Replace patterns or strings`PrependNormalizer`Prepend a string to the input`PrecompiledNormalizer`Use precompiled normalization rules (for SentencePiece models)`NormalizerSequence`Chain multiple normalizers together`PassThroughNormalizer`No-op, passes text through unchanged### Pre-tokenizers [](#pre-tokenizers) Pre-tokenizers split text into smaller chunks before subword tokenization. Pre-tokenizerDescription`BertPreTokenizer`Split on whitespace and punctuation (BERT-style)`ByteLevelPreTokenizer`Convert to byte-level representation (GPT-2 style)`WhitespacePreTokenizer`Split on whitespace characters`WhitespaceSplit`Split only on whitespace, keep punctuation attached`MetaspacePreTokenizer`Replace spaces with ▁ (SentencePiece style)`PunctuationPreTokenizer`Split on punctuation characters`DigitsPreTokenizer`Isolate digit sequences`SplitPreTokenizer`Split using custom regex patterns`PreTokenizerSequence`Chain multiple pre-tokenizers together`IdentityPreTokenizer`No-op, returns text unchanged### Models [](#models) Models perform the core subword tokenization algorithm. ModelDescription`BPEModel`Byte-Pair Encoding - iteratively merges most frequent pairs`WordPieceModel`Greedy longest-match-first subword tokenization (BERT)`UnigramModel`Probabilistic subword selection (SentencePiece)`FallbackModel`Simple vocabulary lookup with unknown token fallback### Post-processors [](#post-processors) Post-processors add special tokens and structure to the tokenized output. Post-processorDescription`BertPostProcessor`Add `[CLS]` and `[SEP]` tokens`RobertaPostProcessor`Add `` and `` tokens with spacing`TemplatePostProcessor`Flexible template-based token insertion`ByteLevelPostProcessor`Handle byte-level special tokens`PostProcessorSequence`Chain multiple post-processors`DefaultPostProcessor`Minimal processing, no tokens added### Decoders [](#decoders) Decoders convert tokens back to readable text. DecoderDescription`ByteLevelDecoder`Decode byte-level tokens back to UTF-8`WordPieceDecoder`Handle `##` continuation prefixes`MetaspaceDecoder`Convert `▁` back to spaces`BPEDecoder`Handle BPE-specific suffixes and spaces`CTCDecoder`Decode CTC (Connectionist Temporal Classification) output`FuseDecoder`Simply join tokens with optional separator`ReplaceDecoder`Replace specific patterns during decode`StripDecoder`Strip specific characters`ByteFallbackDecoder`Handle byte fallback tokens (e.g., ``)`DecoderSequence`Chain multiple decoders togetherExtending the Library --------------------- [](#extending-the-library) All components implement simple interfaces that you can extend: ``` use Codewithkyrian\Tokenizers\Contracts\NormalizerInterface; class CustomNormalizer implements NormalizerInterface { public function normalize(string $text): string { // Your custom normalization logic return $modifiedText; } } ``` Available interfaces: - `NormalizerInterface` — Text normalization - `PreTokenizerInterface` — Pre-tokenization splitting - `ModelInterface` — Core tokenization algorithm - `PostProcessorInterface` — Post-processing and special tokens - `DecoderInterface` — Token-to-text conversion Contributing ------------ [](#contributing) Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change. ``` # Clone the repository git clone https://github.com/codewithkyrian/tokenizers-php.git cd tokenizers-php # Install dependencies composer install # Run tests vendor/bin/pest ``` License ------- [](#license) This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. Credits ------- [](#credits) - [Kyrian Obikwelu](https://github.com/CodeWithKyrian) — Creator and maintainer - [Hugging Face](https://huggingface.co) — Tokenizers specification and model hosting - All [contributors](https://github.com/codewithkyrian/tokenizers-php/contributors) --- Made with ❤️ for the PHP community ### Health Score 41 — FairBetter than 89% of packages Maintenance81 Actively maintained with recent releases Popularity14 Limited adoption so far Community6 Small or concentrated contributor base Maturity52 Maturing project, gaining track record Bus Factor1 Top contributor holds 100% of commits — single point of failure How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window. **Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores. **Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement. **Maturity (30%)** — Project age, version count, PHP version support, and release stability. ### Release Activity Cadence Every ~28 days Total 3 Last Release 101d ago Major Versions 1.0.0 → 2.0.02026-02-04 PHP version history (2 changes)1.0.0PHP ^8.1 2.0.0PHP ^8.2 ### Community Maintainers ![](https://www.gravatar.com/avatar/146c2eb02350558d165083e0018f5377f15f0e71493439c93ee60e7c7ac97fc1?d=identicon)[CodeWithKyrian](/maintainers/CodeWithKyrian) --- Top Contributors [![CodeWithKyrian](https://avatars.githubusercontent.com/u/48791154?v=4)](https://github.com/CodeWithKyrian "CodeWithKyrian (10 commits)") --- Tags phptokenizationtokenizersencodingtokenizernlpmachine learningllmgpttext-processingnatural language processingbpetransformershuggingfacetokenizationBertwordpiecesentencepiece ### Code Quality TestsPest Static AnalysisPHPStan Code StylePHP CS Fixer Type Coverage Yes ### Embed Badge ![Health badge](/badges/codewithkyrian-tokenizers/health.svg) ``` [![Health](https://phpackages.com/badges/codewithkyrian-tokenizers/health.svg)](https://phpackages.com/packages/codewithkyrian-tokenizers) ``` ### Alternatives [codewithkyrian/transformers State-of-the-art Machine Learning for PHP. Run Transformers in PHP 749231.8k5](/packages/codewithkyrian-transformers)[deepseek-php/deepseek-php-client deepseek PHP client is a robust and community-driven PHP client library for seamless integration with the Deepseek API, offering efficient access to advanced AI and data processing capabilities. 47073.9k5](/packages/deepseek-php-deepseek-php-client)[yethee/tiktoken PHP version of tiktoken 1583.1M15](/packages/yethee-tiktoken)[grok-php/laravel Seamlessly integrate Grok AI into Laravel applications with an elegant, developer-friendly package. Leverage powerful AI models for chat, automation, and NLP while maintaining Laravel's expressive simplicity. 1633.4k](/packages/grok-php-laravel)[helgesverre/toon Token-Oriented Object Notation - A compact data format for reducing token consumption when sending structured data to LLMs 11841.4k9](/packages/helgesverre-toon)[sbsaga/toon 🧠 TOON for Laravel — a compact, human-readable, and token-efficient data format for AI prompts & LLM contexts. Perfect for ChatGPT, Gemini, Claude, Mistral, and OpenAI integrations (JSON ⇄ TOON). 6115.6k](/packages/sbsaga-toon) PHPackages © 2026 [Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)