PHPackages juniora/tokenizer-x - PHPackages - PHPackages [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register) 1. [Directory](/) 2. / 3. [Parsing & Serialization](/categories/parsing) 4. / 5. juniora/tokenizer-x ActiveLibrary[Parsing & Serialization](/categories/parsing) juniora/tokenizer-x =================== TokenizerX calculates required tokens for given prompt v1.0.3(3mo ago)01MITPHPPHP ^8.1 Since Apr 11Pushed 3mo agoCompare [ Source](https://github.com/JunioraTeam/tokenizer-x)[ Packagist](https://packagist.org/packages/juniora/tokenizer-x)[ Docs](https://github.com/rajentrivedi/tokenizer-x)[ RSS](/packages/juniora-tokenizer-x/feed)WikiDiscussions main Synced 3w ago READMEChangelog (1)Dependencies (15)Versions (2)Used By (0) Calculates the OpenAI tokens for the given prompt ================================================= [](#calculates-the-openai-tokens-for-the-given-prompt) [![Latest Version on Packagist](https://camo.githubusercontent.com/e5cfde66635be04deaae49d2faa3c47cc9914394eea6a753eafc6c4861cd6978/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f762f72616a656e747269766564692f746f6b656e697a65722d782e7376673f7374796c653d666c61742d737175617265)](https://packagist.org/packages/rajentrivedi/tokenizer-x) [![GitHub Code Style Action Status](https://camo.githubusercontent.com/0ba8471f6d7848fe85d36a90a990836de8fac10723cce2622c9ef2ef121c9527/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f616374696f6e732f776f726b666c6f772f7374617475732f72616a656e747269766564692f746f6b656e697a65722d782f6669782d7068702d636f64652d7374796c652d6973737565732e796d6c3f6272616e63683d6d61696e266c6162656c3d636f64652532307374796c65267374796c653d666c61742d737175617265)](https://github.com/rajentrivedi/tokenizer-x/actions?query=workflow%3A%22Fix+PHP+code+style+issues%22+branch%3Amain)[![Total Downloads](https://camo.githubusercontent.com/e3afd59c0f3d99a5d9ef8e9b9e5a83363d4cbca60271f8fb9d118072eaf6f686/68747470733a2f2f696d672e736869656c64732e696f2f7061636b61676973742f64742f72616a656e747269766564692f746f6b656e697a65722d782e7376673f7374796c653d666c61742d737175617265)](https://packagist.org/packages/rajentrivedi/tokenizer-x) TokenizerX supports Laravel 10, 11 & 12. ### Installation [](#installation) ``` composer require rajentrivedi/tokenizer-x ``` TokenizerX ---------- [](#tokenizerx) TokenizerX is a Laravel package designed to streamline tokenization processes in your applications. With the latest update, TokenizerX now supports cutting-edge GPT-4 models, providing advanced natural language processing capabilities. It calculates the tokens required for a given prompt before requesting the OpenAI REST API. This package helps to ensure that the user does not exceed the OpenAI API token limit and can generate accurate responses. To access the OpenAI Rest API, you may consider the beautiful Laravel Package [OpenAI PHP](https://github.com/openai-php/client). Supported OpenAI Models ----------------------- [](#supported-openai-models) - o1 - o3 - o4-mini - gpt-5 - gpt-4 - gpt-4.1 - gpt-4.5 - gpt-4o - gpt-3.5-turbo - gpt-3.5 - davinci-002 - babbage-002 - text-embedding-ada-002 - text-embedding-3-small - text-embedding-3-large - text-davinci-003 - text-davinci-002 - text-davinci-001 - text-curie-001 - text-babbage-001 - text-ada-001 - davinci - curie - babbage - ada - code-davinci-002 - code-davinci-001 - code-cushman-002 - code-cushman-001 - davinci-codex - cushman-codex - text-davinci-edit-001 - code-davinci-edit-001 - text-similarity-davinci-001 - text-similarity-curie-001 - text-similarity-babbage-001 - text-similarity-ada-001 - text-search-davinci-doc-001 - text-search-curie-doc-001 - text-search-babbage-doc-001 - text-search-ada-doc-001 - code-search-babbage-code-001 - code-search-ada-code-001 Supported Encoding ------------------ [](#supported-encoding) - r50k\_base - p50k\_base - p50k\_edit - cl100k\_base - o200k\_base Installation ------------ [](#installation-1) You can install the package via composer: ``` composer require rajentrivedi/tokenizer-x ``` Usage ----- [](#usage) By default package will consider GPT-3 model ``` use Rajentrivedi\TokenizerX\TokenizerX; TokenizerX::count("how are you?"); ``` If you want token counts for specific OpenAI model, you can pass model as a second argument from above given supported model list. ``` use Rajentrivedi\TokenizerX\TokenizerX; TokenizerX::count("how are you?", "gpt-4"); ``` You can also read the text from file ``` TokenizerX::count(file_get_contents('path_to_file')); ``` Please make sure that text of the file don't change while reading the file programmatically, this may happen due to encoding. You can check the generated token IDs by using following ``` TokenizerX::tokens(file_get_contents('path_to_file')); ``` This will return an array of tokens generated & compare those token IDs with [OpenAI Tokenizer](https://platform.openai.com/tokenizer) You can also use the OpenAI Tokenizer to double-check package generated token counts. Support ------- [](#support) If you find TokenizerX helpful and would like to support its ongoing development, you can contribute by buying me a coffee! Your support helps in maintaining and improving the package for the Laravel community. [![ko-fi](https://camo.githubusercontent.com/201ef269611db7eb6b5d08e9f756ab8980df3014b64492770bdf13a6ed924641/68747470733a2f2f6b6f2d66692e636f6d2f696d672f676974687562627574746f6e5f736d2e737667)](https://ko-fi.com/T6T4T0NM4) Testing ------- [](#testing) ``` composer test ``` Changelog --------- [](#changelog) Please see [CHANGELOG](CHANGELOG.md) for more information on what has changed recently. Contributing ------------ [](#contributing) Please see [CONTRIBUTING](CONTRIBUTING.md) for details. Security Vulnerabilities ------------------------ [](#security-vulnerabilities) Please review [our security policy](../../security/policy) on how to report security vulnerabilities. ⭐ Star the Repository ⭐ ----------------------- [](#-star-the-repository-) If you find this project useful or interesting, I kindly request you to give it a ⭐ star on GitHub. Your support will encourage and motivate me to continue improving and maintaining this project. By starring the repository, you can show appreciation for the work put into developing this open-source project. It also helps to increase its visibility, making it more accessible to other developers and potentially attracting contributors. To give a ⭐ star, simply click on the **Star** button at the top-right corner of the repository page. Credits ------- [](#credits) - [Rajen Trivedi](https://github.com/rajentrivedi) - [All Contributors](../../contributors) License ------- [](#license) TokenizerX is developed using ============================= [](#tokenizerx-is-developed-using) - [GPT-3 Encoder PHP](https://github.com/CodeRevolutionPlugins/GPT-3-Encoder-PHP) - [Tiktoken-php](https://github.com/yethee/tiktoken-php) - [Laravel Package Skeleton by Spatie](https://github.com/spatie/package-skeleton-laravel) The MIT License (MIT). Please see [License File](LICENSE.md) for more information. ### Health Score 35 — LowBetter than 77% of packages Maintenance80 Actively maintained with recent releases Popularity1 Limited adoption so far Community12 Small or concentrated contributor base Maturity43 Maturing project, gaining track record Bus Factor1 Top contributor holds 61.9% of commits — single point of failure How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window. **Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores. **Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement. **Maturity (30%)** — Project age, version count, PHP version support, and release stability. ### Release Activity Cadence Unknown Total 1 Last Release 104d ago ### Community Maintainers ![](https://www.gravatar.com/avatar/b9c0c38d50cc76c71c5d48021e1358a1ea0f4f12e60d74113a5f58687cfecab8?d=identicon)[ntrolly79](/maintainers/ntrolly79) --- Top Contributors [![rajentrivedi](https://avatars.githubusercontent.com/u/69707769?v=4)](https://github.com/rajentrivedi "rajentrivedi (39 commits)")[![rajenindianic](https://avatars.githubusercontent.com/u/94968368?v=4)](https://github.com/rajenindianic "rajenindianic (15 commits)")[![dependabot[bot]](https://avatars.githubusercontent.com/in/29110?v=4)](https://github.com/dependabot[bot] "dependabot[bot] (5 commits)")[![nimah79](https://avatars.githubusercontent.com/u/20343056?v=4)](https://github.com/nimah79 "nimah79 (1 commits)")[![mechelon](https://avatars.githubusercontent.com/u/26432041?v=4)](https://github.com/mechelon "mechelon (1 commits)")[![gaichao168](https://avatars.githubusercontent.com/u/20994327?v=4)](https://github.com/gaichao168 "gaichao168 (1 commits)")[![laravel-shift](https://avatars.githubusercontent.com/u/15991828?v=4)](https://github.com/laravel-shift "laravel-shift (1 commits)") --- Tags phplaravellanguageprocessingtokennaturalcodexGPT-3openaidall-etokenizerchatChatGptchat gptcl100k\_basetokenizer-xr50k\_basep50k\_basep50k\_edit ### Code Quality TestsPest Code StyleLaravel Pint ### Embed Badge ![Health badge](/badges/juniora-tokenizer-x/health.svg) ``` [![Health](https://phpackages.com/badges/juniora-tokenizer-x/health.svg)](https://phpackages.com/packages/juniora-tokenizer-x) ``` ### Alternatives [rajentrivedi/tokenizer-x TokenizerX calculates required tokens for given prompt 92252.4k3](/packages/rajentrivedi-tokenizer-x)[laravel/ai The official AI SDK for Laravel. 1.0k3.2M246](/packages/laravel-ai)[openai-php/laravel OpenAI PHP for Laravel is a supercharged PHP API client that allows you to interact with the Open AI API 3.7k9.5M90](/packages/openai-php-laravel)[spatie/laravel-health Monitor the health of a Laravel application 87912.0M177](/packages/spatie-laravel-health)[masterix21/laravel-licensing Laravel licensing package with polymorphic assignment to any model, activation keys, expirations/renewals, and seat control via LicenseUsage. Supports offline verification with public-key–signed tokens, a CLI to generate/rotate/revoke keys, and an extensible architecture via config and contracts. 1613.3k4](/packages/masterix21-laravel-licensing) PHPackages © 2026 [Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)