PHPackages megao-phpunit/php-token-stream - PHPackages - PHPackages [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register) 1. [Directory](/) 2. / 3. [Parsing & Serialization](/categories/parsing) 4. / 5. megao-phpunit/php-token-stream ActiveLibrary[Parsing & Serialization](/categories/parsing) megao-phpunit/php-token-stream ============================== Wrapper around PHP's tokenizer extension. 4.0.4(5y ago)145BSD-3-ClausePHPPHP ^7.3 || ^8.0 Since Sep 23Pushed 5y agoCompare [ Source](https://github.com/gkfsunshine/php-token-stream)[ Packagist](https://packagist.org/packages/megao-phpunit/php-token-stream)[ Docs](https://github.com/sebastianbergmann/php-token-stream/)[ GitHub Sponsors](https://github.com/sebastianbergmann)[ RSS](/packages/megao-phpunit-php-token-stream/feed)WikiDiscussions master Synced today READMEChangelogDependencies (1)Versions (38)Used By (0) phpunit/php-token-stream ======================== [](#phpunitphp-token-stream) [![CI Status](https://github.com/sebastianbergmann/php-token-stream/workflows/CI/badge.svg)](https://github.com/sebastianbergmann/php-token-stream/actions)[![Type Coverage](https://camo.githubusercontent.com/629ec76c83169e6ed9575a5e681c318627af31097193175b780af32ff394229f/68747470733a2f2f73686570686572642e6465762f6769746875622f73656261737469616e626572676d616e6e2f7068702d746f6b656e2d73747265616d2f636f7665726167652e737667)](https://shepherd.dev/github/sebastianbergmann/php-token-stream) Installation ------------ [](#installation) You can add this library as a local, per-project dependency to your project using [Composer](https://getcomposer.org/): ``` composer require phpunit/php-token-stream ``` If you only need this library during development, for instance to run your project's test suite, then you should add it as a development-time dependency: ``` composer require --dev phpunit/php-token-stream ``` ### Health Score 34 — LowBetter than 75% of packages Maintenance20 Infrequent updates — may be unmaintained Popularity9 Limited adoption so far Community17 Small or concentrated contributor base Maturity80 Battle-tested with a long release history Bus Factor1 Top contributor holds 88.5% of commits — single point of failure How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window. **Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores. **Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement. **Maturity (30%)** — Project age, version count, PHP version support, and release stability. ### Release Activity Cadence Every ~83 days Recently: every ~42 days Total 37 Last Release 2041d ago Major Versions 1.4.11 → 2.0.02017-08-03 1.4.12 → 3.0.02018-02-01 3.1.1 → 4.0.02020-02-07 PHP version history (7 changes)1.1.4PHP >=5.2.7 1.1.5PHP >=5.3.3 2.0.0PHP ^7.0 3.0.0PHP ^7.1 4.0.0PHP ^7.3 4.0.3PHP ^7.3 || ^8.0 3.1.x-devPHP >=7.1 ### Community Maintainers ![](https://avatars.githubusercontent.com/u/30225986?v=4)[gkfsunshine](/maintainers/gkfsunshine)[@gkfsunshine](https://github.com/gkfsunshine) --- Top Contributors [![sebastianbergmann](https://avatars.githubusercontent.com/u/25218?v=4)](https://github.com/sebastianbergmann "sebastianbergmann (324 commits)")[![edorian](https://avatars.githubusercontent.com/u/247397?v=4)](https://github.com/edorian "edorian (9 commits)")[![christeredvartsen](https://avatars.githubusercontent.com/u/25402?v=4)](https://github.com/christeredvartsen "christeredvartsen (6 commits)")[![whatthejeff](https://avatars.githubusercontent.com/u/306525?v=4)](https://github.com/whatthejeff "whatthejeff (6 commits)")[![localheinz](https://avatars.githubusercontent.com/u/605483?v=4)](https://github.com/localheinz "localheinz (2 commits)")[![sun](https://avatars.githubusercontent.com/u/41992?v=4)](https://github.com/sun "sun (2 commits)")[![ivyhjk](https://avatars.githubusercontent.com/u/12478859?v=4)](https://github.com/ivyhjk "ivyhjk (2 commits)")[![orklah](https://avatars.githubusercontent.com/u/9605520?v=4)](https://github.com/orklah "orklah (2 commits)")[![GrahamCampbell](https://avatars.githubusercontent.com/u/2829600?v=4)](https://github.com/GrahamCampbell "GrahamCampbell (2 commits)")[![rayrutjes](https://avatars.githubusercontent.com/u/5570853?v=4)](https://github.com/rayrutjes "rayrutjes (1 commits)")[![fredemmott](https://avatars.githubusercontent.com/u/360927?v=4)](https://github.com/fredemmott "fredemmott (1 commits)")[![slashmili](https://avatars.githubusercontent.com/u/585764?v=4)](https://github.com/slashmili "slashmili (1 commits)")[![StephenClouse](https://avatars.githubusercontent.com/u/468025?v=4)](https://github.com/StephenClouse "StephenClouse (1 commits)")[![tamcap](https://avatars.githubusercontent.com/u/5289937?v=4)](https://github.com/tamcap "tamcap (1 commits)")[![tomzx](https://avatars.githubusercontent.com/u/188960?v=4)](https://github.com/tomzx "tomzx (1 commits)")[![gkfsunshine](https://avatars.githubusercontent.com/u/30225986?v=4)](https://github.com/gkfsunshine "gkfsunshine (1 commits)")[![henriquemoody](https://avatars.githubusercontent.com/u/154023?v=4)](https://github.com/henriquemoody "henriquemoody (1 commits)")[![loilo](https://avatars.githubusercontent.com/u/1922624?v=4)](https://github.com/loilo "loilo (1 commits)")[![mazzcris](https://avatars.githubusercontent.com/u/6614577?v=4)](https://github.com/mazzcris "mazzcris (1 commits)")[![oker1](https://avatars.githubusercontent.com/u/431237?v=4)](https://github.com/oker1 "oker1 (1 commits)") --- Tags tokenizer ### Code Quality TestsPHPUnit ### Embed Badge ![Health badge](/badges/megao-phpunit-php-token-stream/health.svg) ``` [![Health](https://phpackages.com/badges/megao-phpunit-php-token-stream/health.svg)](https://phpackages.com/packages/megao-phpunit-php-token-stream) ``` ### Alternatives [yethee/tiktoken PHP version of tiktoken 1624.0M30](/packages/yethee-tiktoken)[gioni06/gpt3-tokenizer PHP package for Byte Pair Encoding (BPE) used by GPT-3. 84591.1k11](/packages/gioni06-gpt3-tokenizer)[rajentrivedi/tokenizer-x TokenizerX calculates required tokens for given prompt 92252.4k3](/packages/rajentrivedi-tokenizer-x)[nicoswd/php-rule-parser Rule Engine - Rule Parser & Evaluator 13181.4k7](/packages/nicoswd-php-rule-parser)[jeremeamia/functionparser Function parser for PHP functions, methods, and closures 48174.6k6](/packages/jeremeamia-functionparser)[danny50610/bpe-tokeniser PHP port for openai/tiktoken (most) 10457.3k2](/packages/danny50610-bpe-tokeniser) PHPackages © 2026 [Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)