PHPackages mathematicator-core/tokenizer - PHPackages - PHPackages [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register) 1. [Directory](/) 2. / 3. [Parsing & Serialization](/categories/parsing) 4. / 5. mathematicator-core/tokenizer ActiveLibrary[Parsing & Serialization](/categories/parsing) mathematicator-core/tokenizer ============================= Math Tokenizer v3.0.1(5y ago)105.0k2[4 issues](https://github.com/mathematicator-core/tokenizer/issues)[4 PRs](https://github.com/mathematicator-core/tokenizer/pulls)6MITPHPPHP ^8.0 Since Jul 25Pushed 1y ago2 watchersCompare [ Source](https://github.com/mathematicator-core/tokenizer)[ Packagist](https://packagist.org/packages/mathematicator-core/tokenizer)[ Docs](https://github.com/mathematicator-core/tokenizer)[ RSS](/packages/mathematicator-core-tokenizer/feed)WikiDiscussions master Synced today READMEChangelog (10)Dependencies (12)Versions (21)Used By (6) PHP Math Tokenizer ==================== [](#----php-math-tokenizer) [ ![](https://avatars3.githubusercontent.com/u/44620375?s=100&v=4) ](https://mathematicator.com) [![Integrity check](https://github.com/mathematicator-core/tokenizer/workflows/Integrity%20check/badge.svg)](https://github.com/mathematicator-core/tokenizer/actions?query=workflow%3A%22Integrity+check%22)[![codecov](https://camo.githubusercontent.com/065b2adc8228bad95df90f6c1a12bac6115c8873fd760d75cf1050e38e22f7bb/68747470733a2f2f636f6465636f762e696f2f67682f6d617468656d6174696361746f722d636f72652f746f6b656e697a65722f6272616e63682f6d61737465722f67726170682f62616467652e737667)](https://codecov.io/gh/mathematicator-core/tokenizer)[![Latest Stable Version](https://camo.githubusercontent.com/7f16ee81243abd8238a97a255e83625e1c5459f901535ed947ef290162ee77f8/68747470733a2f2f706f7365722e707567782e6f72672f6d617468656d6174696361746f722d636f72652f746f6b656e697a65722f762f737461626c65)](https://packagist.org/packages/mathematicator-core/tokenizer)[![Latest Unstable Version](https://camo.githubusercontent.com/9d9b575ac058b6bf9bbc28a680e864c99c113c1333cd52d6baade0a30a287112/68747470733a2f2f706f7365722e707567782e6f72672f6d617468656d6174696361746f722d636f72652f746f6b656e697a65722f762f756e737461626c65)](https://packagist.org/packages/mathematicator-core/tokenizer)[![License: MIT](https://camo.githubusercontent.com/1a2e0606685ce00663bf829868f794fd3fc9c86f8d80cae324734129e0723a58/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4c6963656e73652d4d49542d627269676874677265656e2e737667)](./LICENSE)[![PHPStan Enabled](https://camo.githubusercontent.com/9142f3d2ec7588a75127608ff41f595540402524559c6f50feb756df4b8cf6c5/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f5048505374616e2d656e61626c65642532304c382d627269676874677265656e2e7376673f7374796c653d666c6174)](https://phpstan.org/) Mathematicator Tokenizer is a simple library for converting string math formulas to a stream of tokens with LaTeX support. Installation ------------ [](#installation) ``` composer require mathematicator-core/tokenizer ``` Features -------- [](#features) - Convert all your math formulas to a stream of tokens - Convert user math input to LaTeX - Render the tokens tree map for debug Usage ----- [](#usage) Inject the `Tokenizer` service via [DI](https://doc.nette.org/en/3.0/dependency-injection)and tokenize your query. ``` use \Mathematicator\Tokenizer\Tokenizer; $tokenizer = new Tokenizer(/* some dependencies */); // Convert math formula to an array of tokens: $tokens = $tokenizer->tokenize('(5+3)*(2/(7+3))'); // Now you can convert tokens to a more useful format: $objectTokens = $tokenizer->tokensToObject($tokens); var_dump($objectTokens); // Return typed tokens with meta data // Render to LaTeX echo $tokenizer->tokensToLatex($objectTokens); // Render to debug tree (extremely fast): echo $tokenizer->renderTokensTree($objectTokens); ``` ### Configuration [](#configuration) The tokenizer uses automatic configuration based on DI. Just use the DI container and the service will be fully available. Mathematicator Framework tools structure ---------------------------------------- [](#mathematicator-framework-tools-structure) The biggest advantage is that you can choose which layer best fits your needs and start build on the top of it, immediately, without the need to create everything by yourself. Our tools are tested for bugs and tuned for performance, so you can save a significant amount of your time, money, and effort. Framework tend to be modular as much as possible, so you should be able to create an extension on each layer and its sublayers. **Mathematicator framework layers** ordered from the most concrete one to the most abstract one: **[ Search ](https://github.com/mathematicator-core/search)** Modular search engine layer that calls its sublayers and creates user interface. **[ Vizualizator ](https://github.com/mathematicator-core/vizualizator)** Elegant graphic visualizer that can render to SVG, PNG, JPG and Base64. Extensions: **[ Mandelbrot set generator ](https://github.com/mathematicator-core/mandelbrot-set)** **[ Calculator ](https://github.com/mathematicator-core/calculator)** Modular advance calculations layer. Extensions: **[ Integral Solver ](https://github.com/mathematicator-core/integral-solver), [ Statistics ](https://github.com/mathematicator-core/statistic)** **[ Engine ](https://github.com/mathematicator-core/engine)** Core logic layer that maintains basic controllers, DAOs, translator, common exceptions, routing etc. **[ Tokenizer ](https://github.com/mathematicator-core/tokenizer)** Tokenizer that can convert string (user input / LaTeX) to numbers and operators. **[ Numbers ](https://github.com/mathematicator-core/numbers)** Fast & secure storage for numbers with arbitrary precision. It supports Human string and LaTeX output and basic conversions. **Third-party packages:** ⚠️ Not guaranteed! **[ REST API ](https://github.com/cothema/math-php-api)** Install the whole pack as a REST API service on your server (Docker ready) or access it via public cloud REST API. Contribution ------------ [](#contribution) > Please help to improve this documentation by sending a Pull request. ### Tests [](#tests) All new contributions should have its unit tests in `/tests` directory. Before you send a PR, please, check all tests pass. This package uses [Nette Tester](https://tester.nette.org/). You can run tests via command: ``` composer test ``` For benchmarking, we use [phpbench](https://github.com/phpbench/phpbench). You can run benchmarks this way: ``` composer global require phpbench/phpbench @dev # only the first time phpbench run ``` Before PR, please run complete code check via command: ``` composer cs:install # only first time composer fix # otherwise pre-commit hook can fail ``` ### Health Score 39 — LowBetter than 84% of packages Maintenance18 Infrequent updates — may be unmaintained Popularity30 Limited adoption so far Community21 Small or concentrated contributor base Maturity73 Established project with proven stability Bus Factor1 Top contributor holds 77.3% of commits — single point of failure How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window. **Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores. **Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement. **Maturity (30%)** — Project age, version count, PHP version support, and release stability. ### Release Activity Cadence Every ~43 days Recently: every ~64 days Total 14 Last Release 1970d ago Major Versions v1.0.6 → v2.0.02020-04-14 v2.1.2 → v3.0.02021-02-10 PHP version history (3 changes)v1.0.0PHP >=7.1 v2.1.0PHP >=7.2 v3.0.0PHP ^8.0 ### Community Maintainers ![](https://avatars.githubusercontent.com/u/3382204?v=4)[baraja](/maintainers/baraja)[@baraja](https://github.com/baraja) --- Top Contributors [![janbarasek](https://avatars.githubusercontent.com/u/4738758?v=4)](https://github.com/janbarasek "janbarasek (68 commits)")[![flokixdev](https://avatars.githubusercontent.com/u/2387790?v=4)](https://github.com/flokixdev "flokixdev (18 commits)")[![dependabot-preview[bot]](https://avatars.githubusercontent.com/in/2141?v=4)](https://github.com/dependabot-preview[bot] "dependabot-preview[bot] (1 commits)")[![iDome89](https://avatars.githubusercontent.com/u/44414108?v=4)](https://github.com/iDome89 "iDome89 (1 commits)") --- Tags binary-treecalculatorlatexmathmath-formulasmath-languagemath-tokenizertokenizerphpmathtokenizermathematicator ### Code Quality Static AnalysisPHPStan Type Coverage Yes ### Embed Badge ![Health badge](/badges/mathematicator-core-tokenizer/health.svg) ``` [![Health](https://phpackages.com/badges/mathematicator-core-tokenizer/health.svg)](https://phpackages.com/packages/mathematicator-core-tokenizer) ``` ### Alternatives [rajentrivedi/tokenizer-x TokenizerX calculates required tokens for given prompt 92252.4k3](/packages/rajentrivedi-tokenizer-x) PHPackages © 2026 [Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)