PHPackages                             nitotm/efficient-language-detector - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Utility & Helpers](/categories/utility)
4. /
5. nitotm/efficient-language-detector

ActiveLibrary[Utility & Helpers](/categories/utility)

nitotm/efficient-language-detector
==================================

Fast and accurate natural language detection. Detector written in PHP. Nito-ELD, ELD.

v3.1.1(4mo ago)59252.9k↓34%7[1 issues](https://github.com/nitotm/efficient-language-detector/issues)4Apache-2.0PHPPHP ^7.4 || ^8.0

Since Jun 19Pushed 1mo ago3 watchersCompare

[ Source](https://github.com/nitotm/efficient-language-detector)[ Packagist](https://packagist.org/packages/nitotm/efficient-language-detector)[ Docs](https://github.com/nitotm/efficient-language-detector)[ Fund](https://linktr.ee/nitotm)[ GitHub Sponsors](https://github.com/nitotm)[ RSS](/packages/nitotm-efficient-language-detector/feed)WikiDiscussions main Synced 1mo ago

READMEChangelog (7)DependenciesVersions (11)Used By (4)

Efficient Language Detector
===========================

[](#efficient-language-detector)

[![supported PHP versions](https://camo.githubusercontent.com/62c238ea2ddcab9b241105566c72f5b5708876995703c4b6580da0733d0231af/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f5048502d253345253344253230372e342d626c7565)](https://camo.githubusercontent.com/62c238ea2ddcab9b241105566c72f5b5708876995703c4b6580da0733d0231af/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f5048502d253345253344253230372e342d626c7565)[![license](https://camo.githubusercontent.com/b29de0acdfd19013f1f02689b15c933e4a6c145be9efa718288f88ba3280b1c5/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667)](https://www.apache.org/licenses/LICENSE-2.0)[![supported languages](https://camo.githubusercontent.com/251c8e2b874e6d3c9250a41a33288285a2f1e203c24bbf56ffc233a477c82d65/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f737570706f727465642532306c616e6775616765732d36302d627269676874677265656e2e737667)](#languages)[![version](https://camo.githubusercontent.com/397dbba818dd8eb495ef6b9542f70719891897a363131dbbc13e241117d2d375/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f7665722e2d332e302d626c7565)](https://camo.githubusercontent.com/397dbba818dd8eb495ef6b9542f70719891897a363131dbbc13e241117d2d375/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f7665722e2d332e302d626c7565)

Efficient language detector (*Nito-ELD* or *ELD*) is a fast and accurate natural language detection software, written 100% in PHP, with a speed comparable to fast C++ compiled detectors, and accuracy rivaling the best detectors to date.

It has no dependencies, easy installation, all it's needed is PHP with the **mb** extension.
ELD scales perfectly with database size.
ELD is also available (outdated versions) in [Javascript](https://github.com/nitotm/efficient-language-detector-js) and [Python](https://github.com/nitotm/efficient-language-detector-py).

1. [Installation](#installation)
2. [How to use](#how-to-use)
3. [Benchmarks](#benchmarks)
4. [Databases](#databases)
5. [Testing](#testing)
6. [Languages](#languages)

Installation
------------

[](#installation)

```
$ composer require nitotm/efficient-language-detector
```

- `--prefer-dist` will omit *tests/*, *misc/* & *benchmark/*, or use `--prefer-source` to include everything
- Install `nitotm/efficient-language-detector:dev-main` to try the last unstable changes
- Alternatively, download / clone the files can work just fine.
    (Only *small* DB install under construction)

### ELD execution options

[](#eld-execution-options)

- ELD has database execution Modes: `array`, `string`, `bytes`, `disk`
- ELD database Sizes: `small`, `medium`, `large`, `extralarge`
- Confused? No worries, use default with no arguments. [How to use](#how-to-use)

**Low memory Modes**
For Modes `string`, `bytes` and `disk`, all size databases can run with *128MB* PHP default setting
`disk` mode with `extralarge` size, can run with almost no RAM as it only uses 0.5MB
`string` and `bytes` are a great choice for general use, as they are just ~2x slower than `array`

**Fastest Mode: Array** (higher memory usage)
For `array` Mode it is recommended to use OPcache, specially for the larger databases to reduce load times
We need to set `opcache.interned_strings_buffer`, `opcache.memory_consumption` high enough for each database

Check [Databases](#databases) for more info.

How to use?
-----------

[](#how-to-use)

`detect()` expects a UTF-8 string and returns an object with a `language` property, containing an *ISO 639-1* code (or other selected scheme), or `'und'` for undetermined language.

```
use Nitotm\Eld\LanguageDetector;

$eld = new LanguageDetector(); // Default Size: 'large', Mode: 'string' (since v3.2)

$eld->detect('Hola, cómo te llamas?');
// object( language => string, scores() => array, isReliable() => bool )
// ( language => 'es', scores() => ['es' => 0.25, 'nl' => 0.05], isReliable() => true )

$eld->detect('Hola, cómo te llamas?')->language;
// 'es'
```

To select database **Size** and **Mode**, or language output scheme. We can import `Eld...` constants to see avalible options.

```
use Nitotm\Eld\{LanguageDetector, EldDataFile, EldScheme, EldMode};

// LanguageDetector(databaseFile: ?string, outputFormat: ?string, mode: string)
$eld = new LanguageDetector(EldDataFile::SMALL, EldScheme::ISO639_1, EldMode::MODE_ARRAY);

// Database Size files: 'small', 'medium', 'large', 'extralarge'.
// Schemes: 'ISO639_1', 'ISO639_2T', 'ISO639_1_BCP47', 'ISO639_2T_BCP47' and 'FULL_TEXT'
// Database Modes: 'array', 'string', 'bytes', 'disk'. Check memory requirements for 'array'
// Constants are not mandatory, LanguageDetector('small'); will also work
```

#### Languages subsets

[](#languages-subsets)

Calling `langSubset()` once, will set the subset.

- In `array` mode the first call takes longer as it creates a new database, if save enabled (default), it will be loaded next time we make the same subset.
- In modes `string`, `bytes` & `disk`, a "virtual" subset is created instantly, `detect()` will just remove unwanted languages before returning results.
- To load a subset with 0 overhead, we can feed the returned file by `langSubset()` in `array` mode, when creating the instance `LanguageDetector(file)`
    - To make use of pre-built subsets in modes `string`, `bytes` & `disk`, getting lower memory usage and increased speed, it is possible by manually converting an `array` database, using `BlobDataBuilder()`
- Check available [Languages](#languages) below.

```
// It accepts any ISO codes. In "array" mode it will return a subset file name, if saved
// langSubset(languages: [], save: true, encode: true);
$eld->langSubset(['en', 'es', 'fr', 'it', 'nl', 'de']);
// Object ( success => bool, languages => ?array, error => ?string, file => ?string )
// ( success => true, languages => ['en', 'es'...], error => NULL, file => 'small_6_mfss...' )

// to remove the subset
$eld->langSubset();

// Load pre-saved subset directly, just like a default database
$eld_subset = new Nitotm\Eld\LanguageDetector('small_6_mfss5z1t', null, 'array');

// Build a binary database for modes 'string', 'bytes' & 'disk', from any 'array' database
// Memory requirements for 'array' database input apply
$eldBuilder = new Nitotm\Eld\BlobDataBuilder('large'); // or subset 'small_6_mfss5z1t'
// Create subset directly: new BlobDataBuilder('extralarge', ['en', 'es', 'de', 'it']);
$eldBuilder->buildDatabase();
```

#### Other Functions

[](#other-functions)

```
// if enableTextCleanup(True), detect() removes Urls, .com domains, emails, alphanumerical...
// Not recommended, as urls & domains contain hints of a language, which might help accuracy
$eld->enableTextCleanup(true); // Default is false

// If needed, we can get info of the ELD instance: languages, database type, etc.
$eld->info();

// Change output scheme on demand
// 'ISO639_1', 'ISO639_2T', 'ISO639_1_BCP47', 'ISO639_2T_BCP47', 'FULL_TEXT'
$eld->setOutputScheme('ISO639_2T'); // returns bool true on success
```

There is a CLI wrapper (BETA version)
`>./bin/eld --help` on Linux.
`>php bin/eld --help` on Windows.

Benchmarks
----------

[](#benchmarks)

I compared *ELD* with a different variety of detectors, as there are not many in PHP.

URLVersionLanguage3.1.0PHP2.0.2Python0.9.2C++Aug 21, 2015C++5.3.0PHP7.2.0JavascriptBenchmarks:

- **Tatoeba**: *20MB*, short sentences from Tatoeba, 50 languages supported by all contenders, up to 10k lines each.

> - For Tatoeba, I limited all detectors to the 50 languages subset, making the comparison as fair as possible.
> - Also, Tatoeba is not part of **ELD** training dataset (nor tuning), but it is for **fasttext**

- **ELD Test**: *10MB*, sentences from the 60 languages supported by ELD, 1000 lines each. Extracted from the 60GB of ELD training data.
- **Sentences**: *8MB*, sentences from *Lingua* benchmark, minus unsupported languages and Yoruba which had broken characters.
- **Word pairs** *1.5MB*, and **Single words** *870KB*, also from Lingua, same 53 languages.

Time execution benchmark for ELD size `large` ( check others sizes at [more benchmarks](https://raw.githubusercontent.com/nitotm/efficient-language-detector/main/misc/table_time_extra_v3.svg) )
[![timetable](https://raw.githubusercontent.com/nitotm/efficient-language-detector/main/misc/table_time_v3.svg)](https://raw.githubusercontent.com/nitotm/efficient-language-detector/main/misc/table_time_v3.svg)

[![accuracy table](https://raw.githubusercontent.com/nitotm/efficient-language-detector/main/misc/table_accuracy_v3.svg)](https://raw.githubusercontent.com/nitotm/efficient-language-detector/main/misc/table_accuracy_v3.svg)

- **Lingua** participates with 54 languages, **Franc** with 58, **patrickschur** with 54.
- **fasttext** does not have a built-in subset option, so to show its accuracy and speed potential I made two benchmarks, fasttext-all not being limited by any subset at any test
- \* Google's **CLD2** also lacks subset option, and it's difficult to make a subset even with its option `bestEffort = True`, as usually returns only one language, so it has a comparative disadvantage.
- Time is normalized: (total lines \* time) / processed lines

Databases
---------

[](#databases)

### Low memory database modes

[](#low-memory-database-modes)

Modes `'bytes'` and `'string'` are very similar, they differ on how they are load, and are just 2x slower than **Array**
Mode `'string'` can be OPcache'd, more expensive compilation, but then instant load, `'bytes'` has always a steady ~fast load
Special mention to `'disk'` mode, while slower, is the fastest uncached load & detect for the larger databases

ModeDiskBytesStringBytesStringDatabase Size optionExtralargeExtralargeExtralargeLargeLargeFile size39 MB39 MB39 MB20 MB20 MBMemory usage0.4 MB40 MB40 MB22 MB22 MBMemory usage Cached0.4 MB40 MB0.4 MB + OP22 MB0.4 MB + OPMemory peak0.4 MB40 MB56 MB22 MB32 MBMemory peak Cached0.4 MB40 MB0.4 MB + OP22 MB0.4 MB + OPOPcache used memory--39 MB-20 MBOPcache used interned--0.4 MB-0.4 MBLoad & detect() Uncached0.0012 sec0.04 sec0.25 sec0.02 sec0.11 secLoad & detect() Cached0.0011 sec0.04 sec0.0003 sec0.02 sec0.0003 secModeBytesStringBytesStringDatabase Size optionMediumMediumSmallSmallFile size6 MB6 MB2 MB2 MBMemory usage8 MB8 MB2 MB2 MBMemory usage Cached8 MB0.4 MB + OP2 MB0.4 MB + OPMemory peak8 MB12 MB2 MB3 MBMemory peak Cached8 MB0.4 MB + OP2 MB0.4 MB + OPOPcache used memory-0 MB-0 MBOPcache used interned-6 MB-2 MBLoad & detect() Uncached0.006 sec0.04 sec0.003 sec0.016 secLoad & detect() Cached0.006 sec0.0003 sec0.002 sec0.0003 sec### Fastest mode *Array*, but memory hungry

[](#fastest-mode-array-but-memory-hungry)

Array Mode, Size:SmallMediumLargeExtralargeProsLowest memoryEquilibratedFastestMost accurateConsLeast accurateSlowest (but fast)High memoryHighest memoryFile size3 MB9 MB28 MB64 MBMemory usage46 MB137 MB547 MB1143 MBMemory usage Cached0.4 MB + OP0.4 MB + OP0.4 MB + OP0.4 MB + OPMemory peak78 MB287 MB969 MB2047 MBMemory peak Cached0.4 MB + OP0.4 MB + OP0.4 MB + OP0.4 MB + OPOPcache used memory21 MB70 MB242 MB516 MBOPcache used interned4 MB10 MB45 MB91 MBLoad & detect() Uncached0.13 sec0.5 sec1.4 sec3.2 secLoad & detect() Cached0.0003 sec0.0003 sec0.0003 sec0.0003 sec**Settings** (Recommended)`memory_limit`>= 128>= 340>= 1060>= 2200`opcache.interned...`\*>= 8 (16)>= 16 (32)>= 60 (70)>= 116 (128)`opcache.memory`>= 64 (128)>= 128 (230)>= 360 (450)>= 750 (820)- \* I recommend using more than enough `interned_strings_buffer` as *buffers overflow* error might delay server response.
    To use *all* databases `opcache.interned_strings_buffer` should be a minimum of 160MB (170MB).
- When choosing the amount of memory keep in mind `opcache.memory_consumption` includes `opcache.interned_strings_buffer`.
    - If OPcache memory is 230MB, interned\_strings is 32MB, and medium DB is 69MB cached, we have a total of (230 -32 -69) = 129MB of OPcache for everything else.
- Also, if you are going to use a subset of languages in addition to the main database, or multiple subsets, increase `opcache.memory` accordingly if you want them to be loaded instantly. To cache all default databases comfortably you would want to set it at 1200MB.

Testing
-------

[](#testing)

Default composer install might not include these files. Use `--prefer-source` to include them.

- For *dev* environment with composer *"autoload-dev"* (root only), the following will execute the tests

```
new Nitotm\Eld\Tests\TestsAutoload();
```

- Or, you can also run the tests executing the following file:

```
$ php efficient-language-detector/tests/tests.php # Update path
```

- To run the accuracy benchmarks run the `benchmark/bench.php` file.

Languages
---------

[](#languages)

- These are the *ISO 639-1 codes* that include the 60 languages. Plus `'und'` for undetermined
    It is the default ELD language scheme. `outputScheme: 'ISO639_1'`

> am, ar, az, be, bg, bn, ca, cs, da, de, el, en, es, et, eu, fa, fi, fr, gu, he, hi, hr, hu, hy, is, it, ja, ka, kn, ko, ku, lo, lt, lv, ml, mr, ms, nl, no, or, pa, pl, pt, ro, ru, sk, sl, sq, sr, sv, ta, te, th, tl, tr, uk, ur, vi, yo, zh

- These are the 60 supported languages for *Nito-ELD*. `outputScheme: 'FULL_TEXT'`

> Amharic, Arabic, Azerbaijani (Latin), Belarusian, Bulgarian, Bengali, Catalan, Czech, Danish, German, Greek, English, Spanish, Estonian, Basque, Persian, Finnish, French, Gujarati, Hebrew, Hindi, Croatian, Hungarian, Armenian, Icelandic, Italian, Japanese, Georgian, Kannada, Korean, Kurdish (Arabic), Lao, Lithuanian, Latvian, Malayalam, Marathi, Malay (Latin), Dutch, Norwegian, Oriya, Punjabi, Polish, Portuguese, Romanian, Russian, Slovak, Slovene, Albanian, Serbian (Cyrillic), Swedish, Tamil, Telugu, Thai, Tagalog, Turkish, Ukrainian, Urdu, Vietnamese, Yoruba, Chinese

- *ISO 639-1 codes* with IETF BCP 47 script name tag. `outputScheme: 'ISO639_1_BCP47'`

> am, ar, az-Latn, be, bg, bn, ca, cs, da, de, el, en, es, et, eu, fa, fi, fr, gu, he, hi, hr, hu, hy, is, it, ja, ka, kn, ko, ku-Arab, lo, lt, lv, ml, mr, ms-Latn, nl, no, or, pa, pl, pt, ro, ru, sk, sl, sq, sr-Cyrl, sv, ta, te, th, tl, tr, uk, ur, vi, yo, zh

- *ISO 639-2/T* codes (which are also valid *639-3*) `outputScheme: 'ISO639_2T'`. Also available with BCP 47 `ISO639_2T_BCP47`

> amh, ara, aze, bel, bul, ben, cat, ces, dan, deu, ell, eng, spa, est, eus, fas, fin, fra, guj, heb, hin, hrv, hun, hye, isl, ita, jpn, kat, kan, kor, kur, lao, lit, lav, mal, mar, msa, nld, nor, ori, pan, pol, por, ron, rus, slk, slv, sqi, srp, swe, tam, tel, tha, tgl, tur, ukr, urd, vie, yor, zho

---

#### Donations and suggestions

[](#donations-and-suggestions)

If you wish to donate for open source improvements, hire me for private modifications, request alternative dataset training, or contact me, please use the following link:

###  Health Score

56

—

FairBetter than 98% of packages

Maintenance83

Actively maintained with recent releases

Popularity49

Moderate usage in the ecosystem

Community24

Small or concentrated contributor base

Maturity55

Maturing project, gaining track record

 Bus Factor1

Top contributor holds 86.7% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Every ~104 days

Recently: every ~122 days

Total

10

Last Release

126d ago

Major Versions

v1.0.3 → v2.0.02023-07-17

v2.1.1 → v3.0.0-BETA2024-09-05

v2.1.2 → v3.0.02024-10-10

PHP version history (2 changes)v1.0.3PHP ^7.3 || ^8.0

v2.0.0PHP ^7.4 || ^8.0

### Community

Maintainers

![](https://www.gravatar.com/avatar/d77b574675808b6e7b429417c476e74824396ec202555dd974217153549cf56f?d=identicon)[nitotm](/maintainers/nitotm)

---

Top Contributors

[![nitotm](https://avatars.githubusercontent.com/u/13847012?v=4)](https://github.com/nitotm "nitotm (26 commits)")[![Bgright](https://avatars.githubusercontent.com/u/186396193?v=4)](https://github.com/Bgright "Bgright (1 commits)")[![r4ndsen](https://avatars.githubusercontent.com/u/592863?v=4)](https://github.com/r4ndsen "r4ndsen (1 commits)")[![rarentsenvisymo](https://avatars.githubusercontent.com/u/126779322?v=4)](https://github.com/rarentsenvisymo "rarentsenvisymo (1 commits)")[![Toflar](https://avatars.githubusercontent.com/u/481937?v=4)](https://github.com/Toflar "Toflar (1 commits)")

---

Tags

languagelanguage-classificationlanguage-detectionlanguage-detectorlanguage-identificationnatural-languagenatural-language-processingnlpphplanguagedetectordetectionlanguage detectorlanguage identificationlanguage detectionlanguage recognitionlanguage classification

### Embed Badge

![Health badge](/badges/nitotm-efficient-language-detector/health.svg)

```
[![Health](https://phpackages.com/badges/nitotm-efficient-language-detector/health.svg)](https://phpackages.com/packages/nitotm-efficient-language-detector)
```

###  Alternatives

[phpoption/phpoption

Option Type for PHP

2.7k541.2M159](/packages/phpoption-phpoption)[patrickschur/language-detection

A language detection library for PHP. Detects the language from a given text string.

8513.2M18](/packages/patrickschur-language-detection)[shipmonk/composer-dependency-analyser

Fast detection of composer dependency issues (dead dependencies, shadow dependencies, misplaced dependencies)

6156.7M435](/packages/shipmonk-composer-dependency-analyser)[landrok/language-detector

A fast and reliable PHP library for detecting languages

132647.0k3](/packages/landrok-language-detector)[tivie/php-os-detector

A small utility library that detects the OS the server is running on

201.6M11](/packages/tivie-php-os-detector)[ideea/language-detector

Detect languages by text

334.4k](/packages/ideea-language-detector)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)