PHPackages elementareteilchen/unduplicator - PHPackages - PHPackages [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register) 1. [Directory](/) 2. / 3. [Utility & Helpers](/categories/utility) 4. / 5. elementareteilchen/unduplicator ActiveTypo3-cms-extension[Utility & Helpers](/categories/utility) elementareteilchen/unduplicator =============================== Finds duplicates in sys\_file and unduplicates them v3.1.1(3w ago)52.9k↑157.7%4GPL-2.0-or-laterShellPHP ^8.2 || ^8.3 || ^8.4 || ^8.5CI passing Since May 8Pushed 3w ago5 watchersCompare [ Source](https://github.com/ElementareTeilchen/unduplicator)[ Packagist](https://packagist.org/packages/elementareteilchen/unduplicator)[ RSS](/packages/elementareteilchen-unduplicator/feed)WikiDiscussions main Synced yesterday READMEChangelogDependencies (8)Versions (12)Used By (0) unduplicator ============ [](#unduplicator) Finds and fixes duplicates of sys\_file entries pointing to the same file. Merges all references to point to the remaining sys\_file entry. Tested successfully with TYPO3 v12. Warning ------- [](#warning) Older versions for TYPO3 v8 may not consider identifiers with mixed case or sys\_file entries on several storages (sys\_file.storage) correctly, see issue [\#2](https://github.com/ElementareTeilchen/unduplicator/issues/2) Portabilty (database) --------------------- [](#portabilty-database) In order to test for duplicates, a database command like this is used: ``` SELECT COUNT(*), MAX(identifier) AS identifier, storage FROM `sys_file` GROUP BY MD5(identifier), storage HAVING COUNT(*) > 1; ``` Therefore, it is necessary, that the underlying database engines support MAX and MD5. This command was tested with the following: - MariaDB - MySQL - Postgres Usage ----- [](#usage) We strongly recommend to run the **reference index update** (before and after): If not run before or the references are out of date, some references may be overlooked and a sys\_file entry deleted which has references. Additionally you should consider to run the **scheduler task "File Abstraction Layer: Update storage index"** (before and after): This makes sense to recalculate the hash or other information. Also, the scheduler may create further duplicates. (If we do not run it now and the files are indexed later, we have new duplicates which are not taken care of). You can run the scheduler task in the scheduler backend module or via CLI, see ` vendor/bin/typo3 help scheduler:run` for details. Create the beforementioned scheduler task and use the UID in the next command: ``` php vendor/bin/typo3 scheduler:run --task= ``` Then use the following commands: ``` # dry-run: php vendor/bin/typo3 unduplicate:sysfile --update-refindex --dry-run # for real php vendor/bin/typo3 unduplicate:sysfile -n ``` ### Options [](#options) OptionAliasDescription`--dry-run``-d`Only shows the duplicates, but does not update the database.`--identifier``-i`Only consider duplicates with the given identifier.`--storage``-s`Only consider duplicates in the given storage.`--force``-f`Enforce keeping or overwriting of metadata of the master record in case of conflict. Possible values: keep, keep-nonempty, overwrite.`--keep-oldest``-o`Use the oldest record as master instead of the newest.`--meta-fields``-m`Check for conflicting metadata fields, see below.`--update-refindex``-u`Update the reference index before the operation.`--no-interaction``-n`Do not ask for the update of the reference index.Run another reference index update when you are done: ``` php vendor/bin/typo3 referenceindex:update ``` ### Examples [](#examples) ``` # run for a specific storage and identifier php vendor/bin/typo3 unduplicate:sysfile --storage 1 --identifier "/user_upload/duplicate.jpg" # `--keep-oldest` can be used in conjunction with `--force keep` to keep the oldest entries as they are and delete all others php vendor/bin/typo3 unduplicate:sysfile -n --force keep --keep-oldest ``` > **Note**For non-composer-mode the path to the CLI script is `typo3/sysext/core/bin/typo3` How the duplicate check is done ------------------------------- [](#how-the-duplicate-check-is-done) - check if sys\_file.storage and sys\_file.identifier is the same, but sys\_file.uid is different - we must make sure comparing identifier is done case-sensitively, this may not be the case for DB queries. In order to keep DB queries portable across different DB servers, we do an additional check in PHP. Check for conflicting metadata ------------------------------ [](#check-for-conflicting-metadata) You can specify meta data fields to check for conflicts. - If a conflict is found, the script will not delete the old record but show a warning. - If the data is the same or the old has none, the old is deleted. - If the data in the new is empty, the data from the old will be copied, if present. ``` php vendor/bin/typo3 unduplicate:sysfile --meta-fields "description, caption" ``` If the extension filemetadata is installed, the default check is on description, copyright and caption. Otherwise only description is checked. Run the functional Tests ------------------------ [](#run-the-functional-tests) ``` composer install ./Build/Scripts/runTests.sh -s functional -d mariadb -p 8.3 ``` ### Health Score 55 — FairBetter than 97% of packages Maintenance95 Actively maintained with recent releases Popularity28 Limited adoption so far Community17 Small or concentrated contributor base Maturity69 Established project with proven stability Bus Factor2 2 contributors hold 50%+ of commits How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window. **Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores. **Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement. **Maturity (30%)** — Project age, version count, PHP version support, and release stability. ### Release Activity Cadence Every ~125 days Recently: every ~109 days Total 10 Last Release 23d ago Major Versions v1.0.0 → v2.0.02024-07-17 v2.4.0 → v3.0.02026-02-16 PHP version history (3 changes)v1.0.0PHP ^7.4 || ^8.0 || ^8.1 || ^8.2 v2.0.0PHP ^8.1 || ^8.2 || ^8.3 v3.0.0PHP ^8.2 || ^8.3 || ^8.4 || ^8.5 ### Community Maintainers ![](https://www.gravatar.com/avatar/df8a3473fc20df57e5fc0b4e373256c9366493d032736bef52bccea07e94cd18?d=identicon)[elementareteilchen](/maintainers/elementareteilchen) --- Top Contributors [![ri-klein](https://avatars.githubusercontent.com/u/144095794?v=4)](https://github.com/ri-klein "ri-klein (12 commits)")[![franzkugelmann](https://avatars.githubusercontent.com/u/11320147?v=4)](https://github.com/franzkugelmann "franzkugelmann (11 commits)")[![morja](https://avatars.githubusercontent.com/u/29610198?v=4)](https://github.com/morja "morja (5 commits)")[![sypets](https://avatars.githubusercontent.com/u/13206455?v=4)](https://github.com/sypets "sypets (2 commits)")[![IchHabRecht](https://avatars.githubusercontent.com/u/1453345?v=4)](https://github.com/IchHabRecht "IchHabRecht (1 commits)") ### Code Quality TestsPHPUnit ### Embed Badge ![Health badge](/badges/elementareteilchen-unduplicator/health.svg) ``` [![Health](https://phpackages.com/badges/elementareteilchen-unduplicator/health.svg)](https://phpackages.com/packages/elementareteilchen-unduplicator) ``` ### Alternatives [friendsoftypo3/content-blocks TYPO3 CMS Content Blocks - Content Types API | Define reusable components via YAML 103519.9k53](/packages/friendsoftypo3-content-blocks)[wazum/sluggi TYPO3 extension for URL slug management with inline editing, auto-sync, locking, access control, and redirects 40529.5k](/packages/wazum-sluggi)[typo3/cms-scheduler TYPO3 CMS Scheduler - Schedule tasks to run once or periodically at a specific time. 169.3M231](/packages/typo3-cms-scheduler)[typo3/cms-lowlevel TYPO3 CMS Lowlevel - Technical analysis of the system. This includes raw database search, checking relations, counting pages and records etc. 178.2M316](/packages/typo3-cms-lowlevel)[typo3/cms-redirects TYPO3 CMS Redirects - Create manual redirects, list existing redirects and automatically createredirects on slug changes. 167.4M80](/packages/typo3-cms-redirects)[typo3/cms-form TYPO3 CMS Form - Flexible TYPO3 frontend form framework that comes with a backend editor interface. 147.6M265](/packages/typo3-cms-form) PHPackages © 2026 [Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)