PHPackages anthonyvipond/deduper-laravel - PHPackages - PHPackages [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register) 1. [Directory](/) 2. / 3. [Database & ORM](/categories/database) 4. / 5. anthonyvipond/deduper-laravel AbandonedLibrary[Database & ORM](/categories/database) anthonyvipond/deduper-laravel ============================= Deduplicate and remap your database tables 03PHP Since Dec 29Pushed 11y agoCompare [ Source](https://github.com/anthonyvipond/deduper-laravel)[ Packagist](https://packagist.org/packages/anthonyvipond/deduper-laravel)[ RSS](/packages/anthonyvipond-deduper-laravel/feed)WikiDiscussions master Synced 1mo ago READMEChangelogDependenciesVersions (1)Used By (0) MySQL DB Dedupe and Remap Tool ============================== [](#mysql-db-dedupe-and-remap-tool) Purpose ------- [](#purpose) - Removes duplicate records from your database and remaps foreign keys in other tables - You can easily define the uniqueness of a row using one or more columns - It works well on large tables (10M+ rows) as well. - Designed to run directly on production tables Installation ------------ [](#installation) The recommended way is through [composer](http://getcomposer.org). Edit `composer.json` and add: ``` { "require": { "anthonyvipond/deduper": "dev-master" } } ``` And install dependencies: ``` composer install ``` Copy `config/database.php.sample` to `config/database.php` and fill it out You should now be able to use the program from the command line (where dlr file is stored) ``` php dlr ``` Purpose ------- [](#purpose-1) \###Deduplicating Tables#### Suppose you have the following `people` table: idname2Mary3Joseph5Mary6mary7Joseph``` php dlr dedupe tableName columnName ``` **i.e.** ``` php dlr dedupe people name ``` Your original table will not be touched, and you will get this table `people_uniques` idname2Mary3JosephYou will also get this table `people_removes` idnamenew\_id5Marynull6Josephnull7Josephnull--- But what if you have a table where the uniqueness of defined over three columns? No problem. idfirstnamelastnamebirthday2MarySmith1991-01-013JosephParker1984-02-025MaryKate1981-08-086marykate2001-03-037JosephParker1984-02-02Seperate the columns with a `:` in the second argument: ``` php dlr dedupe people firstname:lastname:birthday ``` You will get a new table `people_uniques` idfirstnamelastnamebirthday2MarySmith1991-01-013JosephParker1984-02-025MaryKate1981-08-086marykate2001-03-03And another table `people_removes` idfirstnamelastnamebirthdaynew\_id7JosephParker1984-02-02null--- You can continue to deduplicate on different columns. Your `uniques` table will get smaller, and your `removes` table will get bigger. Take another look at the last stage our tables were in. Let's keep deduplicating further on new rules... ``` php dlr dedupe tableName firstname:lastname ``` Now `people_uniques` is like this: idfirstnamelastnamebirthday2MarySmith1991-01-013JosephParker1984-02-025MaryKate1981-08-08And `people_removes` is like this: idfirstnamelastnamebirthdaynew\_id7JosephParker1984-02-02null6marykate2001-03-03null\###Linking#### The next stage is add the new ids to the `removes` table ``` php dlr link uniquesTable removesTable col1:col2:col3 ``` *ie* ``` php dlr link people_uniques people_removes firstname:lastname:birthday ``` When doing linking, you should pass in the same columns as you did when deduping If you ran the dedupe command several times with different combinations, you want to link from the least specific to the most. *ie* ``` php dlr link people_uniques people_removes lastname:placeOfBirth php dlr link people_uniques people_removes firstname:lastname:birthday ``` This way the more specific and higher quality groupings overwrite the lower quality ones If you ran the `dedupe` command multiple times on different rules, you may end up with a small percentage of records that weren't linked after running the `link` command You can pass the `--fillerMode` option to fill the rest with of the `new_id` with ids Check how many `new_id` have not been remapped after each run: ``` SELECT count(1) FROM table_removes WHERE new_id IS NULL; ``` *ie* ``` php dlr link people_uniques people_removes firstname:lastname --fillerMode=true ``` \###Remapping#### After you run `dedupe` you will have **table\_uniques** and **table\_removes**, as well as your original table. The removes table **needs to be present** for remapping to work. It won't be written to but **needs to be read from**. Suppose you have this `teams` table: idteam2Knicks3Knicks4Lakers5KnicksAnd a `champions` table that needs remapping idteam\_id23324552And the `teams_uniques` table (remember, you deduped already) idteam2Knicks4LakersAnd you also have this `teams_removes` table which is used for remapping: idteamnew\_id3Knicks25Knicks2You can now remap the foreign keys on other tables pointing to `teams.id` ``` php dlr remap remapTable removesTable foreignKey ``` **i.e.** ``` php dlr remap champions teams_removes team_id ``` The `champions` table now looks like this: idteam\_id22324252You should backup your remap table prior to running the `remap` command. If your remapping doesn't finish the first time, just run it again. It won't hurt anything. \###Swapping in the new table### Going back to the `people` table example... Finish remapping foreign keys for all tables that point to `people.id` Now for the final coup-de-grace! ``` RENAME TABLE table TO table_bak; RENAME TABLE table_uniques TO table; DROP TABLE table_bak -- optional ``` Congrats! You've deduped and remapped your table. Contribution Guidelines ----------------------- [](#contribution-guidelines) - Post an issue! - Fork and pull. Notes ----- [](#notes) - For the time being, your original table with duplicates must have an `id` column - Only works on MySQL, but I'm open to adding more support ### Health Score 19 — LowBetter than 10% of packages Maintenance20 Infrequent updates — may be unmaintained Popularity3 Limited adoption so far Community6 Small or concentrated contributor base Maturity41 Maturing project, gaining track record Bus Factor1 Top contributor holds 100% of commits — single point of failure How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window. **Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores. **Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement. **Maturity (30%)** — Project age, version count, PHP version support, and release stability. ### Community Maintainers ![](https://www.gravatar.com/avatar/5766366d20e71a791d33e68ab01fa92311a8d854b2f5c8d2679a395a86e0b90a?d=identicon)[AnthonyVipond](/maintainers/AnthonyVipond) --- Top Contributors [![thewinterwind](https://avatars.githubusercontent.com/u/2460820?v=4)](https://github.com/thewinterwind "thewinterwind (89 commits)") ### Embed Badge ![Health badge](/badges/anthonyvipond-deduper-laravel/health.svg) ``` [![Health](https://phpackages.com/badges/anthonyvipond-deduper-laravel/health.svg)](https://phpackages.com/packages/anthonyvipond-deduper-laravel) ``` ### Alternatives [doctrine/orm Object-Relational-Mapper for PHP 10.2k285.3M6.2k](/packages/doctrine-orm)[jdorn/sql-formatter a PHP SQL highlighting library 3.9k115.1M102](/packages/jdorn-sql-formatter)[illuminate/database The Illuminate Database package. 2.8k52.4M9.4k](/packages/illuminate-database)[mongodb/mongodb MongoDB driver library 1.6k64.0M546](/packages/mongodb-mongodb)[ramsey/uuid-doctrine Use ramsey/uuid as a Doctrine field type. 90340.3M211](/packages/ramsey-uuid-doctrine)[reliese/laravel Reliese Components for Laravel Framework code generation. 1.7k3.4M16](/packages/reliese-laravel) PHPackages © 2026 [Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)