On the De-duplication of the Lakh MIDI Dataset

Anonymous submission to ISMIR 2025

This study investigates the dataset duplication issues regarding Lakh MIDI Dataset (LMD), one of the largest publicly available sources in the symbolic music domain. To find and evaluate the best retrieval method for duplicated data, we employed the Clean MIDI subset of the LMD as a benchmark test set, in which different versions of the same songs are grouped together. We first evaluated rule-based approaches and previous symbolic music retrieval models for de-duplication and also investigated with a contrastive learning-based BERT model with various augmentations to find duplicate files. As a result, we propose three different versions of the filtered list of LMD, which filters out at least 38,134 samples in the most conservative settings among 178,561 files.

Detected Duplicates in the LMD

Query from Clean subset of LMD, using the conservative threshold of ≥ 0.99 (CAugBERT U CLaMP). Currently, the top 50 queries with the highest number of duplicates can be played in this page. You can scroll left and right in the MIDI visualizer.

Survived file (largest note count)

Please select an artist - song sample.

Detected duplicate file

Please select a detected duplicate sample.

Duplicate Types in the Paper

Hard Duplicates

We define hard duplication as files with exactly same shared parts of arrangements with minor differences. The differences between these hard duplicates include instrument mapping, instrument order, tempo, offset of the beginning, file length, missing tracks, or alterations at the note level—such as pitch, duration, or velocity. Melodic variations may also appear, such as added ornamentation or changes in the number of chord tones played by a specific instrument.

4/46f0bec94ec9468e7fa0d9162832d185.mid

6/656d7bde3a2cf7dd2531856f821bb9d2.mid

Soft Duplicates

In soft duplication, MIDI files preserve essential musical elements such as melody, harmony, ornaments, and instrumentation, but differ in arrangement style, reflecting the diverse skills and styles of individual arrangers. Such duplication includes cases where the core melody remains unchanged or similar, but accompaniment styles (e.g., arpeggio, waltz), pitch ranges, key transpositions, and overall structure vary considerably. In cases of extremely different arrangements, these variations might lead a listener to perceive them as distinct songs.

d/dd5c535dbbf10b64763ec057cf14a544.mid

d/d1f3909d044439d013515b99fa6db5ca.mid

Non-duplicate Examples

Here, we present non-duplicate examples from 100 randomly selected queries in LMD-full, as described in Section 7 of the paper.

Similar Examples

b/b706dec12991e3680a65d5f689848ead.mid

b/bbabcfcac516c0b021799807aa8ddacf.mid

Irrelevant Examples

Case #1: Similar Genre

3/3c67e89d92755cae483e97bd5992c07a.mid

5/577bb0d0b7e67f10a12d09aa3a8d6392.mid

Case #2: Baroque Keyboard Piece

5/51b73bc08704282bbf655862af73b038.mid

4/4093667708ad8862fc26fffab022a8f4.mid

Case #3: Hymn-like

3/361444618e940c496fe23c9e07cc9bac.mid

3/3a563a18fb3491e665b13fdbd39103df.mid

Case #4: Single Instrument Short Segment

f/fce8d7c34280be172b478c14f1f50707.mid

a/a10822111fa9259072bcf24e097a5a60.mid