Designed for Datasets of Any Size*

Its goal is to enable dataset analysis and optimization at any scale. Removing duplicate and redundant data improves the overall quality of a dataset, provides a more accurate representation of the data available for training, and reduces unexpected issues during model training.

In summary:

• Small datasets: Fast processing with low memory usage.
• Large datasets: High performance while maintaining a very small memory footprint.

Install

pip install git+https://github.com/JuanForge/datasetforge.git

Examples commands

1 - duplicates

datasetforge duplicates \
    --input /data/0.0.0 \
    --phash-live \
    --phash-max-percent 10 \
    --phash-min-percent 2

---

The duplicates subcommand is used to analyze a cache previously generated by the index subcommand.

• --input specifies the directory containing the dataset(s).
• --phash-live enables live pHash filtering mode. This mode allows the use of pHash-specific filtering options instead of displaying only the top-k results.
• --phash-max-percent 10 sets the maximum allowed pHash difference. Image pairs with a pHash difference greater than 10% will not be displayed.
• --phash-min-percent 2 is optional and defaults to 0. Setting it to 2 hides image pairs with a pHash difference lower than 2%.

With these settings, the command will only display image pairs whose pHash difference is between 2% and 10%.

This range is useful for detecting images that are visually similar but have undergone strong recompression, format conversion, resizing, or quality degradation.

For general duplicate detection, it is recommended to keep --phash-min-percent at its default value (0) or omit it entirely. This allows all pHash differences up to the value defined by --phash-max-percent to be detected, including exact or near-exact duplicates.

Comparison

• n = dataset size (number of indexed images)
• k = --top-k value
• t = --threads value

Mode	Comparisons	Disk reads	Memory usage
Default	$\frac{n(n-1)}{2}$	O(n)	$O(n + k \times t)$

pHash Explained Simply

The similarity percentage reported by the tool should be interpreted with care. Perceptual hashing (pHash) does not compare images the way a human does, especially when using different hash sizes.

In practice, using a larger value for --phash-bits (for example, 262144) allows the algorithm to preserve significantly more visual information. As a result, pHash can perform a much finer comparison, often producing substantially higher similarity scores than smaller hash sizes such as 1024 or 4096.

Conversely, a smaller --phash-bits value effectively reduces the amount of visual information available to the algorithm. With fewer features to compare, similarity scores may be noticeably lower.

Therefore, it is completely normal for the same pair of images to receive much higher similarity percentages—sometimes nearly twice as high in the top-k results—when using a larger --phash-bits value. This does not necessarily mean the images have become more similar; rather, the higher-resolution perceptual hash enables a more precise comparison.

Command-Line Arguments

--phash-bits

Controls the resolution of the perceptual hash used during comparison.

• Default: 64
• Higher values preserve more visual information, allowing finer comparisons.
• Lower values are faster and use less memory but may miss subtle similarities.

Examples:

datasetforge duplicates --phash-bits 4096
datasetforge duplicates --phash-bits 262144

Note Higher values often produce higher similarity percentages because the algorithm has more information available for comparison. This does not mean the images themselves are more similar.

--input ( duplicates )

Specifies one or more dataset directories. Indexes are generated automatically, making this suitable for small datasets where pre-indexing is unnecessary.

Optimizer

Operation	Performance
--phash-optimizer default	Speed x0.00
--phash-optimizer numpy	Speed x3.50
--phash-optimizer xor	Speed x23.5

---

I strongly recommend using --phash-local-xor for significantly better performance. It calculates the Hamming distance directly using a bitwise XOR operation, making it especially efficient on 64-bit processors. It is not enabled by default because I cannot guarantee that this implementation will always produce exactly the same behavior as the original distance operation.

--phash-local-numpy uses NumPy to compare the individual hash elements instead. While it provides lower performance than the XOR implementation, it more closely follows the original element-wise distance calculation and is therefore the safer choice when exact compatibility is preferred.

The optimizer has nothing to do with the one used for model training; it’s simply a different implementation for calculating the distance more efficiently.

Python

Developed with Python 3.12.3

---

---

• Tested on a dataset of 30k images: stays under 300 MB of memory (PSS + main), using 20 workers and XOR acceleration, and completes in 20 seconds (excluding indexing).

---

Windows System

Indirect Windows support is possible through WSL2, which allows Linux code to run on a Windows system.

WARNING

Warning

Experimental Project

This project is currently under active development and should be considered experimental. Breaking changes, bugs, and unexpected behavior may occur between releases.

At this time, the project has only been tested on Linux. While some features may work on other operating systems, compatibility is not guaranteed until they have been officially tested and validated.

If you encounter any issues, please report them through the Issues page.