Duometer allows to efficiently identify near-duplicate pairs of documents in large collections of texts. It is written in Scala and implements a MinHash algorithm.
For example, to extract text from all files in ~/text-files and identify
those that have similar content, run:
./duometer -i ~/text-files -o text-files.duplicatesFor more information about how to use duometer see this tutorial.
- Efficiently finds pairs of documents that contain similar text.
- Automatically extracts text from files in a huge number of different formats (including HTML, PDF, Microsoft Office document formats and the OpenDocument format)
- Works well with very large collections of documents.
- Makes use of multiple CPU cores.
- The default settings should work well in most cases but you can
customize the duplicate detection for your purposes (
run
./duometer --helpfor a full set of options).
The only prerequisite is a Java runtime.
- Download the current version of the tool here.
- Extract the archive and go to
./bin. - Run
./duometer(on Linux and Mac) orduometer.bat(on Windows).
Download a package and run:
sudo dpkg -i duometer_0.1.3_all.debduometer is now installed and should be available as a shell command.
Duometer tries to use memory efficiently, however when using it to detect
duplicates in a large collection of documents, you should make sure that
enough memory is available to the JVM in which duometer is running. If you do
not know what your default settings are or do not want to change them, you can
easily override them by adding -J-Xmx argument when calling duometer. For
example, if you want to make 10GB of memory available to duometer, you should
run:
duometer -J-Xmx10G -i ~/text-files -o text-files.duplicatesDuometer uses sbt-native-packager.
You can build the tool from source by running the dist command in sbt
to create a .zip archive that can be run on any machine with Java installed.
Debian binary package can be created by executing debian:packageBin.
For background information about the algoritm see a relevant chapter in Manning and Schütze (2008) or read the original Broder (1997) paper.
Duometer uses Apache Tika to extract text from a huge number of different file types. For the full list of supported formats see here.
- Issue Tracker: https://github.com/pmandera/duometer/issues
- Source Code: https://github.com/pmandera/duometer
The tool was developed at Center for Reading Research, Ghent University by Paweł Mandera.
The project is licensed under the Apache License 2.0.
Broder, A. Z. (1997). On the resemblance and containment of documents. In Compression and Complexity of Sequences 1997. Proceedings (pp. 21–29). IEEE.
Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to information retrieval. New York: Cambridge University Press.