finddups is a command line tool to find and remove duplicate files and
directory trees for Windows, macOS and Linux. It aims to be as efficient as
possible, only reading as much of each file as required to determine if it is
unique.
It considers only the contents of files and directories and the destination of symbolic links to determine duplicates. Filesystem metadata such as permissions, ACLs, MAC times and xattrs are ignored.
Certain "unimportant" files that are created automatically by operating
systems, file managers or text editors are ignored for the purpose of
comparing directories, such as .DS_Store, Thumbs.db or ._* files. For
a complete list see ignore-rules.js.
-
Install Node.js version 8.7 or greater.
-
Install the package via
npm:$ npm install -g finddups
Run it and pass it one or more files or directories to scan for duplicates.
Once finished, it will present a list with the biggest duplicate sets first.
You can step through the list with n and p, action the current duplicate
set with 1,2,3..etc and D, or quit with q.
Example:
$ finddups ~/my_stuff /media/some_drive ~/some_file.doc
Scanning /home/bob/my_stuff
Scanning /media/some_drive
Scanning /home/bob/some_file.doc
Found 378,082 files, 941.79 GB
Reading file data of potential duplicates
1.64% of 15.98 GB, 52.73 MB/s, ETA 00:04:58
21.73% of 15.89 GB, 62.79 MB/s, ETA 00:03:18
...
83.65% of 15.76 GB, 73.25 MB/s, ETA 00:00:35
100.00% of 15.76 GB, 76.57 MB/s, ETA 00:00:00
Found 441 duplicate sets, 7.82 GB duplicated
1/441: dir 10945 (3 copies, 734.82 MB duplicated)
Please select an option:
1: Keep only "/home/bob/my_stuff/some dir/Foo Photos"
2: Keep only "/media/some_drive/blah/Photos/Foo Photos"
3: Keep only "/media/some_drive/Desktop/Old Stuff/Foo Photos"
D: Delete ALL
n: Next duplicate
p: Previous duplicate
q: Quit
> █
finddups is a Node.js project using Babel,
Flow and Prettier and has no
runtime dependencies. Sources are in src/ and are compiled into dist/
with babel src --out-dir dist --source-maps. The main entry point is
dist/main.js.