Find duplicate files

This post is not really "forensic" related but it may have use to others...

If you are like me, you have several hard drives laying around that have all sorts of "stuff" on them. Things you have acquired over time and save with the thought that the information, programs or files may come in handy someday in the future. Then after I fill that drive up, I end up upgrading to a larger drive, but I don't want to delete anything in case I need it! After reviewing several of these drives, I found I had lots of duplicate files scattered throughout the drives in various folders.

My solution to this was to write an EnScript so that I could preview all my drives at once and then the EnScript will hash all the files and list all the duplicates for me. I can then decide if I want to manually delete the dupes or not. It also lists how much space is wasted by having all the dupes.

Most of my drives are formatted using NTFS. The NT file system has a feature called hard links. This basically allows you to have multiple directory entries for the same file, but it only takes up the space of one file. This is because all the directory entries can point to the same MFT record and the same data. For example, you can have a file named "lance mueller.doc" in a folder named "e:\documents\", and then have a file named "john smith.doc" in a folder named "e:\old stuff". There are two separate directory entries and the names don't even have to match, but they can point to the same exact MFT record number, and therefore to the same exact data, reducing the amount of spaces being wasted by having duplicates. Opening one of the files and editing it affects both directory entires.

I took the below listed EnScript one step further and actually have the EnScript locate an original file, then for all duplicates, it deletes the dupes and then creates a hardlink to the original file. This basically leaves the directory entry in place, but reduces the amount of space being wasted by pointing all the dupes to the same data as the original. I am not making that version of the EnScript available yet, but I am posting the EnScript that will list all the dupes in the console pane of EnCase and then you can decide what to do with the dupes.


Download Here