Maine State Police - Keyword Search & Export EnScript

Maine State Police Keyword Search and export EnScript – v1.0
April 28, 2009
EnCase v 6.13

This post is an update to a previous post here. This is an updated EnScript with new features and an official v1.0 project release. The following description is from the instruction document in the zip file linked below.

1. Introduction

This EnScript was born out of a concept from Sgt. Glenn Lang with the Maine State Police. Sgt. Lang needed a way to quickly and effectively identify known multimedia files that are commonly possessed by persons trafficking and possessing child pornography.

2. How it works

Sgt. Lang had an application named "harvester" written for the purposes of extracting a 10-byte string of hex values from inside any file. The thought was that instead of relying on file headers, to instead grab 10 hex values from the middle of a known video file or graphic file as a "mini" known signature for that specific file. The "harvester" program creates a simple text file with one 10-byte keyword per line (CRLF delimited). Investigators can use the "harvester" program to scan all their known bad media files and extract a mini "signature" for each of them, placing them into a small text file.

This EnScript was designed to read the text file created by "harvester" (or by any other means) and then begins searching the disk for those keywords. TheEnScript was designed to work in two ways. The first way is to search unallocated space, the second it to search all areas of the disk(s) (allocated files and unallocated clusters).

3. Configuration

When you start the EnScript, the investigator is presented with the following initial screen :



A. The first option asks for the text file where the keywords are stored. This text file should be a simple ASCII text file, one 10-byte keyword per line, in the format of:



B. This is the offset into the known file where the keyword was harvested from. The EnScript will search and export using this offset.

C. Total size of export - When a keyword is found, the EnScript will back up “x” bytes, as dictated by the offset value described above. The EnScript will then export from that offset (presumably the beginning of the file) for a total of “x” megabytes, as indicated by this value.

D. File Extension - When a keyword is found and the data around the keyword is exported (as described above), the exported data will be placed into the case default export folder and given the extension as indicated by this value. This is so if the investigator is searching for movie files and the data is carved from unallocated space into the default export folder, the investigator can quickly double click and use a viewer, such as VLC, to view the contents.

E. Comprehensive search - This checkbox dictates how the EnScript will search for the keywords. The normal built-in keyword search process in EnCase searches every byte of the disk (or unallocated cluster object). This EnScript, in an effort to speed this process up, by default, will search for the keyword only at the specified offset of each cluster, then move to the next cluster and look at the specific offset in that cluster and then move again to the next cluster. A typical Windows installation uses the NTFS file system and defaults to a cluster size of 4096 bytes (8 sectors per cluster). This means you are only searching 10 bytes out of those 4096 bytes, effectively only .2% of a cluster. The purpose of this is speed. If you think about how a file will always be saved on disk starting at a cluster boundary, then the keyword your looking for will always be found at the offset you specified in option B into a cluster. Searching the other areas of the cluster is unnecessary.

By checking this box (comprehensive search), the EnScript will instead search every sector. The reason for this option is in case the target drive had some files you are searching for and then the volume was formatted at some point in the past. The formatting process may inadvertently either change the number of sectors per cluster (i.e. was FAT, now is NTFS) or the boundaries of the volume have changed. Therefore, by selecting this option, the EnScript will search for the 10-byte keyword at a specific place in the sector, then move to the next sector and search again at the specific offset. This will increase the amount of time it takes to complete the search, but is still faster than a traditional keyword search where every byte of every sector is searched.

F. Bookmark - This will cause the specific keyword hit to be bookmarked when it found. Check in the bookmarks folder for a folder named “Found keywords in Unallocated – DATE & TIME”.

G. Search all files - The default is for this EnScript to only search unallocated space for the supplied keywords. Checking this box forces the EnScript to search unallocated, as well as every file in the case (allocated files).

4. Console - Real-time information is displayed in the console as the EnScript is running. If a keyword is found, the offset as well as full path of the file of where it was found is displayed.



5. Alert Sound - The EnScript will automatically check for the presence of a .wav file named "alarm.wav" in the “C:\Program Files\EnCase6\” root folder. If this file exists, this .wav file will be played every time a keyword is found during the search process. If the .wav file does not exist, the alert sound function will be skipped, but real-time information is still displayed in the console tab of EnCase.

Project information:
Sgt. Glenn Lang, Maine State Police
glang (at) mcctf.org

Download Here

Keyword List

Video demonstration #1
Video Demonstration #2

Filter to remove duplicates for export

A reader emailed me about needing a solution to remove some duplicates to then export some files. The scenario was that a keyword search was run and thousands of files were found that were responsive to the keywords. The reader tagged the files and then found that some of the files were duplicates, even though they were named or located in different places on the evidence. So to reduce the number of files that needed to be exported, he needed a way to remove the duplicate files.

EnCase comes with a standard filter that is named "remove duplicates by hash". This filter does exactly what he needed, except it did it against all files. He only wanted to remove the duplicates from the selected files. By adding one quick line, the following filter will remove duplicate files, based on the hash value, of the SELECTED files. So if you have 100 selected files and some of those files have the same hash value and then run this filter, what will be left will be only unique selected files.

You can create a new filter and paste the following code:

--------cut here------------

class MainClass {
NameListClass HashList;
bool UserCancel;

MainClass() :
HashList()
{
if (SystemClass::CANCEL ==
SystemClass::Message(SystemClass::ICONINFORMATION |
SystemClass::MBOKCANCEL, "Unique Files By Hash",
"Note:\nFiles must be hashed prior to running this filter."))
UserCancel = true;
}
bool Main(EntryClass entry) {

if (UserCancel)
return false;

if (entry.IsSelected()){
HashClass hash = entry.HashValue();
if (!HashList.Find(hash))
new NameListClass(HashList, hash);
else
return false;

return true;
}
else
return false;
}
}
------------------ cut here----------------

EnScript to obtain DHCP and Static IP Address information

Per a reader's request, here is an EnScript that will recurse through all evidence in a case and parse the SYSTEM registry hive located in the \system32\config folder. It will then display any DHCP or static IP address information for all the interfaces found in the SYSTEM registry hive.

The EnScript will also parse any SYSTEM registry hives found in the XP System Restore Points (System Volume Information Folder - "_REGISTRY_MACHINE_SYSTEM") and display those as well. This EnScript is compatible with Windows 2000/XP/Vista/2003.

All output is in the console tab for review. Example of output:
Reading file: Case 1\Fiske\C\System Volume Information\_restore{F7B7E177-A202-4882-ADC2-D0A88A676F63}\RP3\snapshot\_REGISTRY_MACHINE_SYSTEM

Interface GUID: {FA987DAF-1C7E-40E2-B570-8EBF1FFFA371}
IPAddress: 0.0.0.0
DhcpServer: 192.168.1.1
Lease: 86400 seconds
LeaseObtainedTime: 08/22/03 08:25:45PM
LeaseTerminatesTime: 08/23/03 08:25:45PM
DhcpIPAddress: 192.168.1.101

Reading file: Case 1\Fiske\C\System Volume Information\_restore{F7B7E177-A202-4882-ADC2-D0A88A676F63}\RP4\snapshot\_REGISTRY_MACHINE_SYSTEM

Interface GUID: {2AF8F12B-22F6-4FAE-974D-564BA481D3FF}
IPAddress: 0.0.0.0

Interface GUID: {FA987DAF-1C7E-40E2-B570-8EBF1FFFA371}
IPAddress: 0.0.0.0
DhcpServer: 67.21.13.74
Lease: 43200 seconds
LeaseObtainedTime: 10/08/03 08:56:49AM
LeaseTerminatesTime: 10/08/03 08:56:49PM
DhcpIPAddress: 68.66.201.16

Download Here

Count unique domains in email list

This EnScript was written by request for someone doing an email spam case and he needed to parse a large list of email addresses and then extract only the unique domain names.

So in this case, he had a very large ASCII file containing thousands and thousands of email addresses, some of which came from the same organization and had the same domain, but different email address. He needed a way to just create a list of just the unique domain names. This EnScript takes an ASCII file, with one email per line, line-delimited with a CRLF like this:

john@test.com
dave@test.com
steve@test.com
mike@mydomain.com
tom@mydomain.com
joe@mydomain.com
etc...etc...etc...



The output of the EnScript in the CONSOLE tab would be:

test.com (2)
mydomain.com (3)

This is a pretty specialized EnScript, but others may have a use for it as well.

Download Here

Recovering video files in unallocated space

Recently, Sgt. Glenn Lang from the Maine State Police contacted me regarding an EnScript request designed to export some data from keyword hits where he was searching for movie files in unallocated. Sgt. Lang is the ICAC coordinator and does a lot of child exploitation investigations. He has had great success in building some excellent GREP keywords to find movie files in unallocated.

The GREP keywords are usually characters that are located at various offsets inside the video files, not at the beginning. He needed a way to quickly export the suspected video files and view them.

By modifying the previous "export x bytes from a search hit" EnScript, I created an EnScript that will export x bytes in front of the keyword hit and then specify the total number of bytes to export:



It then saves the data into a file named after the original filename where the hit was found (usually unallocated) the search term, the offsets and then you can specify a extension for the export:



You can then double-click and use your registred viewer to view (vlc in this example).

Sgt. Lang has put together some basic videos demonstrating this technique and they can be viewed here:

Adding keywords and starting a search.wmv
Recovering Movies Located Using Harvester Key Words.wmv

Download GREP keyword list here (Import into EnCase Keyword tab)
Download EnScript here

Export files with selected search hits

, So this EnScript was a suggestion from a reader named Scott (you know who you are). The premise behind this is many times examiners are asked to run several keywords (sometimes hundreds) then export the files here the keyword were found and produce them for review.

This EnScript automates the export process by allowing you to select the search hits in the "Search Hit" tab and then running the EnScript. It will then go through all the selected search hits and export all the files that contain those search hits into folders named after the keyword. For example, if I searched for "lance", "mueller" and "lance mueller",and selected the root of each one of these search hit results in the Search Hit tab, a root folder named "Exported Search Hits - 04.07.09 07.24.29AM" will be created in the default export folder specified for the case. Inside this folder will be subfolder for each keyword: "lance", "mueller" and "lance mueller".



Inside each of these folders will be all the files that contain that specific keyword. An index file is created in the root folder that specifies the keyword, hit offset, original path of the file in the evidence and the new path of that file in the export folder tree.





In the example below, if you select the "enscript" and/or "\e\n\s\c\r\i\p\t" search hits, two folders would be created will all the files that contain those keywords.



A few comments:
1. Duplicates - If a keyword such as "lance mueller" is found in several locations in a particular file, it is only exported once into that specific keyword folder. If the file ALSO contains another keyword, then it will also be exported once into the folder for that keyword. If a keyword is found in multiple locations in the sam file, it is only exported once, but all the hits and offsets are referenced in the index. The last column will indicate the file was previous exported, but the hit offseet will reference the current hit.



2. File naming - The exported files have a number appended to the original filename to prevent multiple files that have the same name, but reside in different locations in the evidence to be exported into the same export folder and overwrite themselves. A number is places at the end of the name stem, before the extent ion. The original name and path is noted in the index file with the corresponding new name as it exists in the export folder.

3. GREP searches - keywords that are used that contain illegal directory name characters, i.e. /,\.:, etc. are stripped and replaced with a bullet: "·"
The original keyword is specified in the index file.

Download here