Reveal Processing

This process evaluates the metadata and/or text of emails to provide the action taken when creating the email (i.e. Reply All, Forward, etc.) and to group emails into their logical threads. Once in their logical thread, additional metadata is created of the total number of items within the thread, and a subset of emails in an email thread that contain all text within the email thread known as inclusive emails.

Email threading can help reduce the amount of time spent reviewing emails by viewing the email conversation as a thread, and/or the inclusive emails within the email thread. Email threading also helps with tagging or coding consistency by using rules to code all emails within a thread consistently.

When evaluating emails for threading the following process is applied to original emails (based on MD5 Hash) in the following order. If an email does not meet the criteria in one process, it will move to the next process. The process applied to a thread is captured in the database to determine which process was applied to the email thread. These values are propagated down to duplicates if other means of deduplication are used at export time (i.e. Custodian Level or No Deduplication).

This process evaluates the text of non-email items within a collection to identify files that are very similar in text but not exact duplicate files based on an 80% similarity value. Files are grouped based on a principle or comparison file which is used as the basis of grouping textually similar items together for analysis. This functionality can be very helpful for finding multiple very similar forms of a marketing or sales presentation, quarterly pricing files, etc.

When evaluating files for near duplicate identification the following process is applied to original files (based on MD5 Hash). These values are propagated down to duplicates if other means of deduplication are used at export time (i.e. Custodian Level or No Deduplication).

This process evaluates all original items that contain text within a defined scope to identify patterns within the text. Regular expressions provide a powerful, flexible, and efficient method for processing text. The extensive pattern-matching notation of regular expressions enables you to quickly parse large amounts of text to find specific character patterns; to validate text to ensure that it matches a predefined pattern (such as an e-mail address); and to add the extracted strings to a collection to generate a report. Regular expressions are an indispensable tool for many applications that deal with strings or that parse large blocks of text.

The Reveal Discovery Manager includes the following system-defined Patterns:

When used as part of a Pattern Job, these Patterns will identify files that contain these items within the text. Along with the system defined Patterns, users can create Patterns using the .NET regular expression language. A quick reference guide to the .NET regular expression language can be found by clicking on this link.

There are also many products on the market that help users create .NET regular expression searches which can be found by doing a quick search on the Internet. One that is free and web based is Regex Storm, which can be accessed by clicking this link. While there may be many different tools out on the Internet, this tool is free and has examples of how each .NET regular expression can be used.

"Latent Semantic Indexing (LSI) is an indexing and retrieval method that uses a mathematical technique called Singular value decomposition (SVD) to identify patterns in the relationships between the terms and concepts contained in an unstructured collection of text. LSI is based on the principle that words that are used in the same contexts tend to have similar meanings. A key feature of LSI is its ability to extract the conceptual content of a body of text by establishing associations between those terms that occur in similar contexts.” (Deerwester et al, 1988)

One great advantage of LSI is that it is a strictly mathematical approach, with no insight into the meaning of the files or words it analyzes. This makes it a powerful, generic technique able to Index any cohesive file collection in any language. It can be used in conjunction with a regular keyword search, or in place of one, with superior results.

Latent semantic indexing looks at patterns of word distribution (specifically, word co-occurrence) across a set of files to determine cohesiveness. In addition to recording which keywords a file contains, the method examines the file collection to see which other files contain some of those same words. LSI considers files that have many words in common to be semantically close, and ones with few words in common to be semantically distant. This simple method correlates surprisingly well with how a human being, looking at content, might classify a file collection. Although the LSI algorithm doesn't understand anything about what the words mean, the patterns it notices can make it seem astonishingly intelligent.

When searching an LSI-indexed database, or a Conceptual Index, the search engine looks at similarity values it has calculated for every content word, and returns the files that it thinks best fit the query. Because two files may be semantically very close even if they do not share a particular keyword, LSI does not require an exact match to return useful results. Where a plain keyword search will fail if there is no exact match, LSI will often return relevant files that don't contain the keyword at all.

When building an index, a training set of files is used to build or train the index, into which the files are mapped. While this mathematical concept space is many-dimensional, it can be thought of as a three-dimensional space such as a cube or a room as seen below. Each dot below represents a training file. The training set of files enables the Conceptual Index to map searchable files into this space. Files that are closer together in this concept space are inherently more conceptually similar than files that are further apart from each other. All files that have text are searchable, but not all files will be used for training purposes. To see which files are included as training files, please see Appendix B below.

To create an Analytics Job, select the Analytics Job Scope and click the Create Analytics Job button in the Analytics Module Ribbon. Then select the Analytics Job Type(s) desired from among Pattern Matching, Near Duplicate Identification, Email Threading and Conceptual Indexing in the New Analytics Job window.

The Reveal Discovery Manager will first isolate the original files within the provided scope(s). Secondly the system will analyze the files to determine if the Patterns that exist in Pattern Management for the project exist in these files. Because of this, it is possible that a scope will yield 0 results when using the provided scope, thus no Analytics Job will be created. If a Pattern is created by the user after a scope has already been used as an Analytics Job, the system will recognize this, and will rerun the newly added Pattern(s) across the scope of data. This allows user created Patterns to be backwards compatible across a given project, but does require that the user create new Analytics Jobs for previously used scopes.

After an Analytics Job is created it is distributed to the Reveal Discovery Agents for processing. To check if a Discovery Agent is running, go to the Environment Module, choose the Processing Machines tab, click the Refresh button, and make sure that one or more machines have TRUE in the Active column. If the all have the value of FALSE, the Discovery Agents will need to be opened on one or all machines. To see more about the Discovery Agents, please see the Environment Module section above. After the Discovery Agents are open, the Analytics Job can be monitored in the Environment, Projects (Processing Jobs tab) or the Analytics Modules. In each Module, the Refresh button should be used, and information regarding these jobs will be updated.

When creating an Analytics Job, the Reveal Discovery Manager will use the provided scope to find all original items that have not been processed for the Analytics Job Type. Because of this it may be necessary at some point to start over for a given scope. Each Analytics Job has a purge button (Delete Analytics Values) within the Analytics Ribbon which will facilitate this process. When selected, all values that have been stored within the project will be removed from the database. If the data are purged for one or all Analytics Job Types, a new Analytics Job should be created for one or all Analytics Job Types.