From LegalTech NY 2010: Defensible document review

This post is one of several summarizing our coverage of LegalTech New York 2010.  For our other posts click here.

Laura Kibbe (Senior Vice President at Epiq Systems) led a spirited panel titled “Achieving a Cost-Efficient, Defensible Document Review”.  Panelists included Vince Neicho (Litigation Support Manager at Allen & Overy) and David Kessler (partner at Drinker Biddle).

As the panelists stated, document review is more than the mechanical process of marking and tagging documents.  The goal on every project should be to facilitate true knowledge transfer between the review room and the legal team.   Epiq Systems is a bit different in this regard because is not a “let’s just fill the seats and click” body shop.  Its reviewers must have the requisite substantive experience to work on a project because Epiq believes you need that to achieve  the best balance of efficiency and knowledge transfer ability such that the legal team is aided in its fact development efforts on a daily basis by the review team, with defensible results.

And QC is imperative.  Laura pointed out that at Epiq Systems standard workflow consists of first level review by a team of 12 attorney reviewers daily.  The QC reviewer assigned to the team will receive a randomized batch comprised of 11%, 15%, 25% or 30% of the prior day’s review (exact percentages of QC review are set by you on a case by case basis). If more than 5 errors are found in any given reviewer’s batch, the entire batch is re‐reviewed, generally elevating total QC percentage above the stated baselines. Once QC is complete daily, a second randomized sample of 2‐3% is batched for attorney review. This third level review batch will contain documents that have gone through the Epiq QC review and those that have not. For purposes of scheduling, they will not proceed with our review unless that firm QC and feedback is complete, thus preventing miscommunication or protocol ambiguity from skewing the review or heading it too far off course.

As David Kessler pointed out, this allows for the legal team to be actively engaged, especially at the beginning of the review, where issues and the protocol are so susceptible to change. 

And Epiq does not limit itself to its own platform, Documatrix.  It uses all available features of whatever review platform the client requests to efficiently organize workflows and ensure quality.  That is rare for most contract attorneys who know that on most reviews we seldom use 1/5 of the features on a given software.  Developers are putting all kinds of helpful bells & whistles on their latest versions.  Why can’t we take advantage of them?

The panelists further discussed the fact that whatever the review tool you need to further refine the workflow by assigning batches comprised of all near duplicates and e‐mail threads, making coding decisions consistent and a more comprehensive understanding of the subject matter possible. If the firm agrees, reviewers should be permitted to batch code like documents, increasing speed and consistency in review.  Epiq Systems also utilizes, where available, key word highlighting and clustering to facilitate the review.   Note: we saw this demonstrated by a vendor at the show.  For example, you can work with the law firm to develop a presumptively privilege search ontology that highlights or identifies all documents containing certain characteristics (e.g., firm domain names, counsel, etc.) and isolate those documents into one workflow so that they are directed to a specific group if reviewers specially trained in that area. Similarly, you can design the workflow by substantive concept so that specifically trained reviewers will see consistent subject matter.  There is tremendous technology out there.  On a project you need to constantly explore new functionality to increase consistency and improve efficiency.

Side note and a blatent plug for Epiq Systems:  Documatrix allows you to prioritize review order by relevancy ranking to front load the review queue with the most likely responsive documents first.  Of course, more traditional, linear review workflows are also available and an appropriate workflow can be structured.

But the best thing is to have a defensibility binder and a validation protocol.  This means that in addition to the QC procedures laid out above, you need to performs a pre‐production QC validation which is basically identifying miscoded documents, creating queries that will find documents miscoded for technical reasons (such as a missing decision on a document), targeted queries to find documents more likely to be coded incorrectly – searching for common mistakes found throughout the course of review, etc.; and privilege queries that identify documents more likely than the general population to contain privileged materials. This validation protocol will be presented to counsel for approval and included in the defensibility binder. These documents are sent back into review and re‐reviewed as necessary, providing a higher level of defensibility to the overall process, and ensuring that documents are not just QC’d, but truly ready for production.

You need, therefore, to memorialize all decisions made and record them as part of the project “defensibility” binder.

The panelists also stressed the utilization of review room metrics through an online dashboard accessible to the project team lead and the legal team.   On a proper metrics dashboard, you can see everything from productivity by reviewer, to calls by reviewer, assess error rates for significance and project timelines for completion.  The panel said that by using such tools, with the involvement of counsel, as a team, you manage the review and spot potential problems (e.g., reviewer 1’s responsiveness rates are significantly off the average).  By drilling down on each of these metrics on a daily basis, the team can decide how to manage each reviewer or the entire team. Specific accuracy metrics are also available by reviewer so that to the extent the legal team prefers a bright line measure, that can be supported as well.

All of this is supportable and realistic and not “too much” because it yields an efficient, cost effective, defensible result.