Keyword Search Debate

We all agree.  Keyword Search technology is ineffective right?  Blair & Maron said it in 1985 and TREC said it again this year   .

Magistrate Judge John Facciola of the D.C. District Court said it twice last year (U.S. v. O’Keefe, 537 F. Supp. 2d 14 (D.D.C. 2008) and Equity Analytics v. Lundin, No. 1:2007cv2033 (D.D.C. March 7, 2008)) and Magistrate Judge Paul Grimm repeated it emphatically in Victor Stanley, Inc. v. Creative Pipe, Inc. (Civil Action No. MJG-06-2662 (D. Md. May 29, 2008)).

But know we have word that TREC is going to embark on an additional study of the publically available ENRON documents. (Which raises a red flag in my mind: the ENRON documents I’ve seen were predominately non-email which had been converted from their original format to text and lost all headers and attatchments. Hopefully TREC has access to the original document corpus) and the EDRM group has established an EDRM Search Group with it’s own  EDRM Search Working Content Wiki  .

So why are we spending so much time talking about it still?  Judge Grimm gave us two excellent resources for evaluating any search used in ESI  production: The Sedona Conference’s Best Practices Commentary on the Use of Search and Information Retrieval Methods in E-Discovery and the federal government’s Text REtrieval Conference (“TREC”) Legal Track initiative.

 Well the problem is one that lit support people have known for years now and Ron Friedmann  pointed out in a blog post in 2007.  “Many lawyers appears honestly to believe that human review is accurate, the “gold standard” for document review. “Honestly held” and “right” can diverge. I, for one, have never seen data to support the commonly accepted “gold standard.”

Craig Ball pointed out this dillema last year in a blog post.  “Keyword search is deemed “good enough” for identifying responsive electronically stored information; yet when privilege is on the line, lawyers insist on page-by-page review. It’s a tacit recognition that keyword search is a blunt instrument — …”

Yet Judge Grimm allows this “blunt instrument” to be used if both sides agree to do so.  As he said in Victor Stanley, “Use of search and information retrieval methodology, for the purpose of identifying and withholding privileged or work product protected information from production, requires the utmost care in selecting methodology that is appropriate for the task … [and] careful advance planning by persons qualified to design effective search methodology.”

Or as Craig puts it, “Victor Stanley preserves the litigants’ inalienable right to be wrong, so long as everyone agrees that wrong is right. It’s a Faustian bargain, but one permitting cases to move forward by simply ignoring pesky questions concerning the integrity and completeness of electronic discovery.”

So we continue to discuss using agreed upon keyword search technology even when we all know it is inadequate.  Faustian bargain indeed.   Which brings us back once again to education.  Until attorneys learn to use the proper technological tools we will continue to have poor results. Studying keyword search technology further isn’t going to change that result.


No comments yet

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: