Ron Friedmann In the EDiscovery Zone

Browning Marean and I continue our interviews of e-discovery luminaries in a discussion with Ron Friedmann on everything from changes in the LPO landscape to our current dialogue with Herb Roitblatt on the best methods of searching ESI.

This interview, as well as all our others in the series, can be found at the site of our gracious host, TechLaw Solutions.

ED Search Discussion Continues

The dialogue on search technology continues with the following post from yesterday on Ron Friedmanns blog

Choice of Concept Search Tool in e-Discovery May Matter Less Then You Think [ Litigation Support / e-Discovery ] — Ron @ 1:16 pm

Tom O’Connor and I recently wrote a joint blog post about concept search software for e-discovery. Subsequently, we received comments from Herb Roitblat of Orcatec, an expert in information management, data mining, statistics, and eDiscovery processes. I share his comments here. 

Tom posted at his docNative Paradigm Blog Herb’s comments on Xerox CategoriX and Musings on the Best Approach to EDD Search (29 Oct 2009) by Tom and Ron:

• Herb Roitblat on ED Searches (12 Nov 2009)
• Herb Roitblat On Search: Part 2 (23 Nov 2009)

I publish here, with permission, additional comments from Herb, who wrote these in response to a message I sent him with my “take aways” from his first comments.

Summary
My summary and interpretation of Herb’s comments below and in the posts at Tom’s blog is that while concept search is a useful tool for e-discovery, the selection of the specific “flavor” of concept search tool matters less than smart application of it. Tool selection needs to be case specific because a “bake-off” among concept search tools only tells you how well a tool does against a specific set of documents. Since it’s not economically feasible to use multiple tools per case, you need to make a reasonable tool selection at the outset of a the case. As important, you need a reasonable and defensible process (which means documenting tool selection and process). The reasonableness standard depends on the stakes of the case.

Herb and Ron Exchange by E-Mail

Ron: So it sounds like what you are saying is that the difference in e-discovery concept search tools is probably overwhelmed by differences in document sets and in process / control.

Herb: I agree with this, but it has to be said carefully. Clothing does not make the man and high-powered tools do not make the builder, but they do help a good builder do better work. No matter how good your tools are, if they are not used well, you get a questionable result.

Ron: Concept search is not a magic bullet but helps expand the universe of documents to consider because it finds docs with words you would not otherwise think of as search terms.

Herb: It helps you think, but it is not a substitute for thinking. It is, as you say, not a magic bullet, just an amplifier.

Ron: Concept search can also help speed review by clustering similar documents.

Herb: Concept search expands queries to return results that are the best match to the expanded query. Thus, the top results are those that best match the query term and its context. (See green search on Truevert.com for an example, search for meat and get organic meat, not Omaha Steaks). There the context is given by green documents.

Ron: I back away from my initial assertion of the need to use multiple tools. I argued that to spur thinking among EDD professionals. Upon further reflection, what I really meant to say is that lawyers should focus more on industrial processes and controls, statistics, and metrics than on software features.

Herb: That’s what I think.

Ron: So that means we have no magic bullets. The legal profession has hard work ahead to industrialize its processes.

Herb: It’s actually not that hard. You just have to be thoughtful about what you are doing. It is not even terribly burdensome if you are realistic about the levels of accuracy that you can really achieve (see below).

Ron: We still don’t seem to have an objective standard by which to judge if a process is ‘good enough’.

Herb: There are lots of ways of deciding whether a process is good enough and lawyers are used to making reasonableness judgments and arguing about them. What are the consequences of different types of errors (e.g., retrieving too many documents, retrieving too few)?

Scientists, by tradition, usually use a standard of .95 confidence. For example, if two treatments are different with 95% confidence, then we accept them as different. That does not tell us how different they are or that the difference is practically important or useful, only that the difference is statistically significant. Scientists often report higher confidence levels than that, but the minimum is usually .95. That tradition has worked well in science where subsequent research can correct the relatively few times when the difference does not really exist, but resulted from sampling (luck of the draw).

As an analogy, if you play slot machines, the things return only about 95 – 98% of the money that gets pumped into them, but that does not mean that some people don’t actually win large amounts. It happens sometimes. The luck of the draw usually returns less than you put in, but sometimes it returns more.

Back to good enough. Engineers typically use confidence levels to tell them how well to build a bridge. They consider the consequences of different kinds of failure (think of the Tacoma Narrows Bridge). NASA uses confidence levels to determine the quality of their systems. Where the consequences are severe, they require higher confidence.

In eDiscovery, we are familiar with proportionality arguments and the like for determining things like cost shifting. The same thing applies here. A bet the company litigation may merit a higher level of confidence than a run of the mill litigation. Different types of errors may be weighted differently depending on the consequences of that kind of error.

None of this is hard nor does it require very much mathematical background. I published some tables a while back showing how many documents you should sample if you want to achieve a certain level of confidence and you are willing to accept the possibility of missing a certain proportion of responsive documents.

As I think I’ve said, I think that another part of reasonableness is transparency. Be able to describe what you did. A scientific publication is intended to describe enough of the methodology so that another scientist can replicate the observations. I don’t think that you necessarily have to publish to the other side what you did, but you should be able to provide that information if required (think Victor Stanley).

Herb Roitblat On Search: Part 2

Tom, your comments about anyone trying to evaluate ED options are well taken.

 How would you evaluate based on the document sets?  You know exactly what the documents were that were used last year by TREC legal, but how does that help?    I think that the things you need to know include how much variability there is among the documents.  How similar are, in this case, the responsive and nonresponsive documents? I’m not sure how your would evaluate that, but that is a variable that has a large impact.

Distinguishing news stories about sports from those about the stock market is easier than distinguishing stories about NYSE from those about NASDAQ.  But how do you measure that in a case like this?

 Many systems use the same underlying tools (WordNet, Lucene, dtSearch), so, as you say, there may not be big differences among them.  But so what?  From an academic point of view that would be disappointing, but from a pragmatic eDiscovery point of view it does not much matter.  They could still differ in text extractors, user interface, and in other tools they provide.  They may differ in the completeness of extracting text from attachments, but once it is extracted, the fact that it came from an attachment does not matter to search.

 Using OCR data (as TREC did), though, does make a difference.  The size of the vocabulary explodes with OCR and this can affect results, especially categorization results.

 I don’t see what the unreliability of human review has to do with the value of concept searching.  It seems like a non-sequitur to me.  The TREC results on Boolean searching are somewhat misleading.  The Boolean searches in earlier years were conducted by people who had had years of experience on these data.  If you know enough, you can find anything with Boolean searches. The 2008 TREC results found that H5 could produce the highest accuracy ever seen in TREC legal with their methodology.  From the 2008 summary report, Figure 1 would seem to show that the Boolean run left behind many documents that were found by other systems (http://trec.nist.gov/pubs/trec17/papers/LEGAL.OVERVIEW08.pdf). 

In Table 1, 5 systems ranked above the reference consensus Boolean run (xrefLo8C).  If your goal is to find more responsive documents, then all of the commercial concept search tools will do better than a Boolean tool, because they expand the query to include more documents (and they do other things).

 Both of you seem to think that it would be better to use multiple approaches to find responsive documents.  As I said, I think that this is a mistake and impractical, and is probably not the lesson to be learned from these studies.  If your goal is to be sure that you find ALL of the responsive documents, then produce all the documents.  Give two people the same documents to categorize and there will be some overlap.  It is logically true, that the set of documents found by either will be larger than the set found by one, but that does not mean that you get better results if you use the union of their decisions, you just get more.  If you review documents and I flip a coin, the union of our decisions will be more documents than you identified as responsive, but it won’t be any better a set.

 Any way, thanks for writing these thoughtful pieces.  I hope that people read them.

As an aside, we started a week or so ago to build our own categorizer.  It can organize documents into mutually exclusive categories (A or B, but not both) or into overlapping categories ( A or not A), (B or not B).  We achieve accuracies that are comparable or higher to those in the Categorix white paper.

  Precision            Recall

0.86                    0.86

0.67                    0.89

0.9                      0.88

0.99                    0.93

 The data are from Reuters news stories (and old TREC collection).  The first row is a set of 4 mutually exclusive categories, the rest are of the A / notA variety.  These articles were originally tagged by the Reuters editors.

  Topics

1. PERFORMANCE

MARKETS

GOVERNMENT/SOCIAL

None of the above

 2. ACCOUNTS/EARNINGS

Not ACCOUNTS/EARNINGS

 3. CORPORATE/INDUSTRIAL

not CORPORATE/INDUSTRIAL

 4. SPORTS

not SPORTS

 Regards, Herb

Herb Roitblat on ED Searches

Herb Roitblat, co-founder and a Principal at OrcaTec, responded to the joint ED Search post that Ron Friedmann and I put up last week. His response is lengthy but well worth reading so I am dividing it into two posts based on the two sections of the response itself. Part 1 appears below and I will post the second half tomorrow. 

Hi, guys.  Thanks for your stimulating discussion about search.  I have some thoughts to share. 

First, Ron, you do have the mathematical chops to understand the Categorix white paper.  It is not mathematical capabilities that limit your ability to understand what they are saying.  They just say it in a manner that you are not accustomed to (and one that is maybe designed to produce some mental dazzle).  They use probabilistic latent semantic analysis (PLSA), which is what Recommind uses, to help them to categorize documents.  Ultimately, what they want to do is to compute the probability that a document is in a given category.  There are lots of ways of doing it and the details of how they do it (the fact that they use PLSA) for example, is not particularly relevant to using it.

 They train it on a set of categorized documents and build a model.  Then they compare new documents with this model and classify the new documents into the most likely category according to the model.

 Then they use a really complex method to analyze their results.  I have no idea why they made it so complicated, but they did.  One result is that their reviewers did not agree as often as one would like with each other.

TREC and we have reported similar things.  The human decisions were not all that reliable.  If you want to measure the accuracy of some process, then you need a “truth” standard to compare it with.  Here, they had 5 truth standards, and reasonably good, but not outstanding, I think, success with each of them.  The different thresholds correspond to how biased the system is to call a document responsive.  The higher the threshold, the less likely the system is to call a document responsive.  Notice that recall (the proportion of responsive documents that were identified) goes down from Threshold = 0.5 to threshold = 0.95, just as one would expect.  Notice also that precision (the proportion of identified documents that were responsive) goes up.  How you use the threshold will depend on your case strategy, how sure do you want to be that you find all of the responsive documents (low threshold) vs. how sure do you want to be that you find only responsive documents (high threshold).  Then they do some other tests about consistency that don’t make a lot of sense to me and that seem a bit circular for what they are trying to accomplish.

 I don’t think that bakeoffs hold much value.  The results depend very much on the dataset and on the questions asked.  The TREC folks have found that the ranking of systems does not depend much on the people who do the “truth” relevancy judgments, but the absolute values do.  If product A scored above product B on a particular task, I would not be confident that they would also score above B on different set of data.  I think that the main thing that differs between tools is the ease with which you can accomplish your task.  Practically any tool can be used to select documents for review.  The review process itself is so sloppy that it trumps every other process.  If you could think of all of the right words to search for you could accomplish everything with keyword searching.  The problem, as you know, is that it is practically impossible to think of all the right stuff, so you need analytic tools to help.  The systems differ in how much help they provide and in the amount of effort it takes to get that help (think H5 and the long process they force you through).

 Thinking that litigants should run more than one eDiscovery tool is, I think, way off base.  It is already expensive and time consuming.  Expecting them to double this effort is simply unreasonable.  A much better and cheaper approach is to get them to evaluate their process using quality control methods.  This approach is cheap and effective.

 I think that what makes an approach defensible is to be able to describe what you have done and why you believe that the results are reliable.

Transparency and measurement are the keys there.

Joint Post With Ron Friedmann On The Best Approach to EDD Search

This is a two-part, joint blog post.  As Ron Friedmann explains “I recently spent some time looking at Xerox’s new CategoriX EDD tool and writing a post about it.  After reading it, I realized it would be helpful to set my discussion in a broader context. So I turned to my friend and e-discovery expert Tom O’Connor and author of the docNative Paradigm Blog . What follows is a combined post; we wrote each section individually and are cross posting this. “

Xerox CategoriX and Musings on the Best Approach to EDD Search by Ron Friedmann

In early October, Xerox Litigation Services released a new e-discovery search and review tool called CategoriX. How should EDD professionals think about this and other new search technologies?

A Xerox PR firm offered me phone time with the CategoriX product manager, Svetlana Godjevac. Always curious about new litigation document review tools, I accepted. I also read the CategoriX product sheet and a statistics-heavy Categorix white paper explaining how Xerox tested the product.

The CategoriX approach sounds interesting and useful. Xerox R&D in Grenoble developed the product and the company appears offers it beyond the litigation market (see the page Text Categorization and Clustering housed under Xerox Technology and Brand Licensing.) The product combines ‘probabilistic latent semantic analysis’ (document clustering) with iterative machine learning.

It sounds powerful but I can’t evaluate its effectiveness. This is by no means a criticism. Both search approaches have been around for years so it’s hard for me to assess how they work in CategoriX. Learning more about Caterorix confirms what I’ve suggested before: mere mortals can no longer evaluate EDD platforms, at least not by assessing the underlying algorithms.

I lament that I don’t know enough statistics to fully comprehend the white paper but Xerox appears to have tested the product (though the nature of the 2 document sets studied and human reviewer groups is not described). One finding I did focus on is that Xerox used this tool to quantify inter-reviewer variability. Not surprisingly, humans are not all that consistent, a fact that lawyers routinely overlook. In my conversation, Ms. Godjevac reports that Xerox does explain the statistics to lawyers and works with them to understand the problems of human review.

How a litigation team should choose among the available advanced tools is a real quandary. The investment to run a “bake off” among competing choices is enormous; moreover, the outcome may well depend on the nature of the documents. What does this say about defensibility in general? Would it be defensible to use product A if an objective study showed that product B was 20% better? And what exactly does 20% better mean anyway?

Courts seem a long way off from considering this question but the leap from the current standard to one that requires comparing tools seems more a matter of degree than of kind. Are litigation support professionals obliged constantly to evaluate new tools to make sure what they now use is adequate?

Of course, I may be way off base here. Which is why I am surprised and dismayed that I haven’t found much commentary on this tool. Many other bloggers comment on EDD but I did not find much blogging (or Tweeting) about CategoriX. I would like to see more discussion of products, comparisons of them, and the future standard of what courts will rule is defensible.

[I felt this did not stop at quite the right spot so am glad Tom stepped in....]

The Challenges of Evaluating EDD Search Tools by Tom O’Connor

Ron, your comments about the problems facing anyone attempting to evaluate ED applications are right on target. First of course is the fact that one needs an engineering degree to even read some of the white papers in this field. But it seems to me that the problem starts even before that with several fundamental problems.

The first, as you mention, is that there is never enough detail given about the document sets being studied. Understanding the documents is a crucial part of any automated litigation process and evaluating products which don’t sufficiently describe the universe of documents they are working with is simply impossible. This is not a failing of Xerox alone but really all the reviews I have seen. It is nearly impossible to cross compare applications if they are “tested’ on widely divergent data sets.

In addition, some search engines use a standardized thesaurus such as the publicly available WordNet Lexical database, an open source thesaurus from Princeton University. It has over 100,000 English words and associations. As an open source resource, the WordNet database is available for download and examination if needed for litigation validation purposes. If, however, the comparison is between one program using this database and another one that uses an internal or closed database, does that really help us?

Even the widely touted TREC (Text Retrieval Conference) study suffers from this failing in my opinion. The TREC study used a test set of 7 million documents available to the public pursuant to a Master Settlement Agreement between tobacco companies and several state attorneys general. Attorneys assisting in the study drafted five test complaints and 43 sample document requests (referred to as topics). The topic creator and a TREC coordinator then took on the roles of the requesting and responding counsel and negotiated over the form of a Boolean search to be run for each document request.

The problem is those documents were not in native format and did not include attachments. Given that typical collections today consist largely of massive volumes of e-mail, many with attachments (and attachments to attachments), this is , a huge issue when evaluating search capability for email.

A second problem I see concerns what type of search is best. We all agree that computer searching is more accurate than human review. The Sedona Conference Commentary on the Use of Search and Information Retrieval Methods in E-Discovery, released in August 2007, states that “Human review of documents in discovery is expensive, time consuming, and error-prone. There is growing consensus that the application of linguistic and mathematic-based content analysis, embodied in new forms of search and retrieval technologies, tools, techniques and process in support of the review function can effectively reduce litigation cost, time, and error rates.” So the assumption that concept search is better than Boolean searching, although widespread, may be wrong.

In Disability Rights Council of Greater Wash. v. Wash. Metro. Area Transit Auth., 2007 WL 1585452 (D.D.C. June 1, 2007) Federal Judge Facciola stated that “concept searching, as opposed to keyword searching, is more efficient and more likely to produce the most comprehensive results.” Judge Grimm made a similar statement in Victor Stanley, Inc. v. Creative Pipe, Inc (Civil Action No. MJG-06- 2662 (D. Md. May 29, 2008).

 But the TREC study results, however, don’t seem to support these judicial positions. In that study, computer scientists from academia and other institutions attempted to locate responsive documents for a number of topics using 31 different automated search methodologies, including concept searching. The result? Boolean searches located 57 percent of the known relevant documents. None of the alternative search methodologies had better results.

In fact, a Boolean search generally equaled or outperformed any of the individual alternative search methods, but those alternative searches also captured at least some responsive documents that the Boolean search had actually missed. The lesson? Manual review misses many documents but so does keyword searching, Boolean searching and concept searching – but they all miss different documents. The best approach is to use multiple applications to do iterative searches which winnow down to the best possible results.

This isn’t late breaking news. Ron, you started a discussion in May 2008 in Concept Searching in E-Discovery. Some of the info above I gleaned from reports on web sites and reports by people like Herb Roitblatt or Gene Eames who know a whole heck of lot more about this than do I. But the point is, one product isn’t going to do the job, no matter how good the product or convoluted their documentation. And irrespective of the tool, the “operator” better be well trained or who knows what the results will be.

I share Ron’s concern about emerging standards of defensibility. Given the technical complexities and the lack of statistical certainty, I don’t see how a clear, stable defensibility standard will emerge other than what we’ve seen, namely, have a plan, apply some smarts, and document what you do. As we’ve seen in other arenas, developing standards by judicial opinions is a long and messy process. Well, I suppose the upside is that consultants will stay busy!

Kellner Say Cooperation Now The Law in 7th Circuit

The Seventh Circuit Court of Appeals (which  covers Illinois, Indiana, and Wisconsin and all  the federal district and  appellate courts therein, including of course Chicago) has begun an Electronic Discovery Pilot Program. It was, according to the report itself “Developed as a result of (a) continuing comments by business leaders and practicing attorneys, regarding the need for reform of the civil justice pretrial discovery process in the United States,(b) the release of the March 11, 2009 Final Report on the Joint Project of the American College of Trial Lawyers Task Force on Discovery (“Task Force”) and the Institute for the Advancement of the American Legal System at the University of Denver (“IAALS”),1 and (c) The Sedona Conference Cooperation Proclamation.   The stated intent of the Pilot Program is to “take action to reduce the rising burden and cost of discovery in litigation in the United States brought on primarily by the use of electronically stored information (“ESI”) in today’s electronic world.”

The Seventh Circuit Electronic Discovery Committee , which developed the program, is a group of trial judges and lawyers, including in-house counsel, private practitioners, government attorneys, academics, and litigation expert consultants headquartered primarily in the Seventh Circuit. Committee members met for the first time in May 2009 and consulted with Ken Withers, the Managing Director of The Sedona Conference. Over the next four months various E-Discovery Committee members working in sub-committee groups continued to work on developing the Pilot Program and the result was the Seventh Circuit Electronic

Discovery Pilot Program’s Principles Relating to the Discovery of Electronically Stored Information (“Principles”). Those Principles will be implemented and evaluated during Phase One from October 1, 2009 through May 1, 2010.

What does this mean in practical terms? I asked that question of Chuck Kellner, the Vice President of Consulting for eDiscovery at Anacomp’s CaseLogistix and he replied: “The Sedona Principles on Cooperation and more, including references to Early Case Assessment, are about to become the law of the land in the Seventh Circuit”

Chuck went on to explain: “Basically what it says is that attorneys will learn about ESI, will come up to speed on their clients’ electronic discovery matters, will assess their cases, and will cooperate effectively and will negotiate scope of electronic discovery, and they will be monitored while doing it.   Later phases of the program will turn the screws tighter on requirements of cooperation, disclosure, and proportionality.  The meat of this starts on page 9 and the principles themselves on page 13.  The back half is a proposed standing order, which the judges can use in every case filed in every federal court in the Seventh Circuit. “

“I think what it will mean for us is increasing emphasis on reading the data early.  The EDRM as a way to describe various litigation support tasks might still be useful, but the EDRM as a linear workflow is under stress, as we predicted, in favor of piling up more concurrent tasks atop each other, and to “the left”.  Everything that we have been discussing and what we are now doing in ECA about mining metrics, about finding the significant documents quickly, and about estimating the size and cost and time of the review, these are all quickly becoming legal requirements, not just good strategy. “

Echoing statements we have heard all year from federal judges, Chuck also said. “ The tone of the judges there was that they were no longer bearing attorneys’ unpreparedness for meet-and-confer, and were prepared to start a rigorous program of enforcement.  To place this more broadly than the Seventh Circuit, the theme echoes what we’ve seen in case decisions in the last few months.”

So if you haven’t read The Cooperation Proclamation, this is a good time to do so. Especially if you practice in the Seventh Circuit.

Quick Report on ECA from The Masters Conference

Ron Friedmann sat in on a session at The Masters Conference called Early Case Assessment: Looking to the Future – From Early Assessment to Early Awareness. His notes included the following key points:

What is ECA? It’s getting an early look at the facts of your case and at the scope of discovery.

Where are the immediate savings; how is this different than past? It’s not a tool per se, it’s a method (a process, the right people, and technology). ECA does not generate savings very early – it’s not about upfront savings. You have to invest at the outset to learn about the case. Clients typically want to delay spending, so ECA is counter-intuitive to many lawyers.

Other than cost, are there other limits that hold back ECA? Clients are the main barrier. We are moving beyond linear review and search terms to a more subjective approach.

Tools that are better at ‘understanding’ data are in the ECA bucket… how do these emerging technologies affect the process, especially if clients bring the tools in-house? To start, who should operate the tools? It depends on organizational structure. It can be IT, Legal, Info Security, other corporate departments.

Ron’s comments on the session include the observation “What struck me most about this discussion is how much education is required “ and he goes on to comment that  it seems we’ve been talking about these issues for over 20 years.  Remember that in 1990, early tools such as scanning and OCR’ing led to full-text databases which could be searched by both conceptual and Boolean tools.

So why are we still trying to convince people to do iterative searches of data to pare down document populations? Ron notes that “The vendor challenge here is that clients are reluctant in this environment to spend upfront. So you need to educate your client – let them know that ECA will reduce the volume of documents that require human review. “

Fascinating comments as always from Ron and you can read more on this and his other reports from The Masters Conference  at his blog, Strategic Legal Technology.

A Few Recent Happenings in the ED World

Here’s several interesting topics that have developed in the last week: 

1. The discussion about ED standards that I broached in my post of Sept. 21 continues to percolate. The most recent take on the debate is a posting on EDD Update by Seattle lawyer and technologist Eric Blank who raises some interesting points about data exchange protocols, capture rates and the use of consultants.
2. You can expect those points, and many others, to be discussed at The Master Conference this week in Washington, D.C.  Chuck Kellner will be there speaking on Information Cooperation, a subject on which Chuck and I also presented a webinar several months ago: the recording of that event is available on the Anacomp web site here.
3. The EDRM web site has a new look and feel and has added a great deal of content, including a link to the E-Discovery Zone, the series of interviews with luminaries of the e-discovery profession conducted by Browning Marean and myself and hosted by TechLaw.

Project Management Redux

The most recent Socha-Gelbman survey stated that “project management has grown in prominence as a means to minimize missteps and deliver more predictable, reliable, and cost-effective results.” And PM is certainly getting attention these days thanks to a good discussion at The Delaware E-Discovery Report and the results of a survey released by Applied discovery. But the discussions are not all positive. In the Delaware ED Report, Paul Easton said, “implementing project management software does not equal implementing project management” and the AD survey found that although 87% of respondents considered project management a critical component of their ED projects, only 8% were satisfied with the clarity of objectives and plans defined in their last e-discovery project.

So if we all know that PM is good why aren’t we doing it better? Part of the problem, it seems to me, is that we are lacking a specific definition of the phrase. As Brett Burney points out in his article The Emerging Field of Electronic Discovery Project Management, “ Formal project management, however, is a recognized professional discipline, complete with educational requirements (Project Management Professional or PMP) and an oversight body called the Project Management Institute (PMI).” Yet I have never heard of a law firm utilizing those standards in designing a project scope or specifications.

Why? Well as Conrad Jacoby says in his article on LLRX called Applying Project Management Techniques to Litigation Discovery, litigation team leadership “often only has time to focus on the highest priority projects and problems.” This dichotomy really highlights the fact that technology tools are just one component of success in e-discovery. Every case is different and there is no cookie cutter application or device that meets the challenges of each of those cases.

The fact is that advice from a good consultant ( like those mentioned here or well myself) is necessary for assistance with project planning, including data scoping, choice of document review methodologies and project training. Given the rise in number and complexity of ED specific applications, the need for good consulting services is at an all time high in order to select the best choices for the workflow of your particular matter. So take a look at the articles mentioned above, start assessing your projects and get the advice of a good consultant to help you decide how best to proceed.

Top Ten Tips for E-Discovery

Yesterday morning I spoke on a webinar for The Masters Conference called “Top 10 Tips for Working with E-Discovery” in which I shared the 10 tips I give to clients at our first meeting.  The idea is give a better understanding of e-discovery and basic obligations under the current state and federal rules.  Hopefully this begins my consultative engagement with the first step towards best practices e-discovery and a better discovery strategy.

My top ten tips are:

1. Read The Rules

2. Read The Decisions

3. Know The Terms

4. Know Where The Data Is

5. Talk To The IT Staff

6. Talk To The RM Staff

7. Help Clients With RM Policy

8. Develop A Litigation Hold Strategy

9. Enforce The Litigation Hold Strategy

10. Talk With Your Clients

You can hear a recording of the prsentation by going to the Masters Conference webinar page to  and don’t forget to stop by the Masters Conference itself, Oct. 13th and 14th at the Ronald Reagan Building in Washington, D.C.. I’ll be there as will Chuck Kellner and all the Anacomp folks, who will be happy to answer your questions about e-discovery and how best to handle the challenges you are facing.