Robert Ambrogi Blog
I have made no secret of the fact that I consider litigation analytics to be one of the most important technologies to have gained traction in recent years. Writing about analytics a year ago on Above the Law, I titled the post, This Tech Can Turn the Tables in Litigation. In my year-end summary of the most important legal tech developments of 2018, my top item was “analytics become essential,” noting, “We could be nearing a point where it would be malpractice for a lawyer not to use analytics.”
While all of that still holds true, and while the technology has continued to evolve and the array of products to expand in the year since then, it is important for legal professionals to understand that litigation analytics is still a developing technology, and that there are weaknesses in both the technology and the data on which they rely.
That means that these products differ not only in the kinds of analytics they offer, but also in the results they deliver.
The differences among these products were dramatically highlighted by a study conducted earlier this year by a group of law librarians who compared federal court results across seven legal analytics products. The products they compared were: Bloomberg Law, Docket Alarm Analytics Workbench (from Fastcase), Docket Navigator, Lex Machina, Lexis Context, Thomson Reuters Monitor Suite and Westlaw Edge.
Consider this seemingly simple research query: “In how many cases has Irell & Manella appeared in front of Judge Richard Andrews in the District of Delaware?” When the librarians tested that query across the seven products, they got widely different answers.
The librarians tested 16 “real world” research questions across the seven products, with similarly varying results. Although they have not yet published their findings, four of the participants — Diana Koppang, director of research & competitive intelligence at Neal, Gerber & Eisenberg; Tanya Livshits, director of research services at Irell & Manella; Kevin Miles, manager of library services at Norton Rose Fulbright; and Jeremy Sullivan, manager of competitive intelligence & analytics at DLA Piper — presented their findings during an analytics “deep dive” session in July at the annual meeting of the American Association of Libraries.
(I was a participant in that panel, as was Jean O’Grady, who wrote about the analytics study in a post at her Dewey B Strategic blog: What Do Law Firms Need to Know About Buying Litigation Analytics Products?)
The librarians limited their study to federal district courts, even though some of these platforms also provide analytics for state courts. They also limited their focus to docket analytics, meaning analytics based on federal docket data, not content analytics that focus on language and citation patterns in judges opinions, such as Precedent Analytics in Westlaw Edge and Context from LexisNexis.
One reason for the divergent results across platforms, the librarians found, was what they called “the PACER problem.” In a nutshell, the problem with PACER data is that it is replete with problems. PACER data is full of typos. The Nature of Suit codes used to label types of matters often do not reflect the true focus of a matter. Names of parties, firms and attorneys are not normalized for consistency. Attorney moves sometimes result in matters being associated with the wrong firm.
(The problems with PACER data were discussed in greater depth in a guest post here earlier this year by Josh Becker, chairman of Lex Machina and head of legal analytics at LexisNexis: Guest Post: Not All Legal Analytics Tools Are Created Equal.)
There are also major differences among the products in the search options and types of analytics they provide. For example, here are how the products compared in their ability to show analytics regarding case and motion outcomes:
| LITIGATION OUTCOMES | Bloomberg Law | Docket Alarm | Docket Navigator | Lex Machina | Lexis Context | Westlaw Edge | Monitor Suite |
| Outcome or Findings | No | Yes | Yes | Yes | Yes | Yes | Yes |
| Motion to Dismiss | Yes – Judge Level only | Yes | Yes | Yes | Yes | Yes | Yes |
| Motion for Summary Judgment | Yes – Judge Level only | Yes | Yes | Yes | Yes | Yes | Yes |
| Motion for Class Certification | Yes – Judge Level only | Yes | No | No | Yes | Yes | No |
| Appeal Outcomes | Yes – Judge Level only | Yes | No | No | Coming 2020 | Yes | No |
| Damages | No | Yes | Yes | Yes | No | No | No |
| Remedies | No | Yes | Yes | Yes | No | No | No |
| Jury Verdicts | No | Yes | Yes | Yes | Yes | Yes | No |
| Expert Witness Challenges | No | No | Yes (but not by expert name; using keywords over judge ruling data) | Yes (only for Product Liability cases) | Yes | Yes (but not by expert name; using keywords over judge ruling data) | No |
| Are additional motion type outcomes available? | No | Yes | Yes | Yes (practice area specific) | Yes | Yes | No |
And here is how they compared in terms of types of search options they offered:
| SEARCH OPTIONS | Bloomberg Law | Docket Alarm | Docket Navigator | Lex Machina | Lexis Context | Westlaw Edge | Monitor Suite |
| Search by Party: Company | Yes – but limited (70,000 public and 3.5 million private companies) | Yes | Yes | Yes | Coming 2019-Q4 | No | Yes |
| Search by Party: Individual | No | Yes | Yes | Yes | Coming 2020 | No | Yes |
| Search by Firm | Yes (only those in Bloomberg law firm database; approximately 7,000 firms) | Yes | Yes | Yes | Coming 2019-Q4 | Yes | Yes |
| Search by District Court Judge | Yes (current) | Yes (current and retired) | Yes (current and retired) | Yes (current and retired) | Yes | Yes (current and retired) | Yes (current and retired) |
| Search by Bankruptcy District Court Judge | No | Yes | No | Yes | No | No | |
| Search by Magistrate Judge | No | Yes | Yes (current and retired) | Yes (current and retired) | Yes | Yes (current and retired) | Yes (current and retired) |
| Search by PTAB Judge | No | Yes | Yes (current and retired) | Yes (current and retired) | No | No | No |
| Search by ITC Judge | No | Yes | Yes (current and retired) | Yes (current and retired) | No | Yes (but analytics are limited) (current and retired) | Yes (current and retired) |
| Search by Delaware Court of Chancery Judge | No | Yes (partial coverage; full coverage forthcoming) | No | Yes (current and retired) | Yes | Yes | Yes (current and retired) |
| Search by Attorney | Yes – but limited (over 100,000 attorneys at over 775 law firms) | Yes | Yes | Yes | Forthcoming 2020 | Yes | Yes |
| Search by Expert | No | No | Forthcoming | Yes (Product Liability only) | Yes | No | No |
| Search by Keyword | No | Yes (across all content) | Yes (across all content) | Yes (across all content) | Yes (but only after selecting a judge or expert, keyword filtering available) | Yes, but only within each “tab” – not across all analytics content | No |
It is important to note that these charts, as well as all of the study’s findings, were current as of July 15, 2019. As I noted above, and as the study authors noted, these products are evolving and changing rapidly, so they may now have capabilities or coverage not reflected above.
Another finding of the study was that the products varied in their ease of use and functionality — and there appeared to be an inverse relationship between the two factors. Here is the spectrum of the two, as presented during the AALL program:
Ease of use from least difficult to more difficult:
Bloomberg >> Context >> Monitor Suite >> Docket Navigator >> Edge>>Lex Machina>> Docket Alarm
Functionality from least complex to most adaptable:
Bloomberg >> Context >> Lex Machina >> Docket Navigator >> Edge >> Monitor Suite >> Docket Alarm
Diana Koppang, a principal organizer of this study, tells me that she hopes to publish the full results at some point. Also, she will be presenting an update on this research in a program scheduled for Legalweek in New York in February.
But at the program last July, the four presenters summed up their “10,000 foot overview” of the findings from their research:
- There is no winner among the products.
- The best platform for your organization depends on your use case and budget.
- Analytics platforms generate better results when combined with research through other platforms (company research, docket searches).
- “Litigation Analytics” is now too broad of term.
- “Analytics Research” is different from “Case Law Research.”
- Assume nothing.
- Transparency is absolutely necessary.
- Remember the PACER Problem.
For me, the bottom line is that often-repeated Latin phrase caveat emptor. As legal professionals, we have an obligation to understand the benefits and risks of the technology we use. Litigation analytics offer many potential benefits, but if you are not aware of the shortcomings, you may also be putting yourself or your client at risk.