Robert Ambrogi Blog
The American Bar Association’s Center for Innovation has launched a website, Justice System Metrics, that catalogues the key metrics and performance indicators that can be used to evaluate progress and innovation in the justice sector, and that also provides real-world examples of how these metrics are being used to understand and drive progress in the justice system.
The site grew out of the work of a committee of the Center for Innovation’s Governing Council that has been studying the question, “How can the successes or failings of regulatory experiments, legal innovation, and new service delivery models be effectively measured and understood?”
“Our working group built a multifaceted tool that catalogues metrics and KPI’s associated with legal innovation and also catalogues use cases related to those metrics,” said Joseph Gartner, director and counsel for the Center for Innovation. “The end goal was to create something that can encourage a greater comfort with data for the legal ecosystem.”
Recently, Gartner and Amanda Brown, executive director of Lagniappe Law Lab and a member of the committee that developed the site, gave me a tour of the site and a briefing on its development.
A modal from the site explains the information presented about each metric.
Some 200 Metrics
The site catalogues some 200 metrics in the form of cards that depict each metric, the most likely “owner” of the metric, and symbols indicating whether the metric is qualitative or quantitative, or both. Each card flips to reveal a brief description on how the metric can be used and a link to the authority or source for the metric.
The committee assembled the metrics both from outside reports and through internal ideation sessions.
The metrics are keyword searchable based on the type of data they supply, the location where the data can be found, or by keywords connecting similar metrics. Metrics can also be filtered by their key focus (consumer harm, market viability, etc.), data source (bar association, regulator, court, etc.), and data type (qualitative, quantitative or hybrid).
Brown said they are continuing to work on expanding both the searchability of the site and the available filters to better conform them to how people talk about these issues. For example, the site provides filters for “hot topics” including alternative business structures, technology, and independent licensed paraprofessionals.
A section of “metrics in action” links to reports and studies that use metrics to assess the justice system.
The site also provides examples of “metrics in action” — a collection of reports, studies, law review articles and other literature that apply justice metrics to evaluate and understand various aspects of the justice system, with links to the full text of each.
These use-case examples are searchable by jurisdiction, publication date, keywords, and author type.
In two additional sections, the site:
- Offers guidance on the basics of using metrics and data-driven decision making. Brown said the information in this section is fairly basic and designed to help entry-level users understand on a step-by-step basis how to collect and use data. “This is really a starting point to where we hope this can be a research hub for people in the legal system,” Brown said. “We want to add resources to it, like how to do data visualization or mechanisms for capturing data.”
- Provides a bibliography of the authorities relied on by the committee in collecting the metrics shown on the site.
An Open Resource
Development and use of metrics is an issue that has taken on greater urgency as states such as Utah and Arizona have experimented with and adopted regulatory reforms aimed at enhancing access to justice, and following the ABA’s 2021 adoption of Resolution 115 urging jurisdictions to consider regulatory innovations that have the potential “to improve the accessibility, affordability, and quality of civil legal service.”
In the context of discussing those types of reforms, a common refrain is that there is not enough data to know if they work, Gartner said. The goal of this site is to help enhance and facilitate the collection of that data so we can know what works and what does not.
In addition to Brown, members of the committee that developed the site are Robert Taylor, managing director, Deloitte; Zachariah DeMeola, director of strategic initiatives, Law School Admissions Council; Patrick Palace, owner, Palace Law LLP; and Natalie Pierce, partner, Gunderson Dettmer.
“We want this to be an open resource where users can submit use cases, ideas for KPIs and continue to grow the content,” Gartner said. “Right now we have about 200 unique entries but this was built with an eye towards scalability and growth. The long term plan for this to be a living and breathing tool that will iterate and evolve.”
I asked Gartner and Brown if their hope for this site is that it helps bring about standardization of the data being used across the justice system.
Brown said that she is not sure that the Center for Innovation is the right entity to be saying what the standard should be, but that she does hope that sites such as this, Measures for Justice, which tracks criminal justice data, and others, could help bring the profession to the point where “we’re all measuring the same things and we’re all talking about the same things.”
Added Gartner: “We want this to be agnostic of what innovations work and don’t work. We want this to be a measurement tool that people can use to say, ‘This is what’s working,’ or ‘This is why this may not be working.’ And to have those decisions based on data.”