Learnpaper® | Luminary Interview
What it takes to write a better brief
Westlaw Edge Quick Check analyzes your brief—whether an old brief of yours, or opposing counsel’s—to identify overlooked and new cases that may strengthen your argument or identify omitted authority your opponent chose not to cite. It also lets you know about cases in the brief at risk of being overruled.
It’s a clever use of artificial intelligence (AI) no doubt, but what’s the story behind Quick Check and how useful is it for litigators like you?
To find out, I spoke with Carol Jo (CJ) Lechtenberg, Sr. Director of Westlaw Edge Product Management, and Merine Thomas, Software Architect at Thomson Reuters’ Center for AI. Over the past 3 years, Lechtenberg and Thomas and their teams collaborated to create Quick Check.
As a law student, Lechtenberg wanted to become a prosecutor. But while clerking for a judge after law school, she enjoyed researching complex legal issues more than the courtroom proceedings. “I loved the legal research!” says Lechtenberg.
Thomas also found her career diverging from her initial plan to become a software engineer. She majored in computer science but also focused on math, machine learning, and natural language processing—some of the disciplines comprising AI. After working as a Software Engineer in AI at Thomson Reuters for 6 years and receiving several promotions, mentors encouraged Thomas to embrace her interest in AI and move to a role where she could be integral in building AI products for the company.
“I joined what is now known as Thomson Reuters’ Center for AI as a software engineer to become one of the few AI Architects in the company,” says Thomas. She and her team solve the company’s hardest technical challenges—most recently Quick Check. Thomas has also worked on PeopleMap, WestSearch Plus, and Westlaw Edge.
Lechtenberg became a Westlaw Reference Attorney after her clerkship and developed a reputation for helping customers with thorny legal research issues. She was promoted to a role in which she assisted stumped reference attorneys.
What feedback have you received on Quick Check, especially among skeptics?
CJ Lechtenberg: "The feedback we’ve received about Westlaw Edge Quick Check thus far has been overwhelmingly positive. What’s really gratifying is hearing from customers like Gabe Ramsey, Partner at Crowell & Moring, who says, 'Quick Check found things that were highly relevant and not easily discoverable. That’s valuable, and it’s going to make this type of work a lot more cost effective.'"
Robbie Rogart, Counsel from Crowell & Moring states, “It’s really important to our clients that we get it right—and that we get it right on the first try. Quick Check helps to give that gut check before you submit something, whether that’s to your internal client, the partner, actual external clients, or perhaps the court. It helps to give that confidence that you have found what you need.”
What distinguishes Quick Check from older products on the market? Describe some of the technical breakthroughs and their embodiment in Quick Check’s user experience and functionality.
Among the many breakthroughs, the biggest is how Quick Check analyzes the document to make recommendations. Our customers have already done good legal research. There may be gaps because they didn’t know about a case, or they didn’t have all the time in the world. Quick Check attempts to recommend highly relevant authorities to the issues in the user’s document and raise the confidence in their work.
This is a very complex task. We use natural language processing techniques to analyze the document and understand it. We use a plethora of content—primary sources (cases and statutes, appellate briefs and trial court documents); editorially enhanced content like headnotes, secondary sources, and metadata like key numbers; KeyCite information; citation graphs; and aggregated user data—to find potential authorities relevant to the issues at hand.
These sources consist of over 6 billion different data views. In short, we created an algorithm that looks in places humans cannot, and in ways that humans do not think to combine. We use machine learning models trained on metadata and linguistic features to find the candidates that are similar in both fact and law to the customer’s issues. We aim to be as precise as we can. And we made it fast.
Quick Check has two modes: analyze your own work or analyze your opponent’s work. Have you received any feedback from customers on how these have impacted their work?
CJ Lecthenberg: "For analyzing their own work, attorneys say that Quick Check helps streamline their research and gets them answers more quickly. For analyzing an opponent’s work, attorneys love that they can rapidly get a feel for where their opponent’s arguments are headed and what their options are for making their own arguments to the court. Attorneys want to see what their opponents are doing, but also what they are not citing to. Quick Check users say they can’t imagine doing legal research without it."
Quick Check uploads a brief that is not yet final or public. How do you ensure security?
Mernie Thomas: "We recognize that security and privacy of our customers’ work product is paramount. Attorney-client privilege is the most sacrosanct of rights in the adversarial process.
The uploaded file or plain text information is encrypted in transit via HTTPS. It stays encrypted as it gets to Quick Check. This cannot be accessed or viewed by any Thomson Reuters employee or contractors.
The uploaded file or plain text is deleted immediately after the Quick Check report is generated. It is not available in any of our systems in any form. The resulting Quick Check report is maintained temporarily subject to expiration for a maximum of 24 hours accessible only by the customer.
We remain committed to maintaining the security of our customers’ data."
Quick Check introduces some new ways to display information. Discuss some of these and the reaction from customers.
CJ Lechtenberg: "For Quick Check, we took a step back and thought about what attorneys do when they either write their own brief or when they examine the brief of an opponent. As Merine mentioned, the algorithms behind Quick Check are the most sophisticated we’ve ever created. Through machine learning and AI, Quick Check has the ability to look for content in places a researcher would not look, in ways they would not look, and in far less time than it would take them to look. Attorneys really appreciate that Quick Check harnesses the best legal research techniques, beyond what they can humanly do, and provides them with resources they may have missed doing their traditional legal research.
We’ve also added many features that aren’t found elsewhere to give the researcher as much context as possible while also making review of the report quick. First, we provide segment headings from the uploaded document, so the researcher knows to which issues the recommendations apply. This is far more beneficial to the researcher than just supplying a laundry list of cases with no indication of why they are relevant."
It’s important to note that these outcomes are not for the case as a whole, but rather for the individual issue for which the case is recommended.
Recommendation tags let the researcher know specific attributes about the case that might be helpful, such as whether they are frequently cited, from a high court, or decided within the last 2 years. (See Figure 1.)
Quick Check leverages Westlaw Edge features such as the “eyeglasses,” filtering by previous user interactions, folders, notes, and highlights. We also bring some KeyCite functionality to Quick Check so the user can get a holistic view of the table of authorities in their document and review any potential weakness in what they have already cited.
Speed is also an important feature, as Merine noted. One customer commented there is really no good reason not to use Quick Check given how fast it works.
KeyCite Overruling Risk, which Quick Check incorporates into its results, seems like an obvious feature. But only Westlaw Edge has it. What’s the story behind it and was it always on the Quick Check roadmap?
Mernie Thomas: "Yes, KeyCite Overruling Risk is only available in Westlaw Edge. We are constantly modernizing our platforms and developing new features and capabilities on these new platforms. We want our customers to benefit from that.
We have been thinking about the risk when an overruled case, also known as red flagged, extends beyond the overruled case itself—such as a case you found that seems to be good law. When the case you found cites the red-flagged case as an authority on the same issue, it could be at risk of being bad law. The challenge lies in identifying that your case cites the overruled case on the same issue, doing so reliably, and figuring out if there is a risk.
KeyCite Overruling Risk uses natural language processing and machine learning to find cases that have been overruled on the same issue. Our machine-learned models analyze the citation network, KeyCite features, and key number features, and then linguistically analyze the text of the cases themselves to identify if there’s a risk that a case has been implicitly overruled by another. (See Figure 2.)"
When we were developing Quick Check on Westlaw Edge, it felt natural to incorporate KeyCite Overruling Risk. It helps in presenting any potential weakness—both for Check Your Work and Analyze Opponent’s Work use cases—that the customer is not aware of because the risk is implicit. When making recommendations, we also do not want to recommend bad law. It is logical that a customer would not want to see bad law and waste their time on it.