Llms Are Bad Judges. So Use Our Classifier Instead

Posted3 months agoActive3 months ago

lordgrenville

41 points

11 comments

papers.ssrn.comTechstory

skepticalmixed

Debate

60/100

AI in LawLlmsClassifier Systems

Key topics

AI in Law

Llms

Classifier Systems

A research paper proposes a classifier system as an alternative to LLMs for judging legal cases, sparking discussion on transparency, trust, and the role of AI in law.

Snapshot generated from the HN discussion

Discussion Activity

Moderate engagement

First comment

Peak period

Day 1

Avg / period

5.5

Comment distribution11 data points

Loading chart...

Based on 11 loaded comments

Key moments

01Story posted
Sep 28, 2025 at 7:08 PM EDT
3 months ago
Step 01
02First comment
Sep 29, 2025 at 12:08 AM EDT
5h after posting
Step 02
03Peak activity
10 comments in Day 1
Hottest window of the conversation
Step 03
04Latest activity
Oct 10, 2025 at 5:49 PM EDT
3 months ago
Step 04

Generating AI Summary...

Analyzing up to 500 comments to identify key contributors and discussion patterns

Discussion (11 comments)

Showing 11 comments

adlumal

3 months ago

1 reply

Having done some work in the legal AI field, I wonder how this classifier deals with issues of transparency, explainability and ultimately trust? It’s valuable to have some idea of how a proceedings might unfold but from my experience most competent lawyers have a high bar when it comes to trusting any AI/ML output.

Taikonerd

3 months ago

I was worried about explainability, too. If the classifier just spat out "INNOCENT" or "GUILTY," it would be useless -- the legal reasoning has to be part of the output.

Looking at the paper, the classifier definitely does output its reasoning:

"The legal issue at hand is whether the 50/50 royalty split in the 1961 contract binds only pre-existing affiliates or if it also includes affiliates that come into being after the agreement..."

leobg

3 months ago

2 replies

This reads like an ad for Arbitrus.ai. It’s copywriting lingo:

> We built one called Arbitrus. We put it through a mini-Choi test and it mopped the floor with the competition

lordgrenvilleAuthor

3 months ago

1 reply

True, but they own that:

> Declaration of Interest: [Authors] have financial interests in...Arbitrus.ai. As the title would suggest, the authors are making no effort to obfuscate this fact.

causal

3 months ago

I'd argue that dressing an ad up as an academic paper is obfuscation

Taikonerd

3 months ago

I had thought: what's the business model for Arbitrus? Is it going to be a sort of "suggested finding" tool for judges? Or are law firms going to use it to screen cases, so they can pick winners?

It seems like the answer is neither: on their website, Arbitrus.ai says it's for private arbitration. "Arbitrus is a private court system with an AI judge. Why use the public court system or expensive AAA arbitration to settle your disputes, when you can do it faster, cheaper, and better with Arbitrus?"

nextaccountic

3 months ago

1 reply

What kind of classifier is this? I mean is it k-NN (for example), or something else?

Even LLMs can be viewed as classifiers, as the paper (ad?) itself admits.

esafak

3 months ago

pg36 "This is proprietary and part of Fortuna’s moat, so we explain it to the extent appropriate."

opwieurposiu

3 months ago

I love the idea of Arbitrus.ai, but they want $2500 a go to test it. I wish they had a demo version to play with.

barbazoo

3 months ago

The margin and line spacing makes this hard to read. Is this how you're supposed to typeset a paper? Some pages have three, maybe four sentences on them.

LawKek

3 months ago

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5587611. Related to this post.

View full discussion on Hacker News

ID: 45408880Type: storyLast synced: 11/20/2025, 2:46:44 PM

Want the full context?

Jump to the original sources

Read the primary article or dive into the live Hacker News thread when you're ready.

Open link View on HN