Taxcalcbench: Evaluating Frontier Models on the Tax Calculation Task

Posted3 months agoActive2 months ago

handfuloflight

70 points

24 comments

arxiv.orgTechstory

calmmixed

Debate

40/100

AITaxationLlms

Key topics

Taxation

Llms

Researchers released TaxCalcBench, a benchmark for evaluating frontier models on tax calculation tasks, highlighting the challenges of applying AI to complex legal tasks like taxation.

Snapshot generated from the HN discussion

Discussion Activity

Very active discussion

First comment

Peak period

Day 1

Avg / period

Comment distribution24 data points

Loading chart...

Based on 24 loaded comments

Key moments

01Story posted
Oct 15, 2025 at 11:45 PM EDT
3 months ago
Step 01
02First comment
Oct 16, 2025 at 1:31 AM EDT
2h after posting
Step 02
03Peak activity
22 comments in Day 1
Hottest window of the conversation
Step 03
04Latest activity
Oct 29, 2025 at 7:55 PM EDT
2 months ago
Step 04

Generating AI Summary...

Analyzing up to 500 comments to identify key contributors and discussion patterns

Discussion (24 comments)

Showing 24 comments

Rudybega

3 months ago

3 replies

I wonder if you could dramatically improve these results with some relatively simple scaffolding and tool access.

If a ton of these mistakes are genuinely simple calculation errors, it seems like giving the models access to a calculator tool would help a fair bit.

sails

3 months ago

I feel like we are already there. I would imagine if you set Claude Code or Codex this task, running in the CLI, you would see a huge improvement, and that is before you start creating task specific guardrails.

I’m surprised they haven’t tried this, I’m running my own in parallel against my accountant in this way.

Lionga

3 months ago

The problem is they do not understand what/how to calculate not the actual act of adding or multiplying. I tried asking ChatGPT to calculate some taxes for three countries, two of which I have been filing taxes already. For the two I know ChatGPT gave wildly wrong numbers (not even right ballpark), so I know I could not trust numbers for the third which was what I was mostly interested in.

michaelrbock

2 months ago

We agree, that's the thesis behind our tax development coding agent: https://www.columntax.com/blog/introducing-iris-our-ai-tax-d...

hodgehog11

3 months ago

1 reply

Am I missing something or did they only assess this on Google and Anthropic models? If so, all I can ascertain from this is that latest Gemini models outperformed Claude on this particular task, which should be surprising to no-one. What about GPT-5? Open weight models?

topaz0

3 months ago

Somebody posted the up-to-date leaderboard up thread: https://news.ycombinator.com/item?id=45603113

stared

3 months ago

1 reply

A bare model may lack a lot.

Yet a week ago I used Claude Code for my personal finances (not taxes) - I downloaded over a year’s worth of my bank account data. Since I pay for most things by card, if I buy lunch, it’s there.

With a single prompt (and about 10 minutes), it produced an analysis. It solved all the technical issues by itself (e.g., realizing it wasn’t CSV but TSV) and ran quite a few different explorations with Pandas. It was able to write an overview, find items that were likely misclassified, etc.

Everything I checked by hand was correct.

So, instead of pursuing a project to write an AI tool for personal finance, I ended up concluding: “just use Claude Code.” As a side note, I used 14 months of data due to my mistake - I wanted to analyze 2 months of data, since I didn’t believe it would handle a larger set, but I misclicked the year. The file was 350 KB.

jasonjmcghee

3 months ago

1 reply

I hear you, but I'd also rather someone else assume the liability if possible. (Assuming there's a company backing the model)

So until there's umbrella AI insurance...

stared

3 months ago

Exploratory data analysis is one thing - here the risk is low. If something does not work, it doesn't. Small omissions are not important.

As of now, I would not use automatic AI to make any financial decisions with direct consequences. Unless system is tested and benchmarked against accountants.

cjbarber

3 months ago

1 reply

Leaderboard: https://github.com/column-tax/tax-calc-bench

throwaway13337

3 months ago

1 reply

Useful.

I wonder what an average accountant would score.

I know LLMs have helped me identify many mistakes accountants have made on my behalf. Some mistakes that could have cost me a lot of money if not caught.

topaz0

3 months ago

Given that they're restricting to very simple situations I'd expect accountants to score 100%.

jgalt212

3 months ago

1 reply

I'm surprised that no LLM has a yet found any unresolved cycles in the US tax code.

anticensor

3 months ago

1 reply

Oh you mean infinite/zero tax glitches.

jgalt212

3 months ago

Yes

michaelrbock

3 months ago

2 replies

Hi, author of this paper + repo here. This dataset is particularly hard to come by, so we’re really proud to be open sourcing it.

Let me know if you have any questions, happy to discuss!

antiloper

3 months ago

1 reply

> For example, in the prompt for this experiment, the model is bootstrapped with the correct Form 1040 lines and short instructions as part of its context.

Given that only short instructions are in context, I would not have expected even a frontier model to score well on this benchmark. For better results, I'd think that giving the model access to the entire tax code is required (which likely requires RAG due to its sheer size).

michaelrbock

3 months ago

We tested models with knowledge cutoffs in 2025 so expect them to have knowledge of Tax Year 2024 forms in their weights. We also tested models with ability to do web search to get any other forms it thinks necessary: https://github.com/column-tax/tax-calc-bench

That all being said, we agree, which is what we've built with our internal tax coding agent, Iris: https://www.columntax.com/blog/introducing-iris-our-ai-tax-d... (ability to get just the right Tax form context on a per-line basis to turn the tax law into code).

anticensor

3 months ago

This topic is so American. In any other country, you wouldn't have had to consult a tax expert to prepare personal tax statements.

anticensor

3 months ago

Whereas almost every other country tries to make it easier to file taxes, even when the underlying tax schedule is complex.

mrfelipppe

3 months ago

This is awesome!

i_dont_know_

3 months ago

I'm actually quite surprized.

From another article today, I discovered the IRS has a github repo with (what seems to be) XML versions of tax questions... surely some combination of LLM and structured data querying could solve this? https://github.com/IRS-Public/direct-file/tree/main

daft_pink

3 months ago

I think AI has problems with law related tasks like taxes, because there are so many words of art. Taxes are essentially just laws and because laws and regulators and courts eventually define words in very very specific narrow ways and sometimes in different ways from one code section to another code section, AI has a lot of trouble using these very very narrow definitions.

Honestly, I think humans have trouble with this as well.

ofrzeta

3 months ago

"Calculating US personal income taxes is a task that requires building an understanding of vast amounts of English text and using that knowledge to carefully compute results. ... Our experiment shows that state-of-the-art models succeed in calculating less than a third of federal income tax returns even on this simplified sample set."

Unsurprisingly. Sometimes I feel like I am in a madhouse. Or in an alchemist's laboratory.

View full discussion on Hacker News

ID: 45601230Type: storyLast synced: 11/20/2025, 12:26:32 PM

Want the full context?

Jump to the original sources

Read the primary article or dive into the live Hacker News thread when you're ready.

Open link View on HN