InjazAI

The CARAT paper shows that materials-focused LLMs often repeat structural relations already present in their input rather than genuinely inferring them, and that accuracy alone cannot reveal the difference.

1 min read

CARAT Study Finds Materials LLMs Recite Structural Answers Instead of Reasoning

What CARAT sets out to test

FAQ

What is the CARAT paper and what does it measure?

CARAT is a research benchmark that holds the question and gold answer fixed across eight matched views of crystal structure, to separate genuine reasoning from copying an answer already printed in the input.

Why is accuracy alone insufficient to measure materials LLM reasoning?

Because a structural description often prints the very field the model is scored against, so high accuracy can appear even when the model copies rather than infers.

How does GraphSpace differ from a plain periodic graph?

GraphSpace wins by 19.3 points, but the paper decomposes this: only 1.96 points when the plain rendering carries everything the question needs, and 46.7 points when it omits those fields entirely.

Does this mean models cannot learn structural reasoning?

No. After matched supervision the model reaches 99.8%, and deleting the link drops it to 23.4%, indicating that repeating the relation and using it are two learnable steps.

Source: arXiv cs.AI

AI-assisted summary of the source above. High-confidence stories publish automatically; others wait for review. Facts that are not in the source are not added.