New Study: Repeated LLM Harness Runs Don't Prove Real Specialization

It is a generated program or wrapper that steers a language model toward a specific task, aiming to improve inference accuracy through task specialization.

Because repeating the same program raises coverage without any real specialization, making coverage alone a misleading metric. The study separates coverage, repeatable task advantage, and gains from pre-execution selection.

If they rely on oracle coverage or average scores to pick harnesses, the paper suggests verifying that advantages persist across executions and under matched inference budgets before adopting any specialization.

It covers 386 MATH-500 tasks with three repeats per program, and the paper itself notes that stable complementarity remains unresolved at three repeats.