The task

From a reaction description to a dollar figure per gram.

An agent reads a reaction as a chemist wrote it down and returns one number: what it costs to buy the material for the run at a 1 mmol scale, per gram of product. Supplier prices are never in the prompt and cannot be recalled reliably, so the answer has to come from the tools. Because the label is computed from a frozen supplier snapshot under a fixed rule, every prediction is scored exactly.

01

Ground the names

Map each name to one structure. Abbreviations collide: TEA can mean triethylamine or triethanolamine.

Typical failureGives up when a name does not resolve and returns nothing.

02

Retrieve quotes

Pull every pack-level quote for each chemical: supplier, pack size, price, reported purity.

Typical failureGuesses a price from memory, then lands 2× off.

03

Select packs

Work out the mass each component needs, then buy whole packs of qualifying purity that cover it.

Typical failureReads 0.23 g as 230 g, an error of three orders of magnitude.

04

Aggregate

Sum the packs bought, leave out solvents, and divide by the grams of product the run yields.

Typical failureDrops the palladium catalyst and comes in 3.6× under.

Ground-truth label

c = Σ_{i ∉ solvent} q_iM_P · 10⁻³ · y/100

q_i is the price of the whole packs bought to cover component i's required mass m_i = e_i M_i · 10⁻³ g. A pack is charged in full even when only part of it is used, so this is a procurement cost, not a consumption cost.

Metric · cost tolerance accuracy

CTA@k = 1N Σ_j 𝟙 [ |ĉ_j − c_j| / c_j ≤ k% ]

The share of reactions priced within k% of the label. Refusals, unparsable answers and exhausted step budgets stay in the denominator. We report CTA@10 and CTA@25.

Results

Tool access is necessary, and still not enough.

Twelve models run as ReAct agents with the same four tools and a 40-step budget. The best reach about half the reactions within 25%, below the human reference. Noise in how the reaction is written, especially free-text or OCR-style formatting, pulls every model down further.

Closed models Open-weight Chemistry-specialized Human reference

Full results table: precision, recall and CTA in every setting

Pre. and Rec. compare the components an agent lists with the ones that must be bought. Bold marks the best automated agent in each column; the human row is a reference and is not ranked.

Findings

Where procurement reasoning breaks.

Every trajectory is logged, so a wrong number can be traced to the stage that produced it.

Tool use

Failed runs make more calls, not fewer.

Across models, average tool calls track CTA@25 closely (r = 0.98), mostly because weak models barely enter the workflow. Within strong models, failures come with extra calls, concentrated in supplier-quote retrieval that never converges. GPT-5 fails the other way, stopping early.

Tool calls in failed trajectories relative to successful ones.

Input noise

Formatting is what makes agents give up.

Rewriting a reaction as prose or OCR-damaged text raises abstention by 26.3 points on average. No other single kind of noise adds more than 6. The causes differ by model: exhausted step budgets, unfinished work, or outright refusal.

Clean+Fmt

Reaction structure

Each synthesis step costs accuracy.

Route depth is the strongest difficulty axis. Every model loses roughly 24 to 25 points of CTA@25 going from single-step to deep routes, while the number of components in a single step matters much less.

CTA@25 by number of synthesis steps, pooled across agents (paper setting).

Domain models

Knowing the chemicals is not the same as pricing them.

Chemistry-tuned models name the right components with high precision and still score zero, because they rarely call the tools and estimate prices from memory instead.

ChemDFM-v2.0-14B

93.9

Precision

49.5

Recall

0.0

CTA@25

ChemLLM-20B

89.0

Precision

54.1

Recall

0.0

CTA@25

Clean setting, percent.

Data & code

Run it on your own agent.

The release includes the evaluation harness, every record with its label and selected packs, and the frozen pricing database the labels come from. One script recomputes every label from that database.

# install and fetch data
git clone https://github.com/HeinzVonHank/ChemCost.git
cd ChemCost
pip install -e ".[eval,tests]"
python scripts/download_data.py
python scripts/verify_labels.py

# evaluate a ReAct agent
python scripts/run_evaluation.py \
  --agent react --provider anthropic \
  --model claude-sonnet-4-6 \
  --output results/react.json

Records: 1,427 reactions: 1,359 single-step, 68 multi-step routes
Pricing: Frozen 2026 snapshot of pack-level supplier quotes, shipped as SQLite
Tools: get_supplier_quotes, calculate
Metadata: Croissant, served by Hugging Face
License: Code MIT · data CC BY-SA 4.0

The public release uses a harder view than the paper. Agents see component names, often systematic or less familiar synonyms, and the amounts used in the lab. Roles, equivalents, molecular weights and yields are withheld, and there is no name-search tool. Reference runs on this view:

Model (Codex CLI, two tools)	CTA@10	CTA@25	Multi-step CTA@10
gpt-5.6-luna	64.2	72.7	41.2
gpt-6-luna	47.5	57.7	22.1

Prices are a research snapshot and are not meant for real purchasing decisions.

Cite

BibTeX

@inproceedings{wu2026chemcost,
  title     = {Can Agents Price a Reaction? Evaluating {LLMs} on Chemical Cost Reasoning},
  author    = {Wu, Yuyang and Huang, Yue and Shen, Shuaike and Wang, Xujian and Zhang, Shuhao and
               Xue, Qiyao and Liu, Weichen and Gao, Runtian and Ma, Jian and Zhang, Xiangliang and
               Isayev, Olexandr},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS),
               Evaluations and Datasets Track},
  year      = {2026},
  url       = {https://arxiv.org/abs/2605.07251},
  eprint    = {2605.07251},
  archivePrefix = {arXiv}
}