ShopOps
(2026)
An environment for evaluating agents on work that takes more than one step. Instead of picking a label, the agent runs a support queue: twelve typed tool actions, inspect, request evidence, contact the carrier, refund, replace, escalate, close, with state coupled across cases.
The difficulty is in the consequences. Closing a case too early costs something later, and the agent has to trade service-level targets against budget, fraud loss and stock rather than optimise one number. Graders are deterministic, so a run is reproducible.
A rule-based baseline scores 0.92 to 0.98, which sets the bar an agent has to clear to mean anything. Published as a Docker Space on Hugging Face, built on Meta’s OpenEnv.
