Kimpton

Kimpton provides evaluation infrastructure for models and agents, running them inside live environments and recording the resulting trajectories for evaluation and training. The recorded observations, actions, rewards, and state transitions form a shared record that can be replayed against a disclosed baseline and used to update a model, prompt, tools, or policy.

The product line includes Koliseum, a service for building private environments and benchmarks, covering long-horizon tasks, multi-agent evaluation, and world models, with task and scoring defined by the client. EvalRouter runs benchmarks and environments across providers through a CLI or API, accepting a model route or an OpenAI-compatible chat endpoint, with versioned evaluations, compatibility and cost review before running, and export to JSON, CSV, or HTML. Adaptive environments generate tasks with persistent world state, where generated events must satisfy explicit rules before changing state and independent graders score observations, actions, and outcomes. Featured environments include Trading Floor and Poker Room.

Kimpton is aimed at teams evaluating models and agents in domains such as finance, life sciences, physical systems, enterprise workflows, cybersecurity, and energy. Environments and benchmarks can be commissioned, with tasks, scoring, and validation criteria specified together with the team. The pages do not state a free plan, trial, or seat-based or usage-based packaging.