Chordio

Chordio is a product experience lab whose main work is PX-bench, a long-horizon benchmark that evaluates the product experience decisions coding agents make when adding features to existing products. It addresses the difficulty of measuring what agents ship when no designer reviews the work, since product experience is hard to grade quickly and at scale. The benchmark scores results across eight categories: intent fidelity, product fit, visual craft, convention adherence, pathway completeness, content and language, resilience, and accessibility.

PX-bench runs agents against held-out host apps they have not seen, each with its own conventions, and scores outcomes against expert-defined ground truth using quasi-objective rubrics. Scenarios include known failure modes such as implied screens, state loss on navigation, and ambiguous primary actions. The harness is Inspect AI, the UK AI Safety Institute's framework, so published scenarios can be independently rerun. Teams building coding agents can run private evaluations, receiving all eight category scores for their agent and harness. Chordio publishes its methodology as research notes, and the taxonomy is versioned with revisions stated as diffs.