AI Agent Product Evaluation Research Intern

About the Role

Develop evaluation methods grounded in product needs and user tasks, informed by the underlying mechanisms of AI agents. Through metric design, experimental analysis and failure case investigation, assess changes in agent capabilities, explore evaluation challenges that existing metrics do not adequately capture, and inform harness improvements and model selection.

Responsibilities

• Contribute to the design of online and offline evaluation metrics, translating user tasks, output quality and user experience into measurable, testable evaluation criteria.
• Design evaluation tasks and experiments around agent planning, tool use, context management and feedback mechanisms to assess the impact of system changes.
• Analyse evaluation results, failure cases and user feedback to distinguish changes in evaluation scores from meaningful changes in agent capabilities and user experience.
• Translate product problems into new evaluation tasks, metrics or experimental methods, and ensure findings inform ongoing product and engineering improvements.

Requirements

• Strong product judgement and sensitivity to user experience, with a focus on how effectively AI agents complete real-world tasks and deliver value to users, rather than evaluation scores alone.
• Skills in experimental design and data analysis, with the ability to independently analyse evaluation results and investigate failure cases.
• Experience with LLM or agent benchmarks, model evaluation or related research.

Preferred Qualifications

• Internship, research or project experience at a leading technology company, university lab or research institute, with hands-on involvement in large language models, AI agents, human–computer interaction or product evaluation.
• Experience building evaluation datasets, designing evaluation criteria, comparing human and automated evaluations, or analysing real-world user tasks.