Research Engineer, Benchmarks

About the Role

Join a small, technically elite team — including International Olympiad medalists and published AI researchers — building high-quality benchmarks to evaluate frontier AI agents on realistic, domain-specific workflows. As a Research Engineer, Benchmarks, you'll own the design and implementation of evaluations that frontier labs and enterprise customers trust. This role is central to ensuring our benchmarks are rigorous, credible, and tightly aligned with real-world agent performance.

This is an on-site role based in San Francisco, CA. Visa sponsorship is available.

What You'll Do

• Design, implement, and own the quality of internal benchmarks for evaluating frontier agents on domain-specific tasks.

• Partner with subject-matter experts to define realistic workflows and tasks for domain-specific evaluations.

• Build reliable infrastructure to run models and agents against benchmark tasks at scale.

• Develop metrics and analyses that measure benchmark difficulty, reliability, and failure modes.

• Validate that benchmark performance correlates with real-world evaluations, customer needs, and frontier lab expectations.

• Write clear documentation and benchmark reports that make results legible and credible to technical audiences.

What We're Looking For

Required

• 2–4 years of experience in software engineering, ML engineering, or research roles.

• Strong proficiency in Python, Docker, and Linux environments.

• Experience building environments, evaluations, or benchmarks for AI systems.

• Published research or technical writing on topics such as public benchmarks, model failure modes, or evaluation methodology.

• Deep understanding of what makes a benchmark realistic, reliable, and practically useful.

• Curiosity and genuine ability to understand how real-world workflows operate across diverse domains.

• Strong attention to detail — a habit of spotting subtle inconsistencies and edge cases in task design.

• Ability to reason from first principles about task design, scoring, and failure modes.

• Comfort thriving in unstructured problem spaces and working independently in fast-paced, early-stage environments.

• Excellent communication skills for collaborating across time zones and with technical teams.

Compensation & Benefits

• Salary: $150,000 – $250,000 USD annually, depending on experience.

• Visa sponsorship available.

• Opportunity for significant early-stage equity and career growth within a high-impact, research-driven team.

Location

This is a full-time, on-site position in San Francisco, CA. Candidates must be willing and able to work in-office.