Speedrun Talent Network

Senior AI Evaluation Product Manager

Location
Hayes Valley, San Francisco
Employment
Full Time
View job and apply

Job summary

Hercules is seeking a senior or staff level product manager to own evals for our AI coding agent. You will own Hercules Bench offline evals and Hercules online evals, guiding evaluation design and data-driven improvements. You will leverage our 500k users, 1M code repos, and millions of chat threads to craft effective evals while anonymizing data to protect privacy. We ship, strive for excellence, and stay hungry for impact. J-18808-Ljbffr

SalaryCheck the source listing for salary
General interview preparation

General preparation guidance from MiPrep; these interview stages are not confirmed for this job.

  • Recruiter phone screen - typically 20-40 min
  • Technical screen - 1 coding problem or take-home; format varies by company
  • Onsite / virtual loop - typically 4-6 rounds over 2-3 weeks; exact split of coding, systems design, and behavioral varies by role and location