Open position
Evaluation and measurement
Own the checks and measurements behind every claim we make: what to measure, how, and how much evidence is enough.
What you'll do
- Design the automatic checks that decide whether an output is accepted.
- Build the benchmarks and dashboards the research and serving teams decide with.
- Run human evaluations where automatic checks fall short, and measure how far the two agree.
- Review claims before they go public, and say when the evidence is not enough yet.
What you bring
- Experience designing evaluations or experiments for machine-learning systems.
- Solid statistics: sample sizes, confidence and the ways a metric can mislead.
- Python, and the patience to look at a lot of outputs.
Good to have
- Experience evaluating image, video or audio generation.
- Published work on evaluation or metrics.