What you’ll do

Build proprietary benchmarks and datasets to evaluate models and model systems on fraud, identity, and risk workflows
Design and run offline and online evals that measure model performance on real customer tasks, not just abstract benchmarks
Define quality metrics for judgment systems, including precision, calibration, consistency, abstention, and failure handling
Study where models and agents break, and turn those failures into better evals, better datasets, and better training loops
Build reusable evaluation tools and quality building blocks that can be used across different product surfaces and workflows
Partner closely with research, engineering, product, and design to improve system quality through rigorous experimentation
Help create a strong culture of scientific experimentation, clear measurement, and continuous iteration
Push the boundary of how AI systems are evaluated in regulated, adversarial, and high-consequence environments

What success looks like

We have a clear, trusted view of how our systems perform across the workflows that matter most
Our evals predict real-world quality better than generic benchmarks
We identify meaningful failure modes earlier and improve system behavior faster
We develop differentiated datasets, benchmarks, and quality loops that compound over time
Research and engineering teams use your work to make better decisions about what to train, ship, and improve next
Variance becomes known for rigorous, domain-specific evaluation of judgment systems

Preferred background

Experience training, evaluating, or improving modern ML systems
Strong programming skills and comfort working in research-heavy codebases
Experience building benchmarks, datasets, evaluation pipelines, or quality systems
Familiarity with LLMs, agent systems, retrieval, post-training, or adjacent areas
Ability to design clean experiments and draw reliable conclusions from noisy results
Strong engineering judgment and a bias toward building
Interest in fraud, risk, trust and safety, compliance, or other regulated and adversarial domains

Our culture

We believe in ownership, urgency, and craft. We enjoy spirited debate, wild ideas, and building things we’re proud of. We’re fully in-person in San Francisco.

What we offer

Competitive salary and meaningful equity
Platinum-level medical, dental, and vision insurance
Unlimited PTO, sick leave, and parental leave
Up to $100 per month in reimbursement for personal health and wellness expenses
401(k) plan

Research Engineer, Evals