Senior AI Evals Engineer

Design and build agent evaluation frameworks — create end-to-end infrastructure for evaluating LLM-based agents across correctness, latency, cost, safety, and task completion rate.

Build golden datasets and test harnesses — curate domain-specific evaluation sets for Rubrik's product verticals (data security, SaaS protection, cyber recovery, compliance).

Establish eval-as-a-service — build and enhance centralized evaluation platform that any IT team members can self-serve to test their agents against standardized and custom…

Automate regression and drift detection — implement CI/CD-integrated eval pipelines that catch regressions on every model or prompt change before production deployment.

Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or equivalent practical experience.

Deep hands-on experience with LLM evaluation — familiarity with frameworks and libraries such as LangSmith, DeepEval,LLM as a Judge, Promptfoo, RAGAS…

Strong understanding of agentic AI patterns — tool use, function calling, multi-step planning, ReAct/CoT reasoning chains, retrieval-augmented generation (RAG).

Production experience with eval pipelines — building automated, CI-integrated evaluation infrastructure (not just notebooks).