As the Tech Lead Manager of AI Evaluations, you will lead a team of applied AI scientists and engineers and build the evaluation strategy and platform that helps Glean ship delightful and reliable AI experiences.
Own the AI evaluation eng roadmap and technical direction end to end: from defining what success means and building the evaluation platform to integrating…
Build scalable, reusable evaluation infrastructure for datasets, experiments, graders, trace capture and replay, result analysis, and dashboards. Make evaluations reproducible, versioned, cost-aware…
Develop a fit-for-purpose mix of deterministic checks, model-based graders (including LLM-based autograders), and human evaluation. Establish clear rubrics…
Make quality tradeoffs measurable by connecting evaluation outcomes to latency, reliability, and cost. Help teams diagnose failures by slice and trace, prioritize improvements…
Hands-on experience designing, building, or operating evaluation systems for LLM/GenAI products, including model-based evaluators or autograders; practical understanding of how to assess model…
Familiarity with evaluation methods and tradeoffs: curated and production-derived datasets, benchmark creation, deterministic metrics, rubric-based and pairwise judgments, human review…
Understanding of agent evaluation, including tool calling, multi-turn behavior, trajectory quality, task completion, and recovery from errors…
Strong coding ability in Python or another production language, with sound engineering fundamentals in testing, APIs, data modeling, and reliable distributed systems.
Experience: Senior-level · 10+ years
More open roles like this one
Bengaluru, KA, IndiaIndia & South Asia
Canberra, AUST CAPITAL TERR, AustraliaAsia-Pacific
Sydney, NSW, AustraliaAsia-Pacific
Tech Lead Manager, AI Evaluations · Glean
Don't have an account? Already have an account?