The Data Scientist anchors the evaluations function for this vertical — designing, building, and maintaining evaluation frameworks that measure model and system quality in operational context.
Design and build evaluation frameworks for Sarvam's AI outputs across domain-specific requirements: document comprehension, command summarisation, geospatial reasoning, enterprise workflow automation…
Define quality metrics in collaboration with domain experts and clients; translate operational requirements into measurable, defensible signals.
Run structured evaluation cycles pre- and post-deployment; build dashboards that surface model quality in production.
Identify failure modes, edge cases, and distribution shifts — with the bias of someone looking for what's wrong, not confirming what's right.
Strong statistics and probability fundamentals — you understand what makes an evaluation valid and what makes it misleading.
Experience designing evaluation frameworks from scratch: custom metrics, inter-rater reliability, red-teaming methodologies.
Python proficiency; comfort with pandas, NumPy, HuggingFace datasets, RAGAS, EleutherAI Eval Harness, LangSmith, or equivalent.
Experience with prompt engineering, model fine-tuning, or RLHF in applied settings.
Experience: Entry-level · 3–6 years · Job type: Full-time
More open roles like this one
Remote · San FranciscoNorth America
Data Scientist - Evaluations, Chanakya · Sarvam AI
Don't have an account? Already have an account?