We build the infrastructure that frontier AI labs use to evaluate coding agents and language models: benchmark datasets, agent evaluation runs at scale, and automated task quality checks.
Design and ship features end to end: data model, API, background workers, and the UI when the change needs it.
Build and improve how agents and models are evaluated: running trials in sandboxed environments, scoring results…
Improve how benchmark tasks are quality-checked: verifiers, rubric-based and LLM-as-a-judge grading, and deciding whether a task is clear, solvable, and fair to score.
Integrate new models, agents, and benchmarks in a way that generalizes, without a special case for each one.
Bachelor’s degree or higher in Software Engineering, Computer Science, or a related field.
Strong Python and experience with Django (or a similar framework), PostgreSQL, and a task queue such as Celery. What matters most is having run…
Hands-on experience in LLM evaluation, beyond calling a model API. You understand how a benchmark task is built…
Judgment about task quality. You can spot ambiguous instructions, weak or gameable verifiers, leaked answers, and checks that pass for the wrong reason.
Experience: Senior-level · 5+ years
More open roles like this one
Bengaluru, KA, IndiaIndia & South Asia
Canberra, AUST CAPITAL TERR, AustraliaAsia-Pacific
Sydney, NSW, AustraliaAsia-Pacific
Senior Software Engineer, AI Evaluation Infrastructure · Turing
Don't have an account? Already have an account?