SDE2 - API Platform

Building and evolving the LLM API platform, from the moment a request hits the edge to the response that goes back out.

The Python Go rewrite: building services in Go alongside the existing Python stack, without compromising reliability.

SSE for Chat Completions and Responses, long-lived connections, clean handling of cancellations and client disconnects.

Routing and failover: weighted provider chains, retries with backoff, and latency-driven admission control across model deployments.

Strong Python in production: asyncio, FastAPI or similar, and services you've built and operated under real load.

Working knowledge of Go, or a strong willingness to learn it: you'll work in both languages through the rewrite.

Experience building low-latency, multi-tenant backend services.

Hands-on experience with streaming systems: SSE or WebSockets, long-lived connections, backpressure.