Senior/Staff AI Engineer, Quality & Evals (m/f/x)

Job description

About us We’re Cortea , a Berlin startup transforming audits with AI . Manual, document-heavy audits waste expert time while demand keeps rising. Our AI-powered software and specialized AI agents remove the repetitive work so auditors can focus on judgment. Backed by top-tier VCs with >15m EUR funding, with a working product and paying customers, we’re rapidly scaling. We value first-principles thinking, speed, trust, and kindness . We build side by side in our Berlin office . Your Role We are looking for an engineer with strong backend, data, and AI systems experience to build the evaluation and observability foundation for production-grade LLM agents used in complex audit workflows. This role sits at the intersection of backend engineering, data infrastructure, and AI quality . You will build the evaluation systems that power our multimodal retrieval agents and continuously improve critical quality metrics across current and future pipelines. You’ll work at the edge of applied AI and information retrieval, building multimodal agentic pipelines and solving hard context and agent-harness engineering problems. This is not a traditional analytics, BI, or dashboarding role. You should expect to write production code, design data architecture, work inside backend systems, and directly improve the quality, cost, reliability, and performance of LLM-based agents. What you’ll do You will help build and operate the technical systems around our AI agents, with a focus on data infrastructure, evaluation, observability, and optimization. You will: Build online and offline evaluation systems for LLM agents, including pipelines that use golden datasets, ground-truth data, human review workflows, and experiment results. Create automated quality gates so changes to prompts, context, models, or agent logic can be tested before reaching production. Analyze large volumes of agent traces and executions in columnar and analytical databases such as BigQuery or ClickHouse to identify failure modes, quality regressions, latency issues, reliability gaps, and cost optimization opportunities. Build reliable data retention and replay mechanisms for long-term analysis of production agent behavior. Manage observability tools for tracing, monitoring, debugging, and experiment management of our audit agents. Team up with backend engineers to improve the speed and reliability of our retrie…