Where we are: evals, observability and the AI Engineer job (2025 to 2026)

On 30 June 2023 Shawn Wang, known as swyx, published The Rise of the AI Engineer and gave a name to a job that already existed: someone who builds products on top of models they did not train, with prompts, tools, retrieval and evals instead of gradients. By 2025 it was a title on job boards. This course is for the interview for that job.

The pieces from the last eleven lessons all work. Instruction following, tool calling, retrieval, agents in a loop, standard plugs for tools. The products that use them are real: coding agents that open pull requests, support agents that resolve tickets, research agents that read fifty pages for you. The public benchmarks say the same. On SWE-bench Verified, the 500 problem human checked subset that OpenAI released in August 2024, GPT-4o solved 33.2 percent at launch. In May 2025 Anthropic reported 72.5 percent for Claude Opus 4. METR measured the length of software tasks that models can finish with 50 percent reliability and found it doubling about every seven months since 2019.

The same benchmarks tell the other half. On tau-bench, Sierra's 2024 test of agents talking to simulated customers while following a policy and calling tools, the best function calling agents at the time succeeded on under half the tasks. When the same task was run eight times, the share that passed every time was under a quarter. The agent that works on Tuesday and fails on Wednesday on the same input is the normal case, not the exception. Every lesson in Parts 1 to 5 of this course is one of these failures, with a trace.

So the work in 2025 and 2026 is not getting the model to do the thing. It is knowing when it did not, finding out why from the trace, and proving the fix held. Three practices came out of that.

Observability. Every run is a tree of spans: the user turn, each model call with its prompt and output, each tool call with its arguments and result, nested as they happened. OpenTelemetry has semantic conventions for generative AI that name these spans and their attributes, and the products that show you the tree all read the same shape: Braintrust, Langfuse, Arize Phoenix, LangSmith and others. Part 5 opens with one of these trees, span by span. If you cannot see the trace, you cannot do the job.

Evals. An eval is a set of inputs with a way to score the output. The score can be code: a string match, a JSON Schema check, a test that runs. Or it can be a model given a rubric, a judge, which you must check against your own labels before you trust it. Hamel Husain's 2024 essay Your AI Product Needs Evals is where most teams start, and its advice is to look at your data, write the check that would have caught last week's bug, and run it on every change. The interview question that comes up most is how big was your eval set and where did the cases come from. The good answer is a number, a source, and a story about a case that was hard to label.

Guardrails and the human. Prompt injection from lesson 0.7 is still unsolved. The practical answers are least privilege for tools, approval gates for anything that cannot be undone, and the assumption that any text the agent reads may be hostile. Part 2 of this course is that round of the interview.

The history in this part is a method waiting on its inputs, then on its objective, then on its plumbing. Now the method, the inputs and the plumbing all exist. What is scarce is engineers who can read a trace, name the line where it broke, write the gate, and prove it with an eval. When an interviewer asks where the field is, say that: the building blocks are solved and the reliability is not, and the job is the gap between them. That is the whole rest of this course. Start with Part 1.