AI Evals Engineer

A machine learned to tell shapes apart from examples in 1958, and a book of proofs in 1969 showed what one layer could never learn.

A four page paper in Nature showed how to train the layer in the middle, and then the field spent twenty years waiting for data and compute.

A 1986 method on two consumer graphics cards beat every hand built vision system by eleven points, because the data had finally arrived.

Two LSTMs learned to translate, attention fixed the bottleneck, and then a 2017 paper threw away the recurrence and kept the attention.

Train a transformer to predict the next word on raw text, make it a hundred times bigger, and tasks it was never trained for show up in the prompt.

A 1.3 billion parameter model trained on human preferences beat a 175 billion parameter one, because the problem was the objective, not the knowledge.

Eight worked examples took a math benchmark from 18 to 57 percent, one sentence did most of that alone, and the prompt became code with a bug class of its own.

ReAct showed a model could write an action in text, every parser broke, and in June 2023 the call became a typed message the model was trained to make.

The window got long enough for a novel, a Stanford paper showed the model reads the middle of it badly, and memory turned out to be retrieval with a write path.

Auto-GPT put a model in a loop and let it go, the loop went in circles, and the agents that worked a year later had checkable results and stop conditions written in code.

Splitting work across agents won on reading and lost on writing, and two protocols gave the model a standard plug for tools and for other agents.

The pieces all work, the same agent passes on Tuesday and fails on Wednesday, and the job is now reading the trace, writing the gate and proving the fix with an eval.

One prompt did three jobs, got the first one wrong, and nothing checked the result before the reply went out.

A draft turned a 0.3 point rise into 30 percent, and nothing read the draft against the source before it was posted.

The double charge run again, this time as the tree of spans every tracing platform shows, with the attribute names that make it readable.

A deterministic check is a plain function that fails on the exact bug you saw, runs on every prompt change, and never asks a model for its opinion.

When the question is a judgment, a model can grade it, but only after you have labeled the cases yourself and measured how often the judge agrees with you.

A factual check is a veto, a judge is an opinion, and a pipeline that averages the two will ship a wrong number wrapped in a nice reply.

An eval that flips without a code change is measuring the clock, the sampler or the vendor, and the fix is to pin what you can and count what you cannot.

Where the cases come from, who labels them, and why the forty you tune against and the forty you report on must never be the same forty.

Your agent already emits OpenTelemetry spans. This lesson wires them into Braintrust and into Arize Phoenix, then does the four verbs every platform does: log, collect, score, compare.

The failure you found, the safe option you chose, the thing you cut. Each told in ninety seconds, past tense, with a line number and a number that moved.

What happens at /interview minute by minute, how each of the ten points per question is earned, and how to walk in ready.