The shape of a trace: OpenTelemetry, span by span

In lesson 1.1 the double charge run was six lines: a ticket, one prompt, a wrong extraction, a reply, an email, status success. That is how a person reads it. A tracing platform stores the same run as a tree. The root is the whole agent call. Under it, one child for the model call and one child for the email tool. Each node is a span.

A span is a timed piece of work with a name, a start, an end, a status and a bag of key value pairs called attributes. Every span carries a trace id (the same 32 hex characters for the whole run), its own span id (16 hex characters) and the span id of its parent. That parent link is what turns a flat list of spans into a tree. The root has no parent. The model call's parent is the root. If the tool had called a model of its own, that span's parent would be the tool span, one level deeper.

The trace below is the double charge run written as that tree. Read the indentation as parent and child. Read the words with dots in them as attributes.

OpenTelemetry has a set of agreed attribute names for model calls, the GenAI semantic conventions. They live in their own repository now, open-telemetry/semantic-conventions-genai, and are marked Development, which means names can still change. The names that matter for reading a trace are few. gen_ai.operation.name says what kind of span it is: chat, execute_tool, invoke_agent, embeddings, retrieval. gen_ai.provider.name and gen_ai.request.model say which model. gen_ai.usage.input_tokens and gen_ai.usage.output_tokens say what it cost. gen_ai.response.finish_reasons says why the model stopped. On tool spans, gen_ai.tool.name and gen_ai.tool.call.id tie the call to the model turn that asked for it.

Span names follow the same rule: the operation, a space, the model or tool. So a chat span is named "chat gemini-2.5-flash", a tool span "execute_tool send_email", the root "invoke_agent support-agent". When you see those names in a platform, that is the convention at work.

One rule to remember: the prompt and the completion are not recorded by default. They are sensitive and large, so the conventions make gen_ai.input.messages, gen_ai.output.messages and gen_ai.tool.call.arguments opt in, gated by an environment variable such as OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT. A trace without the content tells you the shape of the run and its cost. A trace with the content is the one you can debug. Decide which one you are storing before the incident, not during it.

Braintrust, Langfuse, Arize Phoenix and LangSmith all accept spans over the OpenTelemetry wire protocol, and all four draw the same picture: a root, children indented under it, a duration bar per span, attributes in a side panel. Langfuse calls the nodes observations and maps the chat span to a generation. Phoenix reads a sister convention called OpenInference, where the kind is an attribute named openinference.span.kind with values like LLM, TOOL, CHAIN and AGENT, and the token counts are llm.token_count.prompt and llm.token_count.completion. Different words, same tree. If you can read one, you can read all of them.

This is why the question "what did you use to see inside the agent" has an answer that does not depend on the product. The answer is: spans, with the GenAI attributes, sent to whichever platform the team already pays for.

Look at the chat span in the trace. Its status is OK. Its finish reason is stop. Its token counts are normal. Nothing about the span's shape says anything went wrong, because the span status only says whether the call completed. The bug is inside the content attribute: gen_ai.output.messages holds the order number 82813 for a ticket that said 88213. If content capture had been off, this trace would look identical to a correct run.

That is the whole lesson in one sentence. The tree tells you where to look. The attributes tell you what happened. A check, which is lesson 5.1, is what turns "I can see it" into "it cannot ship".

"A trace is a tree of spans. Each span has a name, a duration, a status, attributes and a parent id. For model calls I use the OpenTelemetry GenAI attributes: operation name, model, token usage, finish reason, tool name. Content is opt in and we turn it on for the agents we are still debugging. Any of the platforms will draw that tree; we send it to the one the team already has." Then name one real span from a run you fixed.