Observability: seeing inside
Users say 'the AI is wrong sometimes.' Which calls? Why? You have no idea.
The idea inside
Trace every request: inputs, prompt, output, tool calls, tokens, latency, cost, and errors.
After this lesson
You can explain LLM observability: what to trace and why it's essential in production.
Where it leads
Seeing failures is step one; proving a change is actually better needs evals.
Inside this lesson
That's the real lesson stage, paused. Claim your pass to operate it.
See how AI actually works, end to end.
This lesson is one stop on the full arc. Unlock all of it, and keep it for life.
What you get
- The 34-lesson main path, a finishable route from a word to agents
- Goal tracks for using AI at work and building AI features
- Boss labs that make you apply a whole act, not just recognize it
- Spaced recall that brings each idea back before you forget
- Course memory: every term defined, with links to where it first appears
- A shareable capability card when you finish the main path
- Lifetime access on every device, every future lesson included
Not videos to watch. You predict, operate the machine, then prove it. That is why it stays.
99 interactive lessons and challenges. No videos, no code.
Free launch pass: lifetime access, no card needed
New here? The first lessons are free to try. Start with lesson 0.1
What this lesson shows
Trace every request: inputs, prompt, output, tool calls, tokens, latency, cost, and errors.
The question it opens with
Users say 'the AI is wrong sometimes.' Which calls? Why? You have no idea.
The walkthrough, in the lesson's own words
- One request, opened up: where the time, tokens, and cost go.
- Two identical-looking requests went in. Tap a span to see what's inside.
- Filter the live stream. Spot the failing and slow runs.
- A trace records everything. Now the vague complaint has an address.
- From the outside, every request looks identical, a question in, an answer out. “Sometimes” and “slow” are rumors, not bugs you can open. You can't fix what you can't see.
- One request, split into timed spans. The model call is one bar, but most of the wait.
- This is a trace: one request, split into timed spans (the labelled bars). Tap any bar to see its own data, tokens, latency, cost, status (200 OK = the request succeeded). You'd never guess where the time goes from the outside.
- A user complained the answer was wrong, but it returned a 200 OK (the success code, no error). Which lens would surface that call?
- Errors are loud; slow runs and 👎 are the quiet failures users feel. Only the 👎 catches a wrong-but-successful answer.
- Traces record real user messages, so they carry personal data. Teams limit who can open them and delete them after a set retention window; observability and privacy travel together.
- A demo dazzles, but once it's live the team has no idea why one user's answer came back wrong. What's missing?
- Observability: a trace that records each request end-to-end, its inputs, retrieved context, tool calls, tokens, latency, and output, plus a way to filter by errors, slow runs, and 👎 feedback. With it, a 'sometimes wrong' complaint becomes one specific call you can open, replay, and fix instead of guess at.
Key takeaway
With tracing you can find the bad calls, measure cost and latency, and catch drift.
What you can do after this lesson
You can explain LLM observability: what to trace and why it's essential in production.
Check yourself: Why add tracing / observability to an LLM app?
- You can't fix failures you can't see(correct)
- It makes the model larger
- It removes the need for evals
- It lets you skip guardrails
Traces show what the model actually saw and did. Without them you are debugging blind; you can't fix failures you can't observe.
Where it leads: Seeing failures is step one; proving a change is actually better needs evals.
This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.