Capstone: fix a broken AI assistant

What this lesson shows

Diagnose the trace, then apply every fix you've learned until it's healthy.

The question it opens with

Here's a shipped assistant failing in five ways. Can you repair it?

The walkthrough, in the lesson's own words

• A shipped assistant is failing. Probe each component to find the broken one.
• Try a repair. A healthy component won't fix a fault that starts upstream.
• Repair the component at the root. Watch the fix flow downstream.
• All roots found and repaired. One last gate before it ships.
• Three more failures, same board. Probe, find each root, repair it yourself.
• 12/12 cases pass. This assistant is ready for the next gate.
• Last gate: prove the repairs hold. Run the eval set before it ships.
• You found five failures and repaired every one at the source.
• Invented run numbers, the shape of the trade-off is what matters.
• Everything downstream is just carrying its bad signal.
• Tap a downstream component and watch nothing change.
• Three more failures are queued. Load each one and repair it on the board above, no hints.
• Four roots found and fixed. One gate left before this assistant ships.
• Same five moves, different root. You localized it and fixed it at the source.
• A healthy component can't fix an upstream fault.
• Five failures, all fixed at the root: a grounded answer, lean context, a gated tool, parseable JSON, and an eval gate on every change.
• That's the whole course in one repair loop. You read the signal flow to localize the fault, then reached for the right fix at the source:
• Real LLM features fail the same handful of ways, and a symptom at the output usually starts upstream. Follow the signal back to the broken stage, then fix it there: ground the answer with retrieval, trim the context, scope the tools, constrain the output, and gate the change on evals. None of it is magic, it's the same few moves every time.
• Your company wants to launch a support bot next month. From what you just fixed, what are the first three things you would demand before launch?
• Retrieval over the real docs (not model memory), an eval set built from real tickets with a gate that must pass, and quota and monitoring so you see failures and costs before customers do.
• Four repairs landed. The fifth failure is quieter: nothing checks this assistant before changes ship.
• Learned by building intuition, not memorizing. No code, no videos.

Key takeaway

You repaired a real-looking assistant end to end: trace, context, retrieval, JSON, permissions, evals.

What you can do after this lesson

You can diagnose a failing LLM feature from its trace and apply the right fixes: trim context, add retrieval, constrain output, set permissions, and gate on evals.

This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.