LLM Observability Pocket Guide

Pick the right LLM observability stack for your team, budget, and compliance constraints - in a couple of hours, with concrete trade-off reasoning rather than vendor slides.

Your LLM feature is in production, or it is about to be. Traditional APM can't see it regress. Accuracy drifts from 94% to 71% over a month and p99 latency looks fine the whole time. A RAG app quietly returns the wrong tenant's data. An agent gets stuck in a tool-call loop and burns $400 in tokens before anyone notices. You need tracing, evals, and cost tracking - this week, not after you read 80,000 words on the topic.

LLM Observability Pocket Guide is the 2-hour decision guide for backend and platform engineers who have to pick that stack, defend the pick, and ship. Across 15 chapters and five parts, it walks the full landscape as of 2026 - Langfuse, LangSmith, Arize Phoenix, Braintrust, DeepEval, Helicone, and the vendor-neutral OpenTelemetry GenAI + Collector + ClickHouse + Grafana DIY stack - and shows exactly when each earns its place and when it becomes the wrong tool.

What you will take away:
• The three pillars done properly - traces, evals, cost - what each one catches that the others cannot, and what the minimum-viable stack actually looks like.
• The six axes - hosting model, eval depth, developer experience, cost shape, compliance posture, lock-in risk - as the lens every tool chapter uses for head-to-head comparison.
• Twelve team scenarios - pre-seed startup, Series-A AI-native, enterprise SaaS, regulated industry, research lab, agent factory, RAG-heavy, eval-heavy, cross-provider, AI-native mobile, air-gapped on-prem - each walked end to end from constraints to shortlist to pick to exit criteria.
• A master decision tree - plus a buy-vs-build-vs-hybrid matrix with honest $/engineer/month math and three canonical migration patterns for when you need to swap tools.
• The anti-patterns that wreck production - "we'll roll our own," "we'll add evals later," "Datadog for everything" - plus the 40-item production-readiness checklist you print and tape to the wall.

This is the pocket-size companion to the full Observability for LLM Applications handbook (The AI Engineer's Library, Book 1). The handbook is the 80,000-word implementation reference - OpenTelemetry GenAI conventions, tracing patterns for agents and RAG, eval methodology, incident response, the full production checklist. This book is the decision guide that tells you which tool to reach for before you open that handbook. Every chapter ends with a pointer to the matching handbook chapter for implementation depth.

Examples are in Python and TypeScript, with YAML for the OTel Collector configs, version-locked to April 2026 and deliberately framework-agnostic. No vendor advertisement. No hype cycles. Just the trade-off reasoning that separates engineers who pick observability tools by demo video from engineers who pick them on purpose.

Who this book is for: backend and platform engineers picking an LLM observability stack for a team of 2-20 engineers, tech leads and EMs evaluating the space, and anyone who knows what a span is but has never had to compare Langfuse to LangSmith on anything more precise than "vibes."

Other books in Pocket Guides for Developers (standalone, no reading order):
• System Design Fundamentals
• System Design Interviews
• AI Agents Pocket Guide
• Prompt Engineering Pocket Guide
• Database Playbook
• This book - LLM Observability Pocket Guide
• Event-Driven Architecture Pocket Guide
• RAG Pocket Guide

Companion handbook: Observability for LLM Applications (The AI Engineer's Library, Book 1).