Pydantic Logfire combines production AI tracing and evaluation: follow an agent through its model calls, tools, APIs, and database queries, then test a fix against the failures you found. It works with Pydantic AI, LangGraph, the Vercel AI SDK, and other supported frameworks.
The short answer: Pydantic Logfire is the strongest overall choice for teams running AI features inside a real application. One trace covers the model call along with the HTTP request, database query, and validation around it, on OpenTelemetry, with evaluations and an AI gateway in the same platform. Logfire telemetry costs $2 per million additional records, with no separate charge for scores.
Alternatives by situation: Langfuse if you need open source and can operate the stack behind it, LangSmith if you're committed to LangGraph, Braintrust or Confident AI if evaluation is the reason you're adopting observability, Datadog if you're standardized on it and can't add a vendor.
An AI feature passes review, ships, and works fine in the demo. A week later the support queue fills up. The assistant is quoting a refund policy that doesn't exist, an agent is looping on the same tool call until it times out, and your model bill has tripled. You open your logs to find a request ID, a 200 status code, and nothing about what the model actually did.
That gap is the reason AI observability exists. Traditional application monitoring was built to answer "is the service up and fast?" It tracks latency, error rates, and throughput, and it does that well. But none of those metrics tell you whether an answer was correct, why an agent chose the wrong tool, which retrieved document poisoned a response, or how a single user session burned through 40,000 tokens. AI systems fail in ways a green dashboard never shows.
The problem AI observability platforms solve
LLM and agent applications are non-deterministic, multi-step, and expensive to run. A single user request can fan out into a chain of model calls, tool invocations, retrieval queries, validation steps, and downstream API calls. When the output is wrong, the failure could sit anywhere in that chain, and most of those steps leave no trace in a conventional logging setup.
Four problems show up often in production:
-
Silent quality failures. The system returns a confident answer that is wrong. Hallucinations, off-topic responses, and broken structured outputs all pass a status-code health check.
-
Invisible agent behavior. Multi-step agents make decisions you never see. Without a trace of the reasoning loop, a tool-selection bug looks identical to a model that's "just being weird."
-
Runaway cost. Token usage and model spend scale with traffic and prompt size in ways that are hard to predict. Teams routinely discover a 10x cost spike after the invoice arrives.
-
Root causes outside the model. A slow database query, a rate-limited API, or a malformed retrieval result often surfaces as what looks like a model problem. Tools that only watch the LLM call miss the actual cause.
An AI observability platform captures the full execution of every request so you can see what happened, in what order, and why, then attach quality scores and cost data to that record.
The benefits of an AI observability platform
Faster debugging. When you can replay an entire request as a single trace, time to resolution drops from hours to minutes. You stop reproducing bugs by hand and read what already happened in production.
Quality you can measure. Pairing traces with evaluations turns "the output felt off" into a number you can track over time. You catch quality regressions the same way you catch latency regressions, before users do.
Cost control. Per-request token and spend tracking shows which features, prompts, and users drive cost, so you can optimize the expensive 5% instead of guessing.
Confidence to ship. Evaluation gates in CI mean a prompt change or model upgrade gets tested against real cases before it reaches users.
One source of truth. When AI and application telemetry live in the same place, on-call engineers, AI engineers, and product teams argue from the same data instead of three disconnected tools.
What to look for in an AI observability platform
The category filled up fast, and a lot of tools look similar on a feature grid. These are the capabilities that separate a platform you'll still be using at scale.
-
AI-native tracing. Purpose-built capture for LLM calls, agent runs, tool calls, and retrieval steps, with token counts, cost, latency, and model parameters attached to every span. Generic APM tracing bolted onto AI workloads leaves most of this out.
-
Integrated evaluation. A built-in path from production traces to datasets to scored evaluations, so the observe-evaluate-improve loop lives in one platform rather than a separate eval tool you have to wire up.
-
Full-stack depth. The ability to follow one request from the HTTP entry point through the agent, the model calls, the database queries, and the validation layer, in a single trace. AI bugs frequently live outside the model, and a platform that can't see the rest of the stack will send you chasing the wrong layer.
-
Open standards, no lock-in. OpenTelemetry-native ingestion matters because it means your instrumentation is portable. If you decide to leave, your data model goes with you. Proprietary SDKs that only talk to one backend are a long-term liability.
-
Polyglot coverage. Your stack is not one language. Look for first-class SDKs across the languages you actually run, plus standards-based ingestion for everything else.
-
Queryable data. Query raw trace data with a language you already know rather than clicking through a fixed UI. A novel question takes 30 seconds instead of a dashboard hunt.
-
Predictable pricing. Look for transparent published pricing, cost that tracks the data you send, and spend cap control rather than an open-ended bill. Watch for separate meters on evaluation scores, which push teams to sample production coverage down to what they can expense.
-
Production scale. Confirm the platform stays fast at your real trace volume. Query speed and ingestion can behave very differently between a demo and millions of daily spans, so test against your own volume before you build on it.
These are the dimensions we assessed each platform against below.
How we compared these platforms
We assessed each platform on the criteria above, using published documentation, public pricing pages, community discussion in GitHub issues and developer forums, and hands-on use where we have it.
Top AI observability platforms in 2026
The market splits roughly into eval-first platforms, LLM-only tracing tools, open-source self-hosted options, and full-stack platforms.
1. Pydantic Logfire: best overall for production AI applications
Pydantic Logfire provides agent tracing, hosted datasets, and experiment review alongside application observability. It ships native integrations for the common agent frameworks and model SDKs and accepts anything else that emits OpenTelemetry, so using Pydantic AI or Pydantic Validation is optional.
Consider an agent answering an inventory question. A FastAPI request starts the run, the model chooses a search tool, that tool queries the database and calls a supplier API, and the model writes the answer. With those components instrumented and trace context propagated, Logfire shows their spans in one trace. If the answer is slow, compare the model, tool, database, and HTTP span durations. If it is wrong, inspect the tool arguments, returned data, and model messages. A slow query and an incorrect answer need different fixes, and the evidence for both sits in the same place. The AI observability walkthrough follows that execution end to end.
Once you find a useful failure, save it to a hosted dataset, define the expected behavior, and run your current implementation and a candidate fix against the same cases. The evaluation tools guide works a real production failure through that loop. Evaluation isn't limited to offline suites: Pydantic Evals attaches the same evaluators to live traffic, so deterministic checks run on every eligible request while LLM judges sample. Logfire charges nothing per score. Evaluation results are OpenTelemetry events attached to their originating trace, billed as ordinary records, so coverage stays a quality decision rather than a budget one.
- One trace, whole stack: the HTTP request, agent loop, model and tool calls, database queries, and validation appear as nested spans, and the live view streams runs as they happen rather than after they finish.
- SQL and MCP: PostgreSQL-compatible queries over raw trace data, and an MCP server so AI coding assistants read production traces while they debug.
- OpenTelemetry-native and polyglot: first-class SDKs for Python, TypeScript, and Rust, agent framework guides across Go and .NET, and integrations for Pydantic AI, FastAPI, LangChain, LlamaIndex, the Vercel AI SDK, and the OpenAI and Anthropic SDKs.
- Human review: run annotations capture verdicts, expected outputs, and tags from domain experts. Annotations are in beta, and annotation queues for systematic review passes are on the Design Partner plan.
- Prompt optimizer: reviews recent agent runs, prioritizes the ones that raised exceptions, and proposes a single prompt edit with the traces cited as evidence, leaving the decision with you.
- AI gateway: Pydantic AI Gateway adds multi-provider routing, cost limits, and failover from the same account.
- The surrounding stack: Issues group exceptions beside the traces that explain them, managed variables and feature flags and managed prompts version runtime configuration next to the telemetry it affects, SQL alerts route to Slack, Discord, and Microsoft Teams, and SLOs and infrastructure monitoring cover services, hosts, Docker, and Kubernetes.
The free Personal plan includes 10 million records a month, perpetually, with no credit card. Paid plans start at $49 a month, additional usage is a flat $2 per million records, and every paid plan includes a configurable spend cap. Pydantic's pricing comparison puts Logfire at roughly 8x cheaper than Arize AX, 27x cheaper than Langfuse, and 40x cheaper than LangSmith at 5 users and 50 million spans a month. Enterprise plans add self-hosting, SSO, custom retention, and SLAs.
Best for: engineering teams shipping production AI applications who want AI-native depth and full-stack visibility in one platform, on open standards, with transparent, predictable pricing.
2. Langfuse: for open-source self-hosting if you'll operate the stack
Langfuse is an MIT-licensed open-source platform focused on LLM tracing, prompt management, evaluation, and dataset workflows. It self-hosts with no usage limits, which makes it a common pick for teams with strict data-residency requirements or a hard preference for owning their infrastructure. Budget for what that involves: the documented architecture is two application containers plus Postgres, ClickHouse, Redis or Valkey, and an S3-compatible blob store, and Langfuse points production deployments at Kubernetes rather than the single-host Docker Compose setup it suggests for testing. ClickHouse acquired Langfuse in January 2026; both companies committed to keeping it MIT-licensed and self-hostable, though teams evaluating it should still weigh the roadmap uncertainty that follows any acquisition. Cloud plans start around $29 a month. See the full comparison.
Best for: teams that want a self-hosted, open-source LLM tracing and prompt platform and have the infrastructure capacity to operate it.
3. LangSmith: for applications built on LangChain and LangGraph
LangSmith is the observability and evaluation layer from the LangChain team, with the tightest integration into LangChain and LangGraph. LangGraph Studio is an agent development environment for teams in that ecosystem, with a visual interface for inspecting and testing agent runs. The 2026 releases added AI-assisted trace debugging and automatic behavior clustering. It is closed source, self-hosting is enterprise-only, and pricing combines per-seat fees with per-trace charges, which adds up as teams and volume grow. See the full comparison.
Best for: teams committed to the LangChain and LangGraph ecosystem.
4. Braintrust: for teams with an existing eval suite and score history
Braintrust combines tracing, datasets, experiments, and a hosted playground. It accepts traces through standard OpenTelemetry exporters. Its free Starter plan includes 1 GB of processed data and 10,000 scores a month with 14-day retention. Pro runs $249 a month and meters three things: $1.50 per thousand scores after the first fifty thousand, $3 per gigabyte of processed data after the first five, and $0.50 per gigabyte per month of retention past thirty days. Every recorded score counts toward that allowance, online or offline, which is what makes coverage a budget question on evaluation-heavy workloads. Score freely runs those numbers at production volume.
Logfire accepts Braintrust SDK eval runs at a compatible endpoint, verified for Python braintrust 0.30.1 and TypeScript 3.24.0, so an existing suite redirects with two environment variables and switches back the same way. Braintrust-hosted datasets, prompts, functions, BTQL, and the model proxy aren't covered, and other SDK languages are in early access. Fork the loop covers the details. See the full comparison.
Best for: teams whose scorers, datasets, and experiment history already live in Braintrust, and whose score volume stays inside the included allowance.
5. Confident AI: for eval-first teams whose application tracing lives elsewhere
Confident AI is an eval-first, OpenTelemetry-native platform that turns production issues into measurable improvements. Online evaluations score live traces, while signals and monitors surface recurring failures, regressions, and changes in quality, cost, or latency. Domain experts review flagged traces through annotation queues with custom forms and criteria, which aligns automated scoring with human judgment, and reviewed failures become datasets that gate future releases in CI. Starter begins at $200 a month with unlimited seats. The platform is scoped to AI quality rather than application monitoring, which its own guidance frames as complementary to existing infrastructure tooling, so the database queries and service calls behind a failing score sit in a different system.
Best for: teams connecting production monitoring, human review, and evaluation in one quality improvement loop.
6. Arize AI (Phoenix and Arize AX): for evaluation-heavy teams running ML models alongside LLMs
Arize AI ships two products: Phoenix, a source-available, OpenTelemetry-native tool self-hostable under the Elastic License 2.0, and Arize AX, the commercial SaaS layer adding alerts, online evals, RBAC, and enterprise compliance, from $50 a month. Both build on Arize's classical ML monitoring roots, with strength in 50+ research-backed evaluation metrics and drift detection. Dynatrace signed a definitive agreement to acquire Arize in August 2026, and the transaction is still pending close. Phoenix stays source-available, but the commercial roadmap would run through an APM vendor, so weigh the proposed integration and support changes ahead. See the full comparison.
Best for: evaluation-heavy teams, especially those already running ML models alongside LLMs.
7. Datadog LLM Observability: for existing Datadog customers who can't add a vendor
Datadog's LLM Observability is an add-on to its established APM platform. For teams already standardized on Datadog, it adds AI tracing with zero new vendor relationship. The tradeoff is cost: AI observability pricing on Datadog runs well above usage-based specialists, and the LLM features are a recent extension rather than the core of the product. See the full comparison.
Best for: enterprises already invested in Datadog who want AI tracing without adding a vendor.
Other tools worth a look depending on your needs include Helicone and Portkey for gateway-level logging and cost tracking, LangWatch for multi-agent tracing with runtime guardrails, and Galileo and Maxim AI in the evaluation-led workflows. Each covers a slice of the problem rather than the whole stack, so weigh them against what you need to see.
Comparison at a glance
| Platform | Full-stack trace | Evaluation | Open standards | Pricing model | Free tier |
|---|---|---|---|---|---|
| Pydantic Logfire | HTTP, agent, model, tool, database, and validation in one trace | Pydantic Evals, online and offline, no score fee | OTel-native; SQL-queryable; MCP | Tiered + usage, $2/M records | 10M records/month, perpetual |
| Langfuse | LLM spans; OTel ingest for the rest | Scoring hooks and custom evaluators | OTel-compatible | Self-host free; cloud from ~$29/mo | Self-hosted; basic cloud tier |
| LangSmith | LangChain and LangGraph execution tree | LLM-as-judge and annotation queues | Partial | Per-seat + per-trace | 5,000 traces/month |
| Braintrust | AI spans; OTel ingest | Scorers, datasets, experiments; billed per score | OTel-compatible | Platform fee + data + per score | 1 GB/month, 10K scores |
| Confident AI | AI spans only | Configured online evaluations and classifications on live traffic | OTel and OpenInference ingest | Flat plan + $1/GB-month | 1 GB/month, 2 seats |
| Arize AI (Phoenix / AX) | AI spans via OpenInference | 50+ metrics and drift detection | OTel-native (OpenInference) | Phoenix free; AX from $50/mo | Phoenix (source-available); AX free tier |
| Datadog | Full APM plus LLM spans | Managed and custom LLM-as-a-judge evaluations, annotation queues | OTel-compatible | Enterprise, high $/span | Limited trial |
How to choose
If your AI features sit inside a larger application: lead with a platform that traces the whole request. A large share of AI bugs turn out to be a slow database query, a rate-limited API, or a bad retrieval result, and a tool scoped to the model call shows you none of them.
If you need to own your infrastructure: start with Langfuse or Arize Phoenix, which are self-hostable but licensed differently: Langfuse is open source, while Phoenix is source-available under the Elastic License 2.0. Ask two questions before committing: whether you need the code itself or just the deployment, and who on your team will operate the databases and object storage underneath it. Logfire offers self-hosting on its Enterprise tier, though the platform is hosted rather than open source.
If evaluation is the main reason you're adopting observability: Braintrust and Confident AI both build the product around scoring rather than tracing. Check how each one meters scores before you plan production coverage.
If non-engineers need to participate in AI quality: Confident AI supports no-code evaluation workflows. Logfire covers this through run annotations and review queues.
If you live inside LangGraph: LangSmith has native LangChain and LangGraph integrations. Weigh it against how much of your debugging happens below the model call, since Logfire instruments LangChain and LangGraph as well and keeps the surrounding application in the same trace.
If you're already on Datadog and adding a vendor isn't realistic: the add-on covers AI tracing and evaluation, at a cost premium.
For most teams building production AI on an open, polyglot stack who want AI-native depth without giving up visibility into the rest of the application, Pydantic Logfire is the strongest starting point.
Frequently asked questions
What is an AI observability platform?
Software that captures the full execution of AI requests, including LLM calls, agent steps, tool calls, and retrieval, then attaches cost, latency, and quality data so you can debug, evaluate, and monitor production AI systems.
How is AI observability different from traditional monitoring?
Traditional monitoring answers whether a service is up and fast. AI observability answers whether an output was correct, why an agent behaved the way it did, and what each request cost. The failure modes are different, so the tooling is different.
Do I need a separate tool for evaluation?
Not if your observability platform includes it. Pydantic Logfire integrates Pydantic Evals so production traces become evaluation cases in the same platform, which removes a second tool from the loop.
Does evaluation cost extra?
It depends on the platform. Several meter evaluation separately from traces, charging per score recorded, which pushes teams to sample production coverage rather than score everything. Pydantic Logfire charges no separate score fee. Evaluation results are OpenTelemetry events attached to their originating trace and billed as ordinary records at $2 per million. Sampling becomes a question of judge-model spend rather than a platform meter.
Which AI observability platforms are open source?
Langfuse is MIT-licensed and self-hostable, now under ClickHouse ownership. Arize Phoenix is source-available under the Elastic License 2.0. Helicone and Portkey are Apache-2.0 and MIT respectively. Pydantic Evals and the Logfire SDKs are open source, while the Logfire platform is hosted, with self-hosting on the enterprise tier. Fully open-source options give you code transparency and infrastructure control, but you assemble the evaluation and alerting workflow yourself.
What's the difference between an AI gateway and an AI observability platform?
A gateway sits between your application and model providers to handle routing, caching, failover, and spend limits. An observability platform records what happened inside your application and scores it. Many teams run both. Pydantic Logfire includes Pydantic AI Gateway, so routing and telemetry share one account rather than two vendors.
Can product managers and domain experts use an AI observability platform?
On most platforms the workflows assume an engineer. Confident AI supports no-code evaluation workflows for running evaluation cycles. Pydantic Logfire supports this through run annotations and review queues, where domain experts grade runs and that judgment feeds back into datasets and evals.
Is Pydantic Logfire only for Python?
No. Logfire is OpenTelemetry-native, so any language that emits OTel data works, and it ships first-class SDKs for Python, TypeScript, and Rust, with integrations across major AI frameworks.
Can I self-host an AI observability platform?
Yes, though it varies by tool. Langfuse's MIT-licensed core and Arize Phoenix's source-available platform can be self-hosted. Pydantic Logfire runs as a hosted cloud service by default, with self-hosting available on the enterprise tier alongside SSO, custom retention, and SLAs. Because Logfire is OpenTelemetry-native, your instrumentation stays portable either way, so the data model is yours regardless of where it runs.
Does an AI observability platform replace error tracking?
Increasingly, yes. Logfire groups exceptions into Issues with triage states, and each exception links to the trace it occurred in, so the surrounding request context is one click away. For the dedicated comparison, see Logfire vs Sentry and the best Sentry alternatives guide.
What does AI observability cost?
It varies by model. Some platforms charge mainly per seat, some per trace or event, some by data volume, and some add a separate meter on evaluation scores. Logfire's pricing is published in full: a free tier of 10 million records a month, plans from $49 a month, a flat $2 per million records for additional usage, no score fee, and a configurable spend cap on every paid plan so costs stay predictable.
Try Pydantic Logfire free
You can have AI-native and full-stack traces flowing in a few minutes. The free Personal plan includes 10 million records a month. Instrument your app with the SDK for your language, point it at Logfire, and see what your AI is actually doing.
Start free with Pydantic Logfire →
AI is still just engineering.
