What is Agnost AI? Monitoring for chat and voice agents

Updated October 8, 2026 to reflect Agnost AI's broader agent monitoring capabilities.

Agnost AI is a monitoring and product analytics platform for AI agents, including chat and voice agents. It connects production conversations with the events and tool calls behind them so teams can understand what users want, where an agent gets stuck, and which problems keep recurring. MCP server analytics is one of its supported use cases.

An agent can return a successful API response while failing the person using it. It might claim to have completed an action that never happened, repeat an unhelpful answer, or miss a request buried in a conversation. Agnost analyzes those interactions to surface problems that latency and error metrics alone can miss.

How Agnost AI works

Agnost collects data through its SDKs, framework integrations, or an OpenTelemetry exporter. Teams can send agent interactions from their application, connect voice agent telemetry, or instrument an MCP server.

The platform organizes that data into conversations, events, and tool calls. A conversation groups related interactions into a timeline; events hold the underlying operations; tool calls show what a tool did and how long it took. The conversation view, labeled User Stories in the dashboard, lets you inspect messages alongside spans, errors, and available audio.

Conversation identity matters. Reuse a stable identifier across turns that belong together so an agent's follow-up responses stay in the same timeline. Keep unrelated users and sessions separate.

Monitoring chat and voice agents

Agnost applies conversation analysis to both text and voice interactions. Its integration guides cover agent frameworks and voice platforms, including LiveKit, Vapi, Telnyx, and Trellus. The useful question is what happened across the user's whole interaction, including the tools the agent called along the way.

User intent and unmet requests

Intents describe what users are trying to accomplish. Teams can define intents for their product or generate suggestions from its description and observed traffic, then inspect the conversations behind each match. This helps reveal recurring needs and requests that a click-based analytics system would not capture.

An intent match does not establish that the user succeeded. Read the supporting exchange and compare it with the outcome and tool activity before deciding what to build or fix.

Silent failures and user frustration

Repeated questions, corrections, and abandoned conversations can expose failures even when individual model calls completed successfully. Agnost connects these signals to the messages and traces behind them, giving a team a way to investigate what went wrong and how often it happens.

For example, if users keep correcting an agent's answer, the conversation provides evidence of the problem. The related tool activity can help explain whether the agent retrieved the wrong information, failed to act on it, or never called the expected tool.

Expected behavior and alerts

Teams can define rules for expected agent behavior and review violations with the supporting messages. These findings help identify conversations that need attention; application code still needs to enforce authorization and other critical checks.

Alerts can send production signals to Slack or email. An intent or violation can become a threshold alert with a chosen lookback window and schedule, so teams can track a recurring issue after they identify it.

MCP server analytics

Agnost also provides MCP server analytics. Instrumenting a server records tool names, inputs and outputs, latency, outcomes, and conversation context. The Tool Calls view lets teams compare usage, success rates, error distribution, and latency over time.

Server instrumentation can capture calls from the clients using that server. Application-level OpenTelemetry captures the calls made by that application. Choose the boundary that answers your debugging question and avoid capturing the same operation twice unintentionally.

This is useful when a slow or failing tool is part of a larger agent conversation. You can investigate the tool's execution while keeping the user's request and the agent's response in view.

Integration options

Agnost's published packages cover several integration paths:

• Custom chat or agent applications can use the Python conversation SDK (pip install agnost) or the TypeScript conversation SDK (npm install agnostai).
• Applications that already emit OpenTelemetry spans can export them to Agnost's hosted collector. Include conversation identifiers and the message content needed for the analysis you want.
• MCP servers have dedicated SDKs for Python FastMCP, TypeScript/Node, and Go. These use different packages from the conversation SDKs.
• Voice agent teams can follow the guide for their platform to connect conversation and execution data.

The conversation SDKs provide begin() and end() calls around an AI interaction and calculate its latency. They also support user identifiers and custom properties, which help distinguish agent versions or other application context when investigating an issue.

Getting started

Create an account at app.agnost.ai and find your organization ID under Settings > Organization. Follow the quickstart or the guide for your framework. Agnost also provides a coding-agent skill that helps select, install, and verify the appropriate instrumentation.

Trigger a real interaction in your application, then check Events and the relevant conversation or tool-call view in the dashboard. Start with one integration path for each call so duplicate telemetry does not distort the results.

Once data is flowing, inspect a recurring user need or failure and open the conversations behind it. Agnost's Auto Improve workflow can investigate a finding, gather repository context, and prepare a proposed change. Review the evidence and any resulting diff, test it, and deploy through your team's normal process. Continue monitoring the outcome after the change ships.

‍