Agnost AI

Agnost AI is a San Francisco-based developer tools and analytics company that positions itself as product analytics for conversational AI agents. Its platform ingests production conversations and traces from a company's AI agents, reads each trace alongside the underlying conversation, and maps activity back to individual users in order to detect "silent failures" — cases where a trace reports success but the user received nothing useful, such as an agent claiming to have sent a document that never arrived or confidently answering the wrong question. Product capabilities described on the company site include automatic discovery of intents and sentiment, auto-clustering of conversations into recurring problems ranked by impact, detection of hallucinations, broken promises, and quality, policy, and compliance violations, alerts for silent failures and rising friction, and self-improvement suggestions for the agent [0][1][2].

A second product line builds on the same data: Agnost uses historical production traces to identify recurring workflows an agent performs, generates evaluation sets from held-out traces, and trains workload-specific "custom" or specialist models for narrow, repeatable jobs such as request routing, tool selection, data retrieval, and response formatting. The company frames this as complementary to, not a replacement for, frontier models, which it says remain appropriate for open-ended work. Customers are benchmarked against their existing model on task success, latency, and cost before considering a switch [0][1][2].

Integration is designed to avoid re-architecting the agent: traces can be sent via OpenTelemetry or Agnost's SDKs, and a published skill package supports installation through a CLI command. SDKs cover conversation/AI interaction tracking (Python package `agnost`, TypeScript package `agnostai`) and Model Context Protocol server analytics (`agnost-mcp` for Python, `agnost` for npm, and a Go module). The API exposes session-creation and event-capture endpoints, with methods for initialization, begin/end interaction tracking with automatic latency calculation, single-call tracking, user identification/enrichment, and flush/shutdown. On security, the company states it processes only the conversation data customers choose to send, uses HTTPS transport and authenticated dashboard access, and explicitly does not perform automatic PII redaction, advising customers to use pseudonymous IDs and redact sensitive fields before ingestion [0][1][3].