Evidently AI

Evidently AI is an open-source framework for evaluating, testing, and monitoring large language models (LLMs), RAG applications, AI agents, and predictive machine learning models. It addresses the challenge that AI systems fail in non-deterministic ways, such as through hallucinations, edge cases, data or PII leaks, risky outputs, jailbreaks, data drift, and cascading errors. The framework is released under the Apache 2.0 license.

Its main capabilities include automated evaluation to measure output quality, safety, and reliability, with results viewable through shareable visual reports. It offers synthetic data generation for creating realistic, edge-case, or adversarial inputs. Continuous monitoring is provided via a live dashboard to catch drift, regressions, and emerging risks. The framework includes a library of over 100 built-in metrics and supports custom evaluations using any prompt, model, or rule. It covers adherence to guidelines, hallucinations, factuality, PII detection, retrieval quality, context relevance, sentiment, toxicity, and tone.

Evidently AI is designed for AI teams working with LLM-powered systems or predictive ML models. It is fully open-source and free to use, with no mention of paid editions, trials, or seat-based pricing on the provided pages.