README

Evidently

An open-source framework to evaluate, test and monitor ML and LLM-powered systems.






!Evidently



  Documentation
  |
  API Reference
  |
  Discord Community
  |
  Blog
  |
  Twitter
  |
  Evidently Cloud


 :bar_chart: What is Evidently?

Evidently is an open-source Python library to evaluate, test, and monitor ML and LLM systems—from experiments to production.

 🔡 Works with tabular and text data.
 ✨ Supports evals for predictive and generative tasks, from classification to RAG.
 📚 100+ built-in metrics from data drift detection to LLM judges.
 🛠️ Python interface for custom metrics.
 🚦 Both offline evals and live monitoring.
 💻 Open architecture: easily export data and integrate with existing tools.

Evidently is very modular. You can start with one-off evaluations or host a full monitoring service.

 1. Reports and Test Suites

Reports compute and summarize various data, ML and LLM quality evals.
 Start with Presets and built-in metrics or customize.
 Best for experiments, exploratory analysis and debugging.
 View interactive Reports in Python or export as JSON, Python dictionary, HTML, or view in monitoring UI.

Turn any Report into a Test Suite by adding pass/fail conditions.
 Best for regression testing, CI/CD checks, or data validation.
 Zero setup option: auto-generate test conditions from the reference dataset.
 Simple syntax to set test conditions as gt (greater than), lt (less than), etc.

| Reports |
|--|
|!Report example|

 2. Monitoring Dashboard

Monitoring UI service helps visualize metrics and test results over time.

You can choose:
 Self-host the open-source version. Live demo.
 Sign up for Evidently Cloud (Recommended).

Evidently Cloud offers a generous free tier and extra features like dataset and user management, alerting, and no-code evals. Compare OSS vs Cloud.

| Dashboard |
|--|
|!Dashboard example|

 :woman_technologist: Install Evidently

To install from PyPI:

sh
pip install evidently

To install Evidently using the Conda installer, run:

sh
conda install -c conda-forge evidently


 :arrow_forward: Getting started

 Reports

 LLM evals

 This is a simple Hello World. Check the Tutorials for more: LLM evaluation.

Import the necessary components:

python
import pandas as pd
from evidently import Report
from evidently import Dataset, DataDefinition
from evidently.descriptors import Sentiment, TextLength, Contains
from evidently.presets import TextEvals


Create a toy dataset with questions and answers.

python
eval_df = pd.DataFrame([
    ["What is the capital of Japan?", "The capital of Japan is Tokyo."],
    ["Who painted the Mona Lisa?", "Leonardo da Vinci."],
    ["Can you write an essay?", "I'm sorry, but I can't assist with homework."]],
                       columns=["question", "answer"])


Create an Evidently Dataset object and add descriptors: row-level evaluators. We'll check for sentiment of each response, its length and whether it contains words indicative of denial.

python
eval_dataset