phoenix-evals
Lets your agent create and run tests for AI applications. It helps verify if artificial intelligence systems are working correctly.
What it does
This tool helps test and improve artificial intelligence (AI) systems. It allows you to create checks to see if the AI is working well, comparing its results against expectations or human evaluations.
When to use it
- ✓When you want to know if your AI is giving correct answers.
- ✓To compare different versions of your AI and see which one is better.
- ✓To ensure the AI behaves safely and as expected.
Example of what to ask your AI
“How can I test if my customer service chatbot is correctly answering questions?”
Want more? Open the sections below.
How to use
The easiest way: copy the message below and paste it into Claude, ChatGPT, Cursor or any AI. It reads the skill and follows the instructions.
Skill link
Ready-to-paste message
How it works
You define rules or use AI models to judge your application's responses. Then, you run large-scale tests and analyze the results to find and fix errors.
Installation
Easiest way: copy the skill link and paste it into Claude, ChatGPT or Cursor asking it to follow the instructions.
In the terminal (Claude Code, Codex, Cursor), in your project folder:
npx skills add github/awesome-copilot --skill phoenix-evalsOr download just the file:
curl -L https://raw.githubusercontent.com/github/awesome-copilot/main/skills/phoenix-evals/SKILL.md -o SKILL.mdTechnical content (original SKILL.md)
Original text by the author, straight from GitHub.
Phoenix Evals
Build evaluators for AI/LLM applications. Code first, LLM for nuance, validate against humans.
Quick Reference
Workflows
Starting Fresh: observe-tracing-setup → error-analysis → axial-coding → evaluators-overview
Building Evaluator: fundamentals → common-mistakes-python → evaluators-{code|llm}-{python|typescript} → validation-evaluators-{python|typescript}
RAG Systems: evaluators-rag → evaluators-code-* (retrieval) → evaluators-llm-* (faithfulness)
Production: production-overview → production-guardrails → production-continuous
Reference Categories
| Prefix | Description |
|---|---|
fundamentals-* |
Types, scores, anti-patterns |
observe-* |
Tracing, sampling |
error-analysis-* |
Finding failures |
axial-coding-* |
Categorizing failures |
evaluators-* |
Code, LLM, RAG evaluators |
experiments-* |
Datasets, running experiments |
validation-* |
Validating evaluator accuracy against human labels |
production-* |
CI/CD, monitoring |
Key Principles
| Principle | Action |
|---|---|
| Error analysis first | Can't automate what you haven't observed |
| Custom > generic | Build from your failures |
| Code first | Deterministic before LLM |
| Validate judges | >80% TPR/TNR |
| Binary > Likert | Pass/fail, not 1-5 |
FAQ
What are 'evaluators'?
They are checks you create to test specific aspects of your AI, like accuracy, safety, or style.
Do I need to know how to code to use it?
Knowing how to code is helpful, especially for creating more complex checks, but the tool offers options for different skill levels.
What does 'validate against humans' mean?
It means comparing the results of automatic tests with the judgment of real people to ensure the tests are reliable.
Details
Original description by the author
Build and run evaluators for AI/LLM applications using Phoenix.
repo: github/awesome-copilot
Popularity
- GitHub stars
- 40k
- Ranking position
- #5264
- License
- MIT
- Updated
- 10/15/2026
Comments
- …
More skills from this repository
acquire-codebase-knowledge
Helps your agent understand and document an existing code project. It's useful for onboarding new people or getting an overview of the architecture.
github/awesome-copilot
acreadiness-assess
Enables your agent to assess a project's readiness for AI and generate a report. It's useful for understanding how well a project is prepared to use AI tools.
github/awesome-copilot
acreadiness-generate-instructions
Helps your agent create instructions for AI tools based on an assessment. It's useful for improving how AI interacts with code and closing identified gaps.
github/awesome-copilot
acreadiness-policy
Enables your agent to help define or apply policies for AI project assessment. It's useful for standardizing how projects are evaluated and ensuring compliance.
github/awesome-copilot
Similar skills
agent-architecture-audit
Makes your agent check if everything is working correctly. It finds and fixes problems in AI systems, ensuring they perform as expected.
affaan-m/ECC
agent-eval
Helps your agent compare different AI coding tools. It measures which one is faster, cheaper, and more reliable for specific tasks.
affaan-m/ECC
agent-harness-construction
Makes your agent better understand how to perform tasks. It optimizes the tools and instructions the agent uses to increase its chances of success.
affaan-m/ECC