arize-evaluator
Helps your agent set up and run LLM evaluations using a model as a judge. It manages tasks and continuously monitors performance.
What it does
This tool helps set up and run automatic evaluations of language models (AI) using another AI as a judge. It allows you to define how the AI judge should assess responses, such as checking if they are truthful or relevant, and then runs this evaluation on your data.
When to use it
- ✓When you want to check if the AI is 'making up' information (hallucination).
- ✓To ensure AI responses are correct and relevant to the question.
- ✓When comparing different versions of an AI model to see which one is better.
Example of what to ask your AI
“Set up an evaluator to check if AI responses are faithful to the provided context.”
Want more? Open the sections below.
How to use
The easiest way: copy the message below and paste it into Claude, ChatGPT, Cursor or any AI. It reads the skill and follows the instructions.
Skill link
Ready-to-paste message
How it works
You define what the AI judge should look for (the 'evaluator') and how it should check the data (the 'column mapping'). Then, you run a 'task' that applies this evaluator to your data, generating scores and explanations.
Installation
Easiest way: copy the skill link and paste it into Claude, ChatGPT or Cursor asking it to follow the instructions.
In the terminal (Claude Code, Codex, Cursor), in your project folder:
npx skills add github/awesome-copilot --skill arize-evaluatorOr download just the file:
curl -L https://raw.githubusercontent.com/github/awesome-copilot/main/skills/arize-evaluator/SKILL.md -o SKILL.mdTechnical content (original SKILL.md)
Original text by the author, straight from GitHub.
Arize Evaluator Skill
SPACE— All--spaceflags and theARIZE_SPACEenv var accept a space name (e.g.,my-workspace) or a base64 space ID (e.g.,U3BhY2U6...). Find yours withax spaces list.
This skill covers designing, creating, and running LLM-as-judge evaluators on Arize. An evaluator defines the judge; a task is how you run it against real data.
Prerequisites
Proceed directly with the task — run the ax command you need. Do NOT check versions, env vars, or profiles upfront.
If an ax command fails, troubleshoot based on the error:
command not foundor version error → see references/ax-setup.md401 Unauthorized/ missing API key → runax profiles showto inspect the current profile. If the profile is missing or the API key is wrong, follow references/ax-profiles.md to create/update it. If the user doesn't have their key, direct them to https://app.arize.com/admin > API Keys- Space unknown → run
ax spaces listto pick by name, or ask the user - LLM provider call fails (missing OPENAI_API_KEY / ANTHROPIC_API_KEY) → run
ax ai-integrations list --space SPACEto check for platform-managed credentials. If none exist, ask the user to provide the key or create an integration via the arize-ai-provider-integration skill - Security: Never read
.envfiles or search the filesystem for credentials. Useax profilesfor Arize credentials andax ai-integrationsfor LLM provider keys. If credentials are not available through these channels, ask the user. - CRITICAL — Never fabricate evaluation results: If an evaluation task fails, is cancelled, or produces no scores, report the failure clearly and explain what went wrong. Do NOT perform a "manual evaluation," invent quality scores, estimate percentages, or present any agent-generated analysis as if it came from the Arize evaluation system. Instead suggest: (1) fix the identified issue and retry, (2) try running from the Arize UI, (3) verify integration credentials with
ax ai-integrations list, (4) contact support at https://arize.com/support
Concepts
What is an Evaluator?
An evaluator is an LLM-as-judge definition. It contains:
| Field | Description |
|---|---|
| Template | The judge prompt. Uses {variable} placeholders (e.g. {input}, {output}, {context}) that get filled in at run time via a task's column mappings. |
| Classification choices | The set of allowed output labels (e.g. factual / hallucinated). Binary is the default and most common. Each choice can optionally carry a numeric score. |
| AI Integration | Stored LLM provider credentials (OpenAI, Anthropic, Bedrock, etc.) the evaluator uses to call the judge model. |
| Model | The specific judge model (e.g. gpt-4o, claude-sonnet-4-5). |
| Invocation params | Optional JSON of model settings like {"temperature": 0}. Low temperature is recommended for reproducibility. |
| Optimization direction | Whether higher scores are better (maximize) or worse (minimize). Sets how the UI renders trends. |
| Data granularity | Whether the evaluator runs at the span, trace, or session level. Most evaluators run at the span level. |
Evaluators are versioned — every prompt or model change creates a new immutable version. The most recent version is active.
What is a Task?
A task is how you run one or more evaluators against real data. Tasks are attached to a project (live traces/spans) or a dataset (experiment runs). A task contains:
| Field | Description |
|---|---|
| Evaluators | List of evaluators to run. You can run multiple in one task. |
| Column mappings | Maps each evaluator's template variables to actual field paths on spans or experiment runs (e.g. "input" → "attributes.input.value"). This is what makes evaluators portable across projects and experiments. |
| Query filter | SQL-style expression to select which spans/runs to evaluate (e.g. "span_kind = 'LLM'"). Optional but important for precision. |
| Continuous | For project tasks: whether to automatically score new spans as they arrive. |
| Sampling rate | For continuous project tasks: fraction of new spans to evaluate (0–1). |
Data Granularity
…
FAQ
What is an 'evaluator'?
It's the configuration for how an AI will judge another AI's responses. It includes the 'prompt' (instruction) for the AI judge, the possible outcomes (e.g., 'correct', 'incorrect'), and which AI model to use.
What is a 'task'?
It's the action of running one or more evaluators on your data. A task defines which evaluators to use, how to connect information from your data to the evaluator's instructions, and which specific data to evaluate.
How does the tool handle errors or missing results?
If an evaluation fails, the tool clearly reports the issue and does not invent results. It suggests fixing the problem, trying through the Arize web interface, verifying credentials, or contacting support.
Details
Original description by the author
Handles LLM-as-judge evaluation workflows on Arize including creating/updating evaluators, running evaluations on spans or experiments, managing tasks, trigger-run operations, column mapping, and continuous monitoring. Use when the user mentions create evaluator, LLM judge, hallucination, faithfulness, correctness, relevance, run eval, score spans, score experiment, trigger-run, column mapping, co
repo: github/awesome-copilot
Popularity
- GitHub stars
- 40k
- Ranking position
- #5003
- License
- MIT
- Updated
- 10/15/2026
Comments
- …
More skills from this repository
acquire-codebase-knowledge
Helps your agent understand and document an existing code project. It's useful for onboarding new people or getting an overview of the architecture.
github/awesome-copilot
acreadiness-assess
Enables your agent to assess a project's readiness for AI and generate a report. It's useful for understanding how well a project is prepared to use AI tools.
github/awesome-copilot
acreadiness-generate-instructions
Helps your agent create instructions for AI tools based on an assessment. It's useful for improving how AI interacts with code and closing identified gaps.
github/awesome-copilot
acreadiness-policy
Enables your agent to help define or apply policies for AI project assessment. It's useful for standardizing how projects are evaluated and ensuring compliance.
github/awesome-copilot
Similar skills
speech
Helps your agent turn text into spoken audio. It's useful for creating narration, lecture audio, or quick voiceovers.
nexu-io/open-design
Docker Hub Automation
Makes your agent automate operations on Docker Hub. It manages repositories, teams, and webhooks for your containers.
ComposioHQ/awesome-claude-skills
New Relic Automation
Helps your agent automate tasks in New Relic. It can manage alerts, monitor applications, and set up notifications.
ComposioHQ/awesome-claude-skills