Skip to content
Skillpedia
PT
Prompts & LLMs40k · #5007

arize-prompt-optimization

Helps your agent optimize and improve LLM prompts using production data and evaluations. It uses a data-driven optimization loop to enhance responses.

What it does

This tool helps improve AI responses. It analyzes how questions (prompts) were asked and suggests how to write them better for more accurate results.

When to use it

  • ✓When AI responses are not good.
  • ✓To make the AI more efficient.
  • ✓When creating new instructions for the AI.

Example of what to ask your AI

“How can I make the AI respond better to my questions?”

Want more? Open the sections below.

How to use

The easiest way: copy the message below and paste it into Claude, ChatGPT, Cursor or any AI. It reads the skill and follows the instructions.

Skill link

https://github.com/github/awesome-copilot/blob/main/skills/arize-prompt-optimization/SKILL.md

Ready-to-paste message

Use this skill to help me: https://github.com/github/awesome-copilot/blob/main/skills/arize-prompt-optimization/SKILL.md Read the SKILL.md file at that link and follow its instructions for my task.
View on GitHub ↗Raw file
How it works

It looks at data from how the AI responded before and uses that information to adjust the instructions. This is done automatically to optimize performance.

Installation

Easiest way: copy the skill link and paste it into Claude, ChatGPT or Cursor asking it to follow the instructions.

In the terminal (Claude Code, Codex, Cursor), in your project folder:

npx skills add github/awesome-copilot --skill arize-prompt-optimization

Or download just the file:

curl -L https://raw.githubusercontent.com/github/awesome-copilot/main/skills/arize-prompt-optimization/SKILL.md -o SKILL.md
Technical content (original SKILL.md)

Original text by the author, straight from GitHub.

Arize Prompt Optimization Skill

SPACE — All --space flags and the ARIZE_SPACE env var accept a space name (e.g., my-workspace) or a base64 space ID (e.g., U3BhY2U6...). Find yours with ax spaces list.

Concepts

Where Prompts Live in Trace Data

LLM applications emit spans following OpenInference semantic conventions. Prompts are stored in different span attributes depending on the span kind and instrumentation:

Column What it contains When to use
attributes.llm.input_messages Structured chat messages (system, user, assistant, tool) in role-based format Primary source for chat-based LLM prompts
attributes.llm.input_messages.roles Array of roles: system, user, assistant, tool Extract individual message roles
attributes.llm.input_messages.contents Array of message content strings Extract message text
attributes.input.value Serialized prompt or user question (generic, all span kinds) Fallback when structured messages are not available
attributes.llm.prompt_template.template Template with {variable} placeholders (e.g., "Answer {question} using {context}") When the app uses prompt templates
attributes.llm.prompt_template.variables Template variable values (JSON object) See what values were substituted into the template
attributes.output.value Model response text See what the LLM produced
attributes.llm.output_messages Structured model output (including tool calls) Inspect tool-calling responses

Finding Prompts by Span Kind

  • LLM span (attributes.openinference.span.kind = 'LLM'): Check attributes.llm.input_messages for structured chat messages, OR attributes.input.value for a serialized prompt. Check attributes.llm.prompt_template.template for the template.
  • Chain/Agent span: attributes.input.value contains the user's question. The actual LLM prompt lives on child LLM spans -- navigate down the trace tree.
  • Tool span: attributes.input.value has tool input, attributes.output.value has tool result. Not typically where prompts live.

Performance Signal Columns

These columns carry the feedback data used for optimization:

Column pattern Source What it tells you
annotation.<name>.label Human reviewers Categorical grade (e.g., correct, incorrect, partial)
annotation.<name>.score Human reviewers Numeric quality score (e.g., 0.0 - 1.0)
annotation.<name>.text Human reviewers Freeform explanation of the grade
eval.<name>.label LLM-as-judge evals Automated categorical assessment
eval.<name>.score LLM-as-judge evals Automated numeric score
eval.<name>.explanation LLM-as-judge evals Why the eval gave that score -- most valuable for optimization
attributes.input.value Trace data What went into the LLM
attributes.output.value Trace data What the LLM produced
{experiment_name}.output Experiment runs Output from a specific experiment

Prerequisites

Proceed directly with the task — run the ax command you need. Do NOT check versions, env vars, or profiles upfront.

If an ax command fails, troubleshoot based on the error:

  • command not found or version error → see references/ax-setup.md
  • 401 Unauthorized / missing API key → run ax profiles show to inspect the current profile. If the profile is missing or the API key is wrong, follow references/ax-profiles.md to create/update it. If the user doesn't have their key, direct them to https://app.arize.com/admin > API Keys
  • Space unknown → run ax spaces list to pick by name, or ask the user
  • Project unclear → ask the user, or run ax projects list -o json --limit 100 and present as selectable options
  • LLM provider call fails (missing OPENAI_API_KEY / ANTHROPIC_API_KEY) → run ax ai-integrations list --space SPACE to check for platform-managed credentials. If none exist, ask the user to provide the key or create an integration via the arize-ai-provider-integration skill
  • Security: Never read .env files or search the filesystem for credentials. Use ax profiles for Arize credentials and ax ai-integrations for LLM provider keys. If credentials are not available through these channels, ask the user.

Phase 1: Extract the Current Prompt

Find LLM spans containing prompts


…
FAQ

What are 'spans'?

Spans are like records of each step the AI takes. They help understand what happened when processing a question.

What are 'prompts'?

Prompts are the instructions or questions you give to the AI. The quality of the prompt directly affects the quality of the response.

What does 'optimize' mean here?

Optimizing means making something work in the best possible way. In this case, it's making the instructions for the AI most effective for getting good answers.

Details

Original description by the author

Optimizes, improves, and debugs LLM prompts using production trace data, evaluations, and annotations. Extracts prompts from spans, gathers performance signal, and runs a data-driven optimization loop using the ax CLI. Use when the user mentions optimize prompt, improve prompt, make AI respond better, improve output quality, prompt engineering, prompt tuning, or system prompt improvement.

repo: github/awesome-copilot

Popularity

GitHub stars
40k
Ranking position
#5007
License
MIT
Updated
10/15/2026

Is this skill useful?

0

Sign in to vote.

Comments

  • …

More skills from this repository

Documentation40k

acquire-codebase-knowledge

Helps your agent understand and document an existing code project. It's useful for onboarding new people or getting an overview of the architecture.

github/awesome-copilot

Documentation40k

acreadiness-assess

Enables your agent to assess a project's readiness for AI and generate a report. It's useful for understanding how well a project is prepared to use AI tools.

github/awesome-copilot

Documentation40k

acreadiness-generate-instructions

Helps your agent create instructions for AI tools based on an assessment. It's useful for improving how AI interacts with code and closing identified gaps.

github/awesome-copilot

Agents40k

acreadiness-policy

Enables your agent to help define or apply policies for AI project assessment. It's useful for standardizing how projects are evaluated and ensuring compliance.

github/awesome-copilot

Similar skills

Prompts & LLMs280k

bengali-nlp

Has your agent process Bengali text. It helps understand and work with the language, normalizing text and handling its unique characteristics.

affaan-m/ECC

Prompts & LLMs280k

prompt-caching-strategy

Helps your agent optimize how it uses prompts for language models. It analyzes how prompts are made to find ways to cut costs and make responses faster by reusing information when possible.

affaan-m/ECC

Prompts & LLMs280k

prompt-optimizer

Has your agent analyze and improve the prompts you write. It suggests how to add information or change the wording so the AI model understands better and gives the desired answer.

affaan-m/ECC