Evaluation methodology

PluginEval scores a skill or a plugin from 0 to 100 by combining up to three layers. The static layer is a lint. It's fast and deterministic, and it's useful for checking structure. The LLM judge and Monte Carlo layers are experimental, not validated against human labels, so treat their numbers as rough signals. For the trace-based eval program, see evals/README.md at the repository root.

The judge rubric anchors are in references/rubrics.md. Fixes for each anti-pattern and tips for each dimension are in references/improving-scores.md.

Evaluation depths

The label names the depth that ran, not a check against human judgment. A plugin directory gets the static layer only, whatever depth you ask for.

Static layer (lint)

The static layer reads SKILL.md and makes no model calls. It computes seven sub-scores, and the first six feed composite dimensions:

• frontmatter_quality (feeds triggering_accuracy)
• orchestration_wiring (feeds orchestration_fitness)
• progressive_disclosure, structural_completeness, token_efficiency, ecosystem_coherence
• harness_portability

harness_portability maps to no dimension, so it doesn't change a skill's composite score. It does carry 6% of the static layer's own score. A plugin's score is built from that layer score, so portability findings can lower a plugin's score a little. Its findings are not counted as anti-patterns.

The static layer flags six anti-patterns: OVER_CONSTRAINED, EMPTY_DESCRIPTION, MISSING_TRIGGER, BLOATED_SKILL, ORPHAN_REFERENCE, and DEAD_CROSS_REF. Each flag cuts the score by 5%, down to a floor of 50%:

penalty = max(0.5, 1.0 - 0.05 * anti_pattern_count)

The report prints a severity for each flag, but the penalty counts flags and ignores severity.

LLM judge layer (experimental)

The judge layer makes four model calls and returns four holistic scores from 0 to 1:

• triggering_accuracy: Haiku reads the description and writes 10 test prompts, 5 that should trigger and 5 that should not. It predicts the outcome for each prompt and reports its own F1. Nothing checks those predictions against real triggering.
• orchestration_fitness and scope_calibration: Sonnet rates the skill on a five-point rubric.
• output_quality: Sonnet imagines three tasks and rates the output it expects.

The three Sonnet calls see only the first 3,000 characters of SKILL.md. Only one judge runs, because nothing reads the judges setting.

Monte Carlo layer (experimental)

Haiku writes 15 prompts that should trigger the skill, and the layer repeats them to reach 50 runs (100 at thorough). Each run sends the SKILL.md text and one prompt to the model, and the layer records four measures:

• Activation rate is the share of runs with any non-empty reply, so it shows whether the model answered, not whether the skill should have fired.
• Output quality is reply length divided by 500, capped at 1.0.
• Failure rate is the share of runs that errored.
• Token efficiency is 1 - median_tokens / 8000.

Every prompt is one that should trigger, so the layer never checks that the skill stays out of unrelated requests. The layer's JSON includes Wilson, bootstrap, and Clopper-Pearson intervals for its own measures. The composite ci_lower and ci_upper fields are always null.

Composite score

First, for each dimension, the engine blends the layer scores that exist, and it renormalizes the blend weights over those layers. Second, it sums the weighted dimension scores, and it renormalizes the dimension weights over the dimensions that have a score. Third, it multiplies the sum by 100 and by the anti-pattern penalty.

…