Arbor — Autonomous Optimization via Hypothesis Tree Refinement

Overview

This skill runs an Autonomous Optimization (AO) loop: starting from an existing artifact and a measurable objective, improve it through many rounds of experiment and evaluation — without step-by-step human supervision, while checking for overfitting to the development signal. It's the right tool when the bottleneck isn't writing one good change, but organizing dozens of trials so that lessons accumulate instead of evaporating.

It implements Hypothesis Tree Refinement (HTR) from Arbor (Jin et al., 2026). The key idea: keep the research state in a persistent hypothesis tree rather than in conversation history. Each node binds a hypothesis, the distilled insight it produced, and a pointer to the artifact version that realizes it. You play the long-lived coordinator that owns this tree and decides where to search; short-lived executor subagents test one hypothesis each in isolated git worktrees and report back. A held-out merge gate admits a change only when it improves on a test evaluator executors do not optimize against. This is what turns trial-and-error into cumulative, auditable research.

Use the scripts/tree.py state manager for all the bookkeeping (creating nodes, writing evidence, propagating insights, pruning, the merge gate, the Observe projection). It records scores and decisions; it does not run evaluators, create worktrees, verify Git refs, or merge Git branches. Only the coordinator writes shared state, serially; executors return evidence. The helper has no multi-writer locking. Its JSON schema is separate from upstream Arbor checkpoints.

When to use this skill

Reach for Arbor when the task is iterative improvement of a concrete artifact under an evaluator:

• Model training: optimizer/architecture/recipe changes to lower loss or hit a target in fewer steps.
• Harness/agent engineering: raising pass rate or accuracy of an agent loop, search harness, or tool-use scaffold.
• Data synthesis: improving a generation/filtering pipeline judged by downstream model behavior.
• Benchmark optimization: MLE-bench / Kaggle-style "improve the submission" tasks.
• Prompt/system optimization where you can score outputs automatically.

The distinguishing signals: there's an artifact you can modify, an objective, a way to score candidates, and you expect to run many experiments. If the user only wants a single fix or a one-shot answer, this is overkill — just do the work directly. If they want open-ended ideation with no evaluator, use hypothesis-generation or scientific-brainstorming instead.

The AO setup — pin this down first

Before any experiments, establish the task tuple (M_0, O, E_dev, E_test). Record the resolved setup, using the user's existing instructions; clarify only missing decisions:

• M_0 — initial material: the artifact to improve (a repo, a script, a config, a prompt). Make sure it's under git and currently runs.
• O — objective: the natural-language goal and the metric direction (maximize accuracy? minimize loss/steps?).
• E_dev — development evaluator: a command you can run freely during search to score a candidate. Fast, repeatable.
• E_test — held-out test evaluator: a separate evaluation protocol (held-out examples or predeclared independent seeds) used only at the merge gate. It must not be used as a search oracle — that's the whole point.

Freeze the evaluator, dataset split, and metric before search, and record their versions with each score. Repeated merge decisions on the same held-out set can still adapt the search to that set; a fresh worktree prevents file contamination, not statistical leakage. Keep a final untouched evaluation set for the final claim, or disclose that the reported result was selected using repeated gate feedback.

If the user hasn't given you a clean dev/test split, construct one and say so. The dev/test separation is the mechanism that catches overfitting: a candidate that wins on dev but not on test isn't a success, it's a warning that you're exploiting the feedback signal. A larger run on the same examples is not an independent holdout. If no defensible holdout exists, label results as development-only.

…