Scarf single-cell analysis

Scarf streams Zarr-backed count matrices in bounded blocks and records every computation as an immutable, content-addressed artifact. This skill encodes how to drive core Scarf for single-cell RNA analysis: which calls to make, in what order, what evidence to check, and what not to do. It does not use or describe scarf.agent.

Read this file first. Then open only the reference modules you need (index below). Paths in this skill are relative to the skill directory. The maintained copy lives in the Scarf repository at https://github.com/NygenAnalytics/scarf/tree/master/skills/scarf-single-cell. Every recipe in the modules was run against a real store. Numbers quoted there come from the 10x 5K PBMC documentation dataset and are illustrations, not thresholds.

When to Use This Skill

• Use when a task involves a Scarf .zarr store, scarf.DataStore, ds.pipeline or a PipelineRun.
• Use when converting H5AD, 10x, MTX or Seurat data into a Scarf store, or opening and mounting a Cytebase dataset.
• Use for single-cell RNA-seq analysis with core Scarf: QC with removal audits, HVG, PCA and neighbour graphs, Leiden and Paris clustering, UMAP, markers and cautious annotation, batch correction, donor-level comparisons, headless plotting, provenance and export.
• Use when counts are too large for memory or live on object storage, so they must be streamed under an explicit memory budget.
• Do not use it for scarf.agent (the automated agent package), which this skill does not cover.

Setup

• Install into the environment you run Python from: pip install "scarf[extra]>=1.0.0rc17", plus scarf[cytebase] for Cytebase. The explicit pre-release floor matters: a bare scarf[extra] resolves to the old 0.32 series, whose API this skill does not describe.
• Set resources per process before importing Scarf. Defaults claim all detected RAM and every CPU: SCARF_MEM_BUDGET=8G SCARF_WORKERS=8 (memory specs need a unit; a bare 8 is rejected).
• Headless: MPLBACKEND=Agg, call plots with show=False, then result.save(path) and result.close().
• Quiet logs for scripts: scarf.configure_output(level="WARNING", progress=False).
• First look at a store: open it read-only and follow "Open and inspect a store" (references/data-access.md) and "Check the matrix first" (references/quality-control.md) for cell and feature counts, runs, artifact kinds and whether counts look raw. Then review every metadata column yourself: design columns (donor, sample, batch, condition), author annotations to hold out (rule 14) and author-derived columns. The maintained copy of this skill in the Scarf repository also ships scripts/inspect_store.py, a read-only command that runs these checks and flags those columns by name (annotation values stay hidden unless you pass --show-annotation-values). This catalogue copy is Markdown only and does not include the script.
• Long steps (a pipeline run on tens of thousands of cells over remote counts can take many minutes): write the step as a script that logs to a file and ends with a DONE or FAILED line, run it in the background, and wait with a bounded loop that also stops on a Traceback or a dead process. If you launch through a wrapper such as uv run, put timeout inside it (uv run timeout N python step.py; the reverse can leave stale running records), and track the process by $! rather than pgrep -f. See references/performance-and-export.md. Never poll without a time limit.

Mental model

• A DataStore is one Zarr directory: a shared cell table ds.cells, one group per assay (ds.RNA, feature table ds.RNA.feats, counts as cell-major counts plus gene-major countsT), artifacts, and pipeline/runs. QC columns are assay-prefixed: RNA_nCounts, RNA_nFeatures, RNA_percentMito, RNA_percentRibo.
• I is the live boolean cell key. Analytical filters never edit it; they return a frozen cell selection artifact.
• Every result is an ArtifactRef (scope, kind, artifact_id, assay). Producers take refs and return refs: select_hvgs -> run_normalization -> run_pca -> build_ann_index -> query_neighbors -> build_connectivity_map -> run_leiden_clustering / run_umap -> run_marker_search. An identical call returns the existing artifact without recomputing.
• ds.pipeline.run(label=...) runs that whole RNA recipe (filtering, cell cycle, HVG,

…