Supporting material availability

This catalog copy contains the inline guidance below, but does not include the following supporting paths. They are external prerequisites, not bundled files:

• <support-root>/assets/pipeline-dag.yaml.template
• <support-root>/references/data-preparation.md
• <support-root>/references/model-training.md

Ask for the original supporting material or an existing installation before using a step that depends on these paths; this catalog does not identify a verified download source for them. Resolve each path relative to that installation, inspect its contents, and verify any applicable license and execution prerequisites. Do not run a command or claim a referenced procedure was followed until its required file exists. Do not fabricate the missing scripts or treat an unrelated file with the same name as equivalent. Independent steps fully documented here can proceed within the user’s authorized scope; report any dependent step as unavailable.

ML Pipeline Workflow

Complete end-to-end MLOps pipeline orchestration from data preparation through model deployment.

Do not use this skill when

• The task is unrelated to ml pipeline workflow
• You need a different domain or tool outside this scope

Instructions

• Clarify goals, constraints, and required inputs.
• Apply relevant best practices and validate outcomes.
• Provide actionable steps and verification.

Overview

This skill provides comprehensive guidance for building production ML pipelines that handle the full lifecycle: data ingestion → preparation → training → validation → deployment → monitoring.

Use this skill when

• Building new ML pipelines from scratch
• Designing workflow orchestration for ML systems
• Implementing data → model → deployment automation
• Setting up reproducible training workflows
• Creating DAG-based ML orchestration
• Integrating ML components into production systems

What This Skill Provides

Core Capabilities

1. Pipeline Architecture

   • End-to-end workflow design
   • DAG orchestration patterns (Airflow, Dagster, Kubeflow)
   • Component dependencies and data flow
   • Error handling and retry strategies

2. Data Preparation

   • Data validation and quality checks
   • Feature engineering pipelines
   • Data versioning and lineage
   • Train/validation/test splitting strategies

3. Model Training

   • Training job orchestration
   • Hyperparameter management
   • Experiment tracking integration
   • Distributed training patterns

4. Model Validation

   • Validation frameworks and metrics
   • A/B testing infrastructure
   • Performance regression detection
   • Model comparison workflows

5. Deployment Automation

   • Model serving patterns
   • Canary deployments
   • Blue-green deployment strategies
   • Rollback mechanisms

Reference Documentation

See the references/ directory for detailed guides:

• data-preparation.md - Data cleaning, validation, and feature engineering
• model-training.md - Training workflows and best practices
• model-validation.md - Validation strategies and metrics
• model-deployment.md - Deployment patterns and serving architectures

Assets and Templates

The assets/ directory contains:

• pipeline-dag.yaml.template - DAG template for workflow orchestration
• training-config.yaml - Training configuration template
• validation-checklist.md - Pre-deployment validation checklist

Usage Patterns

Basic Pipeline Setup

# 1. Define pipeline stages
stages = [
    "data_ingestion",
    "data_validation",
    "feature_engineering",
    "model_training",
    "model_validation",
    "model_deployment"
]

# 2. Configure dependencies
# See assets/pipeline-dag.yaml.template for full example

Production Workflow

1. Data Preparation Phase

   • Ingest raw data from sources
   • Run data quality checks
   • Apply feature transformations
   • Version processed datasets

2. Training Phase

   • Load versioned training data
   • Execute training jobs
   • Track experiments and metrics
   • Save trained models

3. Validation Phase

   • Run validation test suite
   • Compare against baseline
   • Generate performance reports
   • Approve for deployment

4. Deployment Phase

   • Package model artifacts
   • Deploy to serving infrastructure
   • Configure monitoring
   • Validate production traffic

Best Practices

Pipeline Design

…