Skip to content
Skillpedia
PT
Machine learning40k · #5105

datanalysis-credit-risk

Cleans and prepares credit risk data for modeling. It removes bad data and irrelevant variables. It's useful before building loan models.

What it does

This tool organizes and cleans credit risk data. It removes unnecessary or low-quality information to prepare the data for future analyses, such as building loan models.

When to use it

  • ✓When you have raw customer data that needs organizing.
  • ✓If you need to identify and remove data with too much missing information.
  • ✓To select the most important information before building a model.

Example of what to ask your AI

“Clean and prepare my credit risk data for analysis.”

Want more? Open the sections below.

How to use

The easiest way: copy the message below and paste it into Claude, ChatGPT, Cursor or any AI. It reads the skill and follows the instructions.

Skill link

https://github.com/github/awesome-copilot/blob/main/skills/datanalysis-credit-risk/SKILL.md

Ready-to-paste message

Use this skill to help me: https://github.com/github/awesome-copilot/blob/main/skills/datanalysis-credit-risk/SKILL.md Read the SKILL.md file at that link and follow its instructions for my task.
View on GitHub ↗Raw file
How it works

The tool loads your data and checks its quality, removing data with too much missing information, irrelevant information, or information that doesn't change much over time. It generates a detailed report of everything that was done.

Installation

Easiest way: copy the skill link and paste it into Claude, ChatGPT or Cursor asking it to follow the instructions.

In the terminal (Claude Code, Codex, Cursor), in your project folder:

npx skills add github/awesome-copilot --skill datanalysis-credit-risk

Or download just the file:

curl -L https://raw.githubusercontent.com/github/awesome-copilot/main/skills/datanalysis-credit-risk/SKILL.md -o SKILL.md
Technical content (original SKILL.md)

Original text by the author, straight from GitHub.

Data Cleaning and Variable Screening

Quick Start

# Run the complete data cleaning pipeline
python ".github/skills/datanalysis-credit-risk/scripts/example.py"

Complete Process Description

The data cleaning pipeline consists of the following 11 steps, each executed independently without deleting the original data:

  1. Get Data - Load and format raw data
  2. Organization Sample Analysis - Statistics of sample count and bad sample rate for each organization
  3. Separate OOS Data - Separate out-of-sample (OOS) samples from modeling samples
  4. Filter Abnormal Months - Remove months with insufficient bad sample count or total sample count
  5. Calculate Missing Rate - Calculate overall and organization-level missing rates for each feature
  6. Drop High Missing Rate Features - Remove features with overall missing rate exceeding threshold
  7. Drop Low IV Features - Remove features with overall IV too low or IV too low in too many organizations
  8. Drop High PSI Features - Remove features with unstable PSI
  9. Null Importance Denoising - Remove noise features using label permutation method
  10. Drop High Correlation Features - Remove high correlation features based on original gain
  11. Export Report - Generate Excel report containing details and statistics of all steps

Core Functions

Function Purpose Module
get_dataset() Load and format data references.func
org_analysis() Organization sample analysis references.func
missing_check() Calculate missing rate references.func
drop_abnormal_ym() Filter abnormal months references.analysis
drop_highmiss_features() Drop high missing rate features references.analysis
drop_lowiv_features() Drop low IV features references.analysis
drop_highpsi_features() Drop high PSI features references.analysis
drop_highnoise_features() Null Importance denoising references.analysis
drop_highcorr_features() Drop high correlation features references.analysis
iv_distribution_by_org() IV distribution statistics references.analysis
psi_distribution_by_org() PSI distribution statistics references.analysis
value_ratio_distribution_by_org() Value ratio distribution statistics references.analysis
export_cleaning_report() Export cleaning report references.analysis

Parameter Description

Data Loading Parameters

  • DATA_PATH: Data file path (best are parquet format)
  • DATE_COL: Date column name
  • Y_COL: Label column name
  • ORG_COL: Organization column name
  • KEY_COLS: Primary key column name list

OOS Organization Configuration

  • OOS_ORGS: Out-of-sample organization list

Abnormal Month Filtering Parameters

  • min_ym_bad_sample: Minimum bad sample count per month (default 10)
  • min_ym_sample: Minimum total sample count per month (default 500)

Missing Rate Parameters

  • missing_ratio: Overall missing rate threshold (default 0.6)

IV Parameters

  • overall_iv_threshold: Overall IV threshold (default 0.1)
  • org_iv_threshold: Single organization IV threshold (default 0.1)
  • max_org_threshold: Maximum tolerated low IV organization count (default 2)

PSI Parameters

  • psi_threshold: PSI threshold (default 0.1)
  • max_months_ratio: Maximum unstable month ratio (default 1/3)
  • max_orgs: Maximum unstable organization count (default 6)

Null Importance Parameters

  • n_estimators: Number of trees (default 100)
  • max_depth: Maximum tree depth (default 5)
  • gain_threshold: Gain difference threshold (default 50)

High Correlation Parameters

  • max_corr: Correlation threshold (default 0.9)
  • top_n_keep: Keep top N features by original gain ranking (default 20)

Output Report

The generated Excel report contains the following sheets:

  1. 汇总 - Summary information of all steps, including operation results and conditions
  2. 机构样本统计 - Sample count and bad sample rate for each organization
  3. 分离OOS数据 - OOS sample and modeling sample counts
  4. Step4-异常月份处理 - Abnormal months that were removed
  5. 缺失率明细 - Overall and organization-level missing rates for each feature
  6. Step5-有值率分布统计 - Distribution of features in different value ratio ranges
  7. Step6-高缺失率处理 - High missing rate features that were removed
  8. Step7-IV明细 - IV values of each feature in each organization and overall

…

FAQ

What is 'credit risk data'?

It is information about customers that helps predict the likelihood of them not repaying a loan.

What are 'IV' and 'PSI'?

IV (Information Value) and PSI (Population Stability Index) are metrics used to assess the quality and stability of information in the data.

Does the tool delete my original data?

No, the tool works with the data without deleting the originals, ensuring you always have the initial file.

Details

Original description by the author

Credit risk data cleaning and variable screening pipeline for pre-loan modeling. Use when working with raw credit data that needs quality assessment, missing value analysis, or variable selection before modeling. it covers data loading and formatting, abnormal period filtering, missing rate calculation, high-missing variable removal,low-IV variable filtering, high-PSI variable removal, Null Impor

repo: github/awesome-copilot

Popularity

GitHub stars
40k
Ranking position
#5105
License
MIT
Updated
10/15/2026

Is this skill useful?

0

Sign in to vote.

Comments

  • …

More skills from this repository

Documentation40k

acquire-codebase-knowledge

Helps your agent understand and document an existing code project. It's useful for onboarding new people or getting an overview of the architecture.

github/awesome-copilot

Documentation40k

acreadiness-assess

Enables your agent to assess a project's readiness for AI and generate a report. It's useful for understanding how well a project is prepared to use AI tools.

github/awesome-copilot

Documentation40k

acreadiness-generate-instructions

Helps your agent create instructions for AI tools based on an assessment. It's useful for improving how AI interacts with code and closing identified gaps.

github/awesome-copilot

Agents40k

acreadiness-policy

Enables your agent to help define or apply policies for AI project assessment. It's useful for standardizing how projects are evaluated and ensuring compliance.

github/awesome-copilot

Similar skills

Machine learning280k

pytorch-patterns

Helps your agent use PyTorch for machine learning models. It applies best practices for efficient model training and ensures reproducible results.

affaan-m/ECC

Machine learning280k

evm-token-decimals

Helps your agent avoid errors with decimal numbers across different blockchain networks. It ensures token values are accurate in transactions and calculations.

affaan-m/ECC

Machine learning101k

knowledge-agent

Creates and queries knowledge bases using AI. It allows asking questions about past work or compiling expertise on a topic.

thedotmack/claude-mem