ML trading signal classifiers using XGBoost and LightGBM with walk-forward validation, SHAP feature importance, and threshold optimization

Signal Classification Predict whether an asset's price will move up or down over a forward horizon using supervised machine learning classifiers. This skill covers the full pipeline: label creation, model training, walk forward validation, feature importance analysis, and threshold optimization for trading applications. Why Tree Based Models Dominate Trading ML XGBoost and LightGBM are the workhorses of quantitative trading ML for good reason: Non linear relationships : Financial features interact in complex, non linear ways that trees capture naturally Robust to feature scale : No need to normalize or standardize inputs — trees split on rank order Built in feature importance : Understand which features drive predictions without separate analysis Fast training and inference : Train on thousands of samples in seconds, predict in microseconds Handle missing values : Native support for NaN without imputation hacks Regularization built in : max depth, min child weight, subsample all prevent overfitting Linear models and deep learning have their place, but for tabular trading features with fewer than 100k samples, gradient boosted trees consistently outperform alternatives. Classification Types Binary Classification The simplest and most common setup. Predict whether forward returns exceed a threshold: Up signal : forward return +1% Down signal : forward return < 1% Neutral (excluded) : 1% to +1% — drop these from training to create cleaner labels Multi Class Classification Three classes for finer signal granularity: Class Condition Typical threshold Strong Up fwd return +2% High confidence long Mild Up +0.5% to +2% Moderate confidence Down fwd return < 0.5% Avoid / short Multi class reduces per class sample size. Use only with large datasets (1000+ samples per class). Probability Calibration Raw model probabilities from XGBoost/LightGBM are not well calibrated. A predicted 0.7 probability does not mean 70% chance of being correct. Use calibration to fix this: Isotonic calibration works better than Platt scaling for tree models. Walk Forward Validation This is the single most important concept in trading ML. Standard cross validation randomly shuffles data, which creates lookahead bias. Walk forward validation respects time ordering. How It Works Each window: 1. Train on past N bars 2. Skip a gap (embargo) equal to the forward return horizon 3. Predict on next M bars 4. Record out of sample predictions 5. Slide forward and repeat Typical Parameters Parameter Value Rationale Train window 30 days (720 hourly bars) Enough data to learn, recent enough to be relevant Test window 7 days (168 hourly bars) Enough predictions for statistical significance Step size 1 day (24 bars) Overlap test windows for more data points Gap (embargo) Same as forward horizon Prevents label leakage Walk Forward Implementation See references/validation methods.md for purged CV, CPCV, and evaluation metrics. Model Training Pipeline Full Pipeline Overview 1. Feature engineering — compute technical indicators, on chain metrics, volume features (see feature engineering skill) 2. Label creation — forward returns with threshold, drop neutral zone 3. Walk forward split — time ordered train/test windows with gap 4. Train model — XGBoost or LightGBM on each training window 5. Predict on test — generate out of sample probability predictions 6. Aggregate predictions — concatenate all out of sample results 7. Evaluate — accuracy, precision, recall, F1, AUC, profit factor Quick Training Example See references/model guide.md for parameter recommendations and tuning. SHAP Feature Importance SHAP (SHapley Additive exPlanations) provides the gold standard for understanding model predictions. Global Feature Importance Which features matter most across all predictions: Local Explanations Why a specific prediction was made: Temporal Feature Importance Track how feature importance drifts over walk forward windows. If a feature's importance drops significantly, the market regime may have shifted. Threshold Optimization The default 0.5 probability threshold is almost never optimal for trading. Why Not 0.5? Class imbalance: if 60% of labels are "up", a 0.5 threshold is too aggressive Trading costs: marginal signals (0.51 probability) rarely cover transaction costs Asymmetric payoffs: precision matters more than recall for trading Optimize for Profit Factor Typical finding: optimal threshold is 0.60 0.75 for crypto trading signals. Crypto Specific Considerations Short Training Windows Crypto market regimes change fast. A model trained on 6 months of data may perform worse than one trained on 30 days. Use shorter training windows and retrain frequently. Class Imbalance Most time periods are "flat" (returns within the neutral zone). Strategies to handle this: Drop neutral zone : only train on clear up/down labels Undersample majority class : scale pos weight in XGBoost SMOTE : synthetic minority oversampling (use cautiously — can introduce lookahead) Adjust threshold : raise the probability threshold to compensate Transaction Costs A model with 55% accuracy sounds good, but after 0.5% round trip costs (slippage + fees), many signals become unprofitable. Always evaluate signals net of costs: Feature Decay Features lose predictive power over time as more participants discover and trade on them. Monitor rolling performance and retrain when metrics degrade. Integration with Other Skills Skill Integration feature engineering Compute input features for the classifier vectorbt Backtest trading strategies from ML signals regime detection Train separate models per regime, or use regime as a feature position sizing Size positions based on classifier confidence risk management Apply portfolio level risk limits to ML generated signals Files References references/model guide.md — XGBoost and LightGBM parameter guide, tuning, and ensembling references/validation methods.md — Walk forward, purged CV, CPCV, and evaluation metrics Scripts scripts/train classifier.py — Train a signal classifier with walk forward validation and feature importance scripts/walk forward backtest.py — Backtest ML signals vs buy and hold with walk forward validation Dependencies Key Takeaways 1. Walk forward validation is non negotiable — random CV will give you wildly inflated results 2. Optimize threshold for profit factor , not accuracy — a high precision, low recall model beats a high accuracy one 3. Short training windows for crypto — 30 days beats 6 months in most regimes 4. Monitor feature decay — retrain when rolling metrics drop below baseline 5. Always evaluate net of costs — a 55% accurate model may be unprofitable after fees 6. SHAP over raw feature importance — SHAP gives consistent, theoretically grounded explanations

signal-classification