📘 Feature Engineering – Advanced Overview for PD Modeling

Feature engineering is the art and science of transforming raw data into powerful, predictive signals that a machine learning model can understand. It's not just a preprocessing step; it's where domain expertise is mathematically encoded.

👉 In credit risk, the predictive power of a Probability of Default (PD) model is often 80% determined by the quality of the features and 20% by the model algorithm itself. A complex model with poor features will fail, while a simple model with brilliantly engineered features will excel.

1. Feature Creation: Beyond Simple Ratios

This is about crafting variables that encapsulate financial health, stability, and risk.

• Liquidity Ratios:
  • Current Ratio: current_assets / current_liabilities. (Standard)
  • Quick Ratio (Acid-Test): (current_assets - inventory) / current_liabilities. A stricter measure, as inventory can be hard to liquidate quickly.
• Leverage & Solvency Ratios:
  • Debt-to-Equity: total_liabilities / total_shareholder_equity. Measures a company's financial leverage.
  • Debt-to-EBITDA: total_debt / EBITDA. A key covenant in loan agreements; indicates how many years of earnings are needed to pay off debt.
• Profitability & Efficiency Ratios:
  • Interest Coverage Ratio (ICR): EBIT / interest_expense. Crucial for assessing a borrower's ability to service debt. A value below 1.5 is often a major red flag.
  • Return on Assets (ROA): net_income / total_assets. Measures how efficiently assets are used to generate profit.
• Macro-Economic Linkages:
  • Create features that link the borrower to the external environment, e.g., sector_growth_rate - GDP_growth_rate.

2. Feature Transformation: The Mathematical Why

• Log Transformation:
  • Why: Addresses right-skewness in financial data (e.g., revenue, assets). Models assume features are somewhat normally distributed. Log transformation compresses the long tail, reduces the influence of extreme outliers, and helps stabilize variance.
  • When: Apply to monetary amounts, sizes, and counts. np.log1p(x) is used to handle zeros (log(1+x)).
• Quantile Transformation:
  • Why: A powerful alternative to scaling. It maps the feature's distribution to a specified distribution (e.g., normal) using quantiles. It's robust to outliers and can handle non-linear relationships.
• Yeo-Johnson Transformation:
  • Why: A generalization of the Box-Cox transformation that works with both positive and negative values. Excellent for making a distribution more Gaussian-like.
• Binning (Discretization):
  • Why: Captures non-linear effects. The relationship between age_of_firm and PD might not be linear. A young firm and a very old firm might be riskier than a middle-aged one.
  • How: Use quantile binning (equal number of records per bin) for balanced groups, or business-defined bins (e.g., LTV: 0-60%, 60-80%, 80-100%, 100%+) based on regulatory or internal risk buckets.

3. Feature Encoding: Advanced Strategies

• One-Hot Encoding:
  • Pitfall: The "Dummy Variable Trap" (perfect multicollinearity). Always use drop='first' to remove one category.
  • Issue with High Cardinality: If a feature has hundreds of categories (e.g., SIC_code), one-hot encoding creates a sparse, high-dimensional dataset. This is inefficient and can lead to overfitting.
• Target Encoding (Mean Encoding):
  • What: Replaces a category with the average value of the target (default flag) for that category. Industry='Construction' is replaced with the historical default rate for all construction firms in the training data.
  • Major Risk: Data Leakage. The target value from the future is used to encode a feature. This must be done strictly within cross-validation folds or using a Bayesian approach to prevent over-optimistic results.
  • Smoothing: Blend the category mean with the overall mean to handle low-frequency categories. smoothed_encoding = (n * category_mean + α * global_mean) / (n + α)
• Frequency Encoding:
  • What: Replaces a category with its frequency of appearance in the dataset. For industry, it would be "how many construction firms are in the dataset?" This can be a useful signal without the leakage risk of target encoding.

4. Handling Missing Data: A Strategic Approach

Missingness is often Not Missing At Random (NMAR) and is a feature in itself.

• Create Missingness Indicators: Always create a binary flag is_[feature]_missing. This is often a powerful predictor. (e.g., a missing cash_flow_statement is a negative signal).
• Imputation Strategies:
  • Numerical Features: Don't just use mean/median. Use a model-based approach (e.g., KNNImputer or IterativeImputer in sklearn), which estimates missing values based on other features.
  • Categorical Features: Impute with a new category like "Unknown" or "Not Provided". This is often better than using the mode.

5. Outlier Treatment: Capping vs. Flagging

• Capping/Winsorizing: Instead of removing outliers, cap them at a certain percentile (e.g., 1st and 99th). This retains the data point but reduces its extreme influence. X_capped = np.clip(X, a_min=np.percentile(X, 1), a_max=np.percentile(X, 99))
• Flagging: As mentioned, create a binary feature is_outlier based on statistical tests (e.g., Z-score > 3) or business rules (e.g., leverage_ratio > 10). This allows the model to learn a specific coefficient for these rare but important events.

6. Feature Selection: Cutting the Noise

• Filter Methods: Fast and model-agnostic.
  • Correlation: Remove one feature from any pair with correlation > 0.8-0.9.
  • Univariate Statistical Tests: SelectKBest using f-classif (ANOVA) for numerical features or chi2 for categoricals. Ranks features based on their individual predictive power.
• Wrapper Methods: Computationally expensive but effective.
  • Recursive Feature Elimination (RFE): Fits the model, removes the weakest feature, and repeats until the desired number of features is left. Great for finding a small, optimal set.
• Embedded Methods: Built into the model training process.
  • Lasso (L1 Regularization): Performs feature selection by pushing coefficients of unimportant features to zero.
  • Tree-based Feature Importance: Models like Random Forest and XGBoost provide a native feature_importances_ attribute.

7. Interaction Features: Capturing Synergistic Risk

• Manual Interactions: Domain knowledge is key here. Create features that represent known risk synergies:
  • high_leverage * volatile_industry
  • low_liquidity * short_loan_tenor
  • high_DTI_ratio * low_collateral_coverage
• Automated Interactions: Tools like PolynomialFeatures can generate all polynomial and interaction features (e.g., a, b, a^2, b^2, a*b), but this can lead to a combinatorial explosion of features. Use this with heavy feature selection afterward.

8. Temporal & Behavioral Features: The Story of Time

This is critical for differentiating a temporary hiccup from a terminal decline.

• Trend Features: Calculate the slope of a linear regression over the last N periods for key metrics like revenue, EBITDA, cash_balance.
• Volatility Features: Calculate the rolling standard deviation or coefficient of variation (std/mean) of monthly cash flows. High volatility is a strong default predictor.
• Worst-Performance Features: Not just the average, but the maximum 30-day past due event in the last year.
• Behavioral Ratios: days_sales_outstanding (DSO) trend. An increasing DSO suggests customers are taking longer to pay, indicating stress.

9. Dealing with Imbalance: Feature-Level Solutions

• Create Risk-Aggregation Features: Instead of just industry, create default_rate_in_industry_and_region. This creates a stronger, more specific signal for the rare default event.
• Use Sample Weights: Instead of oversampling/undersampling, many algorithms (e.g., class_weight='balanced' in sklearn) allow you to assign a higher weight to the minority class (defaults) during training. This is often cleaner than manipulating the data itself.

10. Domain Knowledge: The Ultimate Guide

The best features are grounded in finance. A PD model is essentially a quantitative credit analyst.

• Altman Z-Score (for manufacturers): A famous bankruptcy prediction model.
  \[ Z = 1.2A + 1.4B + 3.3C + 0.6D + 1.0E \]

  Where A=Working Capital/Total Assets, B=Retained Earnings/Total Assets, C=EBIT/Total Assets, D=Market Value of Equity/Total Liabilities, E=Sales/Total Assets.

• Debt Service Coverage Ratio (DSCR): (Net Operating Income / Total Debt Service). A cornerstone of commercial lending. A DSCR < 1.0 means the company does not generate enough cash to cover its debts.
• Basel II/III Guidelines: Regulatory frameworks explicitly define what constitutes risk, guiding feature creation (e.g., definitions of past due, loss given default).

Putting It All Together: A Robust PD Model Feature Set