Machine Learning (ML) technique taxonomy and selection criteria for analytics…

Research Question

What is the complete, structured landscape of machine learning techniques and algorithms that an advanced analytics department should know, use, and actively pursue — covering foundational concepts, when to use each technique, when not to, best practices, and the latest advancements — and what distinguishes "normal" analytics model construction from genuinely advanced practice?

Findings

Executive Summary

Gradient-boosted decision trees (GBDTs — XGBoost, LightGBM, CatBoost) are the correct default algorithm for tabular analytics, outperforming alternatives including deep learning on the majority of benchmarked datasets (111-dataset comparison, arXiv:2408.14817; Kaggle AI Report 2023). The single most common technique failing that elevates "normal" analytics to "advanced" is out-of-time validation: any analytics team evaluating temporally-ordered models with standard k-fold cross-validation is operating below the floor for regulated financial services. The 2023–2025 advancements most immediately actionable for an analytics team are: conformal prediction for calibrated uncertainty intervals (now production-deployed via MAPIE), causal ML libraries (DoWhy, EconML, CausalML) for treatment effect estimation, AutoML frameworks (AutoGluon, H2O, FLAML) for rapid model development, and TabPFN-2.5 as a tuning-free strong baseline for datasets under 50,000 rows. RBNZ's model risk management expectations are principles-based rather than prescriptive, but require that AI/ML models be explainable, validated, and governed — SHAP satisfies the explainability requirement, and an MLflow + Evidently AI + GitHub Actions stack satisfies the MLOps requirement at minimum viable level.

Key Findings

1. Gradient-boosted decision trees (XGBoost, LightGBM, CatBoost) outperform all other algorithm families on the majority of real-world tabular datasets, and any analytics team not using them as the default supervised learning method is operating below the competency floor. (confidence: high)

2. Deep learning architectures for tabular data (FT-Transformer, TabR, SAINT) outperform GBDTs specifically on "hard" datasets — those with high dimensionality, non-linear interactions, or very large sample counts — but not on the majority of analytics datasets, which are lower-dimensional and smaller. (confidence: high)

3. TabPFN-2.5 (November 2024) achieves 100% win rate over default XGBoost on datasets up to 10,000 rows using a single, training-free forward pass with no hyperparameter tuning, making it the strongest out-of-the-box baseline for small-to-medium analytics datasets. (confidence: high, caveat: benchmarked on i.i.d. splits; temporal financial data requires separate validation)

4. Out-of-time validation — evaluating model performance on data from a later time period than training, with no temporal leakage — is the single most important evaluation practice for temporally-ordered analytics data, and using standard k-fold cross-validation for such data produces systematically inflated performance estimates. (confidence: high)

5. SHAP (Shapley Additive Explanations) is the current standard for model explainability in regulated industries, providing both global feature importance and per-prediction attribution that satisfies RBNZ's principles-based requirement for explainable model decisions; LIME is adequate for rapid local debugging but less rigorous and less consistent across runs. (confidence: high)

6. Conformal prediction provides finite-sample, distribution-free prediction intervals with guaranteed coverage, is production-deployed (Husqvarna demand forecasting, credit risk) via the MAPIE Python library, and represents the most technically sound approach to uncertainty quantification for analytics teams that must communicate model uncertainty to decision-makers or regulators. (confidence: high)

7. Causal ML libraries — DoWhy (causal graph modelling), EconML (heterogeneous treatment effects), and CausalML (Uber, uplift modelling) — are production-stable and enable analytics teams to move from average treatment effects to personalised intervention analysis for A/B testing, marketing attribution, and policy evaluation. (confidence: high)

8. AutoML frameworks (AutoGluon for AWS environments and best ensemble accuracy, H2O AutoML for regulated industries requiring strong explainability, FLAML for resource-constrained or Azure-native environments) are production-ready in 2024 and appropriately used to accelerate prototyping and provide strong baselines before manual model development. (confidence: high)

9. The Temporal Fusion Transformer (TFT) is the strongest deep learning approach for multivariate, multi-horizon time series forecasting with external covariates, while N-HiTS is preferred for long-horizon univariate forecasting at lower computational cost; ARIMA and Prophet remain appropriate for simple univariate series where interpretability and minimal data are constraints. (confidence: high)

10. ML systems accumulate hidden technical debt (Sculley et al. 2015) through data dependency entanglement, hidden feedback loops, and undeclared consumers — the "CACE" principle (Changing Anything Changes Everything) — and this debt is the primary cause of silent production failures in analytics ML deployments. (confidence: high)

11. RBNZ's model risk management expectations require regulated entities to apply existing risk frameworks to AI/ML models, including validation, explainability, and outcome monitoring, but impose no specific technical method requirements; the practical standard that satisfies these expectations is SHAP for explainability, out-of-time validation for temporal models, and an MLflow-based experiment tracking and registry workflow. (confidence: high)

12. The minimum viable MLOps stack for an analytics team — MLflow (experiment tracking + model registry), DVC (data versioning), Evidently AI (drift detection), and GitHub Actions (CI/CD for model retraining) — is entirely open-source, cloud-portable, and sufficient to meet the operational requirements of a regulated analytics function. (confidence: high)

Assumptions

• Assumption: TabPFN-2.5's benchmark results (i.i.d. splits) translate to meaningful performance advantage on financial analytics datasets with temporal structure. Justification: TabPFN's in-context learning mechanism is not specifically designed for temporal data; the i.i.d. assumption is strong for financial data. Conservative assumption: treat TabPFN-2.5 as a strong first-pass baseline requiring out-of-time validation before accepting its predictions.
• Assumption: RBNZ's principles-based approach will not shift to prescriptive method requirements in the near term. Justification: RBNZ's published guidance (November 2024, May 2025) is explicitly principles-based. No published consultation paper indicates movement toward prescriptive technical standards.
• Assumption: The minimum viable MLOps stack (open-source tools) is adequate for regulatory scrutiny. Justification: RBNZ expects adequate governance, not a specific tooling stack. The adequacy of any stack is determined by the audit trail it produces, not the tools used.

Analysis

Three sources of evidence were weighted most heavily: the Lazebnik et al. 2024 arXiv benchmark (111 datasets, 20 models) for the tabular DL vs. GBDT question; the Kaggle AI Report 2023 for practitioner behaviour; and RBNZ's own primary publications for the regulatory picture. The NeurIPS 2023 study and TabArena living benchmark corroborate the DL findings.

The primary tension in the evidence is between benchmark performance and production context. Benchmarks optimise for i.i.d. accuracy on held-out datasets. Analytics in regulated financial services has temporal ordering, concept drift, regulatory constraints on model complexity, and interpretability requirements that benchmarks do not capture. Where these tensions exist, this analysis prioritises production context over benchmark ranking — hence the conservative framing of TabPFN-2.5's advantage for financial data.

The "normal vs. advanced" distinction is treated as an empirical question (what do top-quartile analytics practitioners actually do?) rather than a normative one (what should they do in theory?). Competition benchmarks and survey data are more relevant evidence for this question than textbooks.

The regulatory analysis is confined to RBNZ primary sources rather than comparator regulators (APRA, FCA, ECB/EBA) — that comparative analysis is covered by the separate backlog item on RBNZ supervisory expectations.

Risks, Gaps, and Uncertainties

• Temporal validity of benchmark results: The tabular ML landscape evolves rapidly. The NeurIPS 2023 and arXiv:2408.14817 results will be superseded by new architectures. TabArena's living benchmark is the appropriate monitoring mechanism.
• TabPFN financial data validity: No published study directly measures TabPFN-2.5 performance on financial time series with temporal leakage controls. This is a genuine gap — not a disqualifying one, but one requiring empirical validation before adoption.
• RBNZ regulatory evolution: RBNZ explicitly acknowledged it is monitoring AI developments and may update its guidance. The principles-based position is current as of May 2025 but could shift.
• Causal ML adoption barriers not quantified: The evidence for causal ML adoption is strong for technology companies (Uber, Microsoft, Netflix scale) but thinner for NZ-scale analytics teams. Minimum viable team size and data volume for causal ML to be productive have not been studied.
• O'Reilly AI survey not directly accessed: The Kaggle 2023 report was used as the primary practitioner survey source; the O'Reilly survey may contain complementary or divergent findings.

Open Questions

• Is AutoML mature enough to replace manual model construction for routine analytics tasks (credit scoring, demand forecasting, churn), and what does that mean for the skills composition of an analytics team? (Suggested priority: medium — this is the make-or-buy question for analytics capability)
• What does a minimum viable causal ML adoption look like for a 5-person analytics team in financial services, and what data volume is required before heterogeneous treatment effects are estimable? (Suggested priority: medium)
• How should conformal prediction intervals be communicated to non-technical decision-makers, and what does a regulatory-grade uncertainty disclosure look like? (Suggested priority: medium)
• What is the right model governance artefact standard for RBNZ-regulated entities — are model cards sufficient, or is a more formal validation report required? (Suggested priority: high — directly unblocks the RBNZ supervisory expectations item)
• Does LLM-assisted automated feature engineering (generating features from text fields or suggesting interactions) produce net positive outcomes for structured tabular analytics, or does the noise introduced exceed the signal gained? (Suggested priority: low — horizon capability)

Output section

• Type: knowledge, backlog-item
• Description: A structured taxonomy of ML techniques with decision guides (when to use / not to use), normal-vs-advanced maturity benchmark, best practices, regulatory compliance mapping (RBNZ), and a capability self-assessment framework. Three follow-on backlog items are identified: minimum viable causal ML for regulated analytics, model governance artefact standards for RBNZ compliance, and AutoML vs. manual construction for routine analytics tasks.
• Links:
  • arxiv.org/abs/2408.14817 — Lazebnik et al. (2024) comprehensive tabular ML benchmark
  • www.kaggle.com — Kaggle AI Report 2023, tabular data section
  • Sculley et al. — "Hidden Technical Debt in Machine Learning Systems" (NIPS 2015) — Sculley et al. (2015), hidden technical debt in ML systems

sources

• [x] Kaggle — "State of Data Science and Machine Learning" annual survey
• [x] Papers with Code — ML benchmarks and leaderboards
• [ ] Hastie, Tibshirani & Friedman — The Elements of Statistical Learning (2nd ed.)
• [ ] Bishop — Pattern Recognition and Machine Learning (Microsoft Research, 2006)
• [ ] Geron — Hands-On Machine Learning with Scikit-Learn, Keras & TensorFlow (3rd ed., O'Reilly, 2022)
• [x] Chen & Guestrin — XGBoost paper (2016)
• [x] Ke et al. — LightGBM paper (2017)
• [x] Prokhorenkova et al. — CatBoost paper (2018)
• [x] Lundberg & Lee — SHAP paper (2017)
• [x] Ribeiro et al. — LIME paper (2016)
• [x] Hollmann et al. — TabPFN (2022)
• [x] Lim et al. — Temporal Fusion Transformer (2021)
• [x] Oreshkin et al. — N-BEATS (2020)
• [x] Peters et al. — DoWhy causal ML
• [x] Microsoft — EconML causal ML
• [x] Angelopoulos & Bates — Conformal prediction tutorial
• [ ] Google — Rules of Machine Learning best practices
• [ ] O'Reilly AI & ML survey (2023–2024)
• [x] MLflow documentation — experiment tracking and model registry
• [x] Sculley et al. — "Hidden Technical Debt in Machine Learning Systems" (NIPS 2015)
• [x] RBNZ — Model risk management guidance (BS11 / related guidance)
• [ ] Mitchell et al. — Model Cards for Model Reporting
• [ ] Research/backlog/2026-03-02-ai-not-a-data-problem.md — organisational context for analytics/AI separation
• [ ] Research/backlog/2026-02-28-ai-strategy-security-focus.md — regulatory framing