Predictive modeling is at the heart of modern machine learning applications. From fraud detection to financial forecasting, the ability to make accurate, reliable predictions can define the success of a data-driven business. But how can machine learning practitioners improve the reliability of their models, particularly when dealing with tabular data? In a recent episode of ODSC’s you have x podcast, Brian Lucenaa leading data scientist and educator, and Principal at Numeristical, shared his insights on gradient boosting, uncertainty estimation, and model calibration—topics crucial for building robust machine learning systems.

You can listen to the full podcast on Spotify, Appleand SoundCloud.

Level Up Your AI Expertise! Subscribe Now:

Why Gradient Boosting Continues to Dominate Tabular Data Problems

Machine learning has seen the rise of deep learning models, particularly for unstructured data such as images and text. Yet, when it comes to structured, tabular data, gradient boosting remains a gold standard. Lucena attributes its dominance to the way gradient boosted decision trees (GBDTs) handle structured information.

Understanding Gradient Boosting

At its core, gradient boosting builds an ensemble of decision trees, iteratively correcting the errors of previous models. Unlike deep learning, which struggles with sharp discontinuities in data, decision trees can model abrupt changes in relationships between variables. This makes them especially effective in business use cases, where real-world relationships are rarely smooth.

Lucena explained how random forests first introduced the power of ensembles, but gradient boosting takes it a step further by focusing on the residual errors from previous trees. The ability to capture subtle patternsincluding small but significant features, makes it superior to traditional methods for many use cases.

Why Gradient Boosting Still Outperforms Deep Learning in Many Cases

Despite the hype around deep learning, Lucena highlights several reasons why gradient boosting remains the top choice for many business applications:

• Handles tabular data effectively: Unlike neural networks, which struggle with structured data, gradient boosting can make sharp splits in variables, capturing discontinuous changes.
• Works well with smaller datasets: Deep learning requires vast amounts of data, whereas gradient boosting can perform well even with limited samples.
• More interpretable: While deep learning models are often black boxes, decision trees provide clearer explanations of why a model makes a particular prediction.

Lucena also mentioned popular open-source libraries for gradient boosting:

• XGBoost: A high-performance implementation of gradient boosting, widely used in Kaggle competitions and industry applications.
• LightGBM: Optimized for speed and scalability, making it useful for large datasets.
• CatBoost: Specialized in handling categorical variables efficiently.

Enhancing Model Reliability with Uncertainty Estimation

One of the major challenges in predictive modeling is that machine learning models often provide a single point prediction—a best guess. However, real-world decisions often require understanding a range of possible outcomes. This is where uncertainty estimation becomes crucial.

Probabilistic Regression: Beyond Point Predictions

Brian Lucena explained how probabilistic regression differs from traditional regression by outputting a probability distribution rather than a single prediction. This is particularly useful in:

• Financial forecasting (e.g., predicting stock prices with confidence intervals)
• Medical risk assessment (e.g., estimating the likelihood of patient outcomes)
• Supply chain and logistics (e.g., forecasting shipping delays with probabilistic estimates)

Traditional models often assume a fixed error marginbut real-world uncertainty varies. Instead, probabilistic models can provide confidence intervalshelping decision-makers assess risk more effectively.

Tools for Probabilistic Regression

To implement probabilistic modeling, Brian Lucena recommended:

• Of boost: A framework for fitting parametric distributions using gradient boosting.
• PyMC: A powerful library for probabilistic programming and Bayesian inference.
• StructureBoost (Lucena’s own package): Helps model categorical variables with structured relationships, such as geographical regions or cyclic data (e.g., seasons, time of day).

The Importance of Probability Calibration

Even when models provide probability scores, they are often miscalibrated. That is, when a model says it’s 80% confident in a prediction, that prediction might only be correct 60% of the time. This can lead to poor decision-making, especially in high-stakes applications like fraud detection or risk assessment.

How to Evaluate and Improve Model Calibration

Brian Lucena discussed reliability diagramsa popular way to visualize calibration. These diagrams plot predicted probabilities against actual observed outcomes, showing where the model is overconfident or underconfident.

To improve calibration, several techniques can be used:

• Isotonic Regression: A non-parametric method that adjusts probability scores to better align with actual outcomes.
• Platt Scaling: Uses logistic regression to recalibrate probabilities.
• Spline Calibration (Lucena’s method): Uses spline-based smoothing to produce more reliable probability scores.

Why Calibration Matters

Consider a loan default prediction model. If a bank’s model predicts a borrower has a 9% chance of defaultbut in reality, similar borrowers default 11% of the timethe bank may be underestimating its risk. Proper calibration ensures that probability scores are truly reflective of real-world likelihoods, leading to better financial decision-making.

Dealing with Model Drift: Ensuring Reliability Over Time

Machine learning models don’t operate in a static environment—customer behavior, market conditions, and economic trends change over time. This phenomenon, known as model driftcan degrade a model’s performance.

Brian Lucena emphasized that model recalibration can be a lightweight alternative to retraining models from scratch. By updating calibration on recent databusinesses can adjust their models to account for shifts in patterns without needing massive retraining efforts.

Some best practices for dealing with model drift include:

• Monitoring performance over time: Regularly checking error rates and recalibration needs.
• Using rolling windows of data: Prioritizing recent data while gradually phasing out older, less relevant samples.
• Adopting adaptive calibration techniques: Making incremental updates rather than waiting for full model retraining cycles.

Final Thoughts: Keeping the Focus on Reliability

Brian Lucena wrapped up the discussion with advice for machine learning practitioners navigating the rapidly evolving AI landscape. While generative AI and large language models (LLMs) are gaining attention, many business-critical machine learning applications still rely on traditional predictive modeling techniques.

For data scientists and engineers, the key takeaways are:

Gradient boosting remains a powerhouse for structured data and should not be overlooked in favor of deep learning.

Uncertainty estimation provides deeper insights into model predictionsallowing for more informed decision-making.

Calibration is crucial for making probability scores meaningful and should be a standard step in model deployment.

Monitoring model drift ensures long-term reliabilitykeeping models aligned with changing real-world conditions.

As businesses increasingly depend on machine learning, ensuring trustworthy, explainable, and well-calibrated predictions is more important than ever. By applying the techniques discussed in this conversation with Brian Lucena, practitioners can build machine learning models that are not just accurate, but truly reliable.