Do Tabular Foundation Models Agree with Themselves?

Tabular Foundation Models (TFMs) are transformer-based predictors that approximate a Bayesian posterior predictive distribution from a pre-training prior. They are typically univariate predictors that can be converted into multivariate predictors autoregressively by sampling one target and feeding it back into the feature set. However, the faithfulness of the resulting joint distribution has not been investigated. Since the true posterior is unknown on real-world data, the paper reframes the question: could a model's predictions result from any joint distribution at all?

To answer this, the authors propose two necessary conditions. Marginalization consistency requires that the conditional distribution obtained by marginalizing a joint predictor over other variables equals the directly predicted marginal conditional. Factorization consistency requires that different factorization orders of the autoregressive generation yield the same joint distribution. The paper evaluates multiple TFMs on classification and regression tasks and reports that every evaluated model violates both properties on every dataset, indicating a systematic internal inconsistency rather than a rare edge case.

This result matters because it suggests that TFM predictive distributions, despite strong performance on point predictions, are not coherent probabilistic models. It opens the door to new evaluation criteria and potential architectural or training changes to make TFMs self-consistent.