What Do Tabular Foundation Models Compute In Context? In-Situ Representation Refinement through Attention-Gated Updates

The paper asks what reusable computation a tabular foundation model should learn when every table defines a new supervised task. It proposes in-situ representation refinement, in which support labels guide updates to the representations of the current episode and those updates transfer to unlabeled queries without changing model parameters.

In deriving this, a regularized leave-one-out objective yields a support correction and its query extension. The leading term separates attention-based reading from state-dependent scaling, which motivates RefineICL: an attention-gated, FFN-free contextual stack that uses selected low-rank feature interaction and typed memory.

RefineICL-L24 reaches 0.93836 OVR-AUC and 0.87173 accuracy on AMLB29. A benchmark-informed continuation reaches 1644.8 Elo on the 38-dataset TabArena snapshot, 31.4 Elo above TabPFN-3 under the same evaluation, and it improves all four reported metrics over TabPFN-v3 on both TabZilla views. In a matched 100K-update depth grid, an expanded FFN gives no consistent validation benefit and uses 60.2% more peak inference memory at L8.

Internal interventions show that support representations are more than a static source of labels: removing one intermediate support update while preserving the query output increases final query cross-entropy in all 72 tested episodes. The authors argue that the derivation and interventions together explain how attention-gated updates can construct a task-specific predictor in context.