Google AI Introduces TabFM: A Hybrid-Attention Tabular Foundation Model for Zero-Shot Classification and Regression
0
0
Google Research introduced TabFM, a foundation model built for tabular data. TabFM performs classification and regression without dataset-specific training. Every prediction comes from a single forward pass. The model reframes tabular prediction as an in-context learning problem. It is available now on Hugging Face and GitHub.
TL;DR
- TabFM predicts on unseen tables with no training, tuning, or feature engineering.
- It reads the full dataset as one prompt, then predicts via in-context learning.
- The architecture combines TabPFN-style row/column attention with TabICL-style in-context learning.
- Training used hundreds of millions of synthetic datasets from structural causal models.
- Google BigQuery will expose TabFM through an AI.PREDICT SQL command soon.
What is TabFM?
Tabular data forms the backbone of enterprise data infrastructure. Tasks like customer churn and financial fraud detection live in tables. For years, tree-based methods dominated this space. XGBoost, AdaBoost, and random forests offered robust results on structured data. Google frames TabFM as the tabular counterpart to TimesFM, its zero-shot time-series model.
That reliability carried a cost. Fitting XGBoost to a new dataset is rarely one .fit() call. Data scientists spend hours on hyperparameter optimization and feature engineering. They do this just to extract a reliable signal from raw data. TabFM targets exactly that bottleneck.
TabFM applies the zero-shot logic that large language models made familiar. LLMs learn new tasks from in-context examples, without updating any weights. This technique is called in-context learning (ICL). TabFM brings the same idea to tables. It generates predictions on previously unseen tables in one pass.
How It Works
Traditional models update parameters for each dataset’s distribution. TabFM skips that step entirely. It takes the whole dataset as a single unified prompt. That prompt holds both training examples and target testing rows. The model reads column and row relationships at inference time.
Tables are not text. They are two-dimensional and inherently orderless. Swapping two rows or two columns does not change their meaning. Standard language models process one-dimensional, ordered sequences instead. To bridge that gap, TabFM synthesizes TabPFN and TabICL into a hybrid design.
It relies on three mechanisms:
- Alternating row and column attention: The raw table passes through a multilayer attention module. Following TabPFN, attention alternates across columns (features) and rows (examples). This deep contextualization captures feature interactions and dependencies. It performs work that would otherwise need manual feature crafting.
- Row compression: Each row’s cross-attended information compresses into a single dense vector.
- In-context learning: A dedicated Transformer runs over these compressed embeddings. Following TabICL, attending to compressed rows cuts computation cost sharply. Prediction stays efficient even on much larger datasets.
Training On Synthetic Data at Scale
Foundation models need vast, diverse data. High-quality tabular datasets are scarce in the open-source space. Industrial tables carry proprietary schemas and sensitive information. That makes them inaccessible for broad pre-training.
Synthetic tables can be generated to be arbitrarily large. Google’s research team calls them effectively the only viable option at this scale. So TabFM trains entirely on hundreds of millions of synthetic datasets. These are generated dynamically using structural causal models (SCMs). Each incorporates a wide variety of random functions. The approach captures distributions and complex feature relationships found in real tables. The research team reports the model generalizes well to unseen real-world data.
Performance and Benchmarking
The research team evaluated TabFM on TabArena. TabArena is a living benchmark that computes Elo scores from head-to-head win rates. The evaluation spans 38 classification datasets and 13 regression datasets. Sample sizes range from 700 to 150,000.
Two configurations were tested. Plain TabFM runs out-of-the-box in a single forward pass. It needs no tuning or cross-validation. TabFM-Ensemble adds cross features and SVD (Singular Value Decomposition) features. It computes optimal weights for a 32-way ensemble using a non-negative least squares solver. For classification, it also adds Platt scaling as a calibration step.
The research team reports TabFM consistently outperforms heavily tuned, industry-standard supervised algorithms. Full per-fold metrics and head-to-head win rates sit on the GitHub page.
| Aspect | Traditional GBDT (XGBoost) | TabFM | TabFM-Ensemble |
|---|---|---|---|
| Per-dataset training | Required | None (in-context learning) | None |
| Hyperparameter tuning | Extensive, manual | None | Ensemble weights via NNLS |
| Feature engineering | Manual, domain-specific | Learned by attention | Adds cross + SVD features |
| Prediction | After full training | Single forward pass | 32-way ensemble |
| Calibration | Manual (optional) | — | Platt scaling (classification) |
Getting Started: Installation and Code
Installation clones the repository and installs it locally. The base install uses CPU-only JAX. A cuda extra pulls the CUDA 12 plugin and NVIDIA libraries for GPU runs.
Core requirements are specific. You need Python 3.11 or later. It pins jax==0.10.1 and flax==0.12.7, using the modern flax.nnx API. Hugging Face Hub downloads the pre-trained weights automatically.
import numpy as np
import pandas as pd
from tabfm import tabfm_v1_0_0
from tabfm import TabFMClassifier
# Load pre-trained TabFM v1.0.0 (downloads from Hugging Face)
model = tabfm_v1_0_0.load()
# scikit-learn compatible classifier
clf = TabFMClassifier(model=model)
X_train = pd.DataFrame({
"age": [25.0, 45.0, 35.0, 50.0],
"job": ["engineer", "manager", "engineer", "manager"],
"income": [80000, 120000, 90000, 130000]
})
y_train = np.array(["low_risk", "high_risk", "low_risk", "high_risk"])
X_test = pd.DataFrame({
"age": [30.0, 48.0],
"job": ["engineer", "manager"],
"income": [85000, 125000]
})
clf.fit(X_train, y_train)
predictions = clf.predict(X_test)
probabilities = clf.predict_proba(X_test)
print("Predictions:", predictions)
print("Class Probabilities:n", probabilities)Here fit() prepares ordinal encoders and numerical scalers. It does not train model weights on your data. The regressor mirrors this pattern with TabFMRegressor and reg.predict().
Use Cases With Examples
The API fits common predictive tasks directly. For customer churn, the context holds past customers labeled churned or retained. TabFM scores churn risk for new customers in one pass.
For credit risk, rows carry age, job, and income features. Labels mark low_risk or high_risk, as in the sample code. New applicants get scored without a training cycle.
For regression, house price prediction is a natural fit. Context rows carry square footage and neighborhood. TabFM returns a predicted price for unseen listings.
Interactive Explainer
Check out the Repo and Technical details. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
The post Google AI Introduces TabFM: A Hybrid-Attention Tabular Foundation Model for Zero-Shot Classification and Regression appeared first on MarkTechPost.
MarkTechPost
