📉 Why Z-Score Normalization Is the Unsung Hero of Time Series Transformers

ICML 2025 quietly confirms what practitioners already knew: normalization can make or break your model.

In the rush to design ever more complex Transformer architectures for time series forecasting, most of the attention has gone to clever attention mechanisms, hierarchical tokenization schemes, and hybrid modules.

But the ICML 2025 paper “A Closer Look at Transformers for Time Series Forecasting” quietly reminds us of something far simpler — and far more important:

Z-score normalization is one of the most critical components for making Transformers work in time series forecasting.

The authors show that removing z-score normalization hurts model performance more than removing positional encodings, skip connections, or even full attention blocks in some settings.

Let’s unpack why this happens — and how z-score normalization fits into the Transformer pipeline for time series tasks.