Qianjue and Tsinghua Team Introduce 'Effective Degree' to Measure and Optimize Neural Network Simplicity

A classic puzzle in deep learning is: why do neural networks with far more parameters than training samples still generalize? A widely accepted explanation is 'simplicity bias': when faced with multiple functions that fit the training data, the training process tends to choose the simpler one. But how to define and measure 'simplicity' has lacked a satisfactory answer.

Existing candidate metrics such as parameter norm, sharpness, and Jacobian norm often fail to simultaneously satisfy universality, quantifiability, and optimizability. Parameter-space metrics may reflect the parameterization rather than the function itself, while directly fitting multivariate polynomials in function space faces combinatorial explosion.

The team proposes to 'slice' high-dimensional neural networks along interpolation paths between data points, fitting the output on each path with Chebyshev orthogonal polynomials, and then computing the effective degree (ED) by weighting the coefficients of each order. Theoretical results show that for multivariate polynomials, random paths almost surely preserve the algebraic degree, so the complexity ordering can be preserved.

ED resides in function space, is independent of parameterization, and is data-distribution-dependent, measuring complexity near the actual data. Compared to only looking at the highest order, ED is continuous and stable with respect to coefficients, and insensitive to noise.

On CIFAR-10, using 27 hyperparameter settings to train ResNet18 and ViT-Tiny, ED achieves a Pearson correlation coefficient of 0.99 with the generalization gap, with an R² of 0.98, outperforming sharpness (0.94/0.88). In CLIP ViT fine-tuning on ImageNet, ED maintains a positive correlation, while other metrics show weak or negative correlations.

In the grokking phenomenon, ED rises during the memorization phase, peaks at the generalization turning point, and then declines, whereas parameter norm and sharpness do not clearly mark the phase transition. This indicates that ED can track the model's shift from memorization to induction.

Since polynomial fitting is a least-squares problem, ED has an analytical gradient with respect to the network output and can be backpropagated. The team adds ED as a regularization term to the training objective, not forcing linearity but penalizing unnecessary high-order variations.

Experiments cover image, text, vision-language model fine-tuning, and reinforcement learning. ED regularization brings generalization improvements, such as a better trade-off between in-distribution and out-of-distribution accuracy in CLIP fine-tuning.

The paper explicitly states that 'simpler is better' is not unconditional: if the simplest feature is a shortcut, ED may also exploit it. ED describes complexity, not correctness or fairness.

Path polynomials are a distribution-dependent proxy, and sampling additional interpolation points during training incurs computational overhead. Future work could explore using ED for selecting pretrained checkpoints, diagnosing overfitting, or designing self-supervised tasks for world models.