Persistent Gradient Disagreement Reweighting: Detecting Shortcut Biases Without Training Group Labels

01 Abstract

Deep learning models can learn rules that are predictive in the training distribution but misaligned with the intended concept. This report studies shortcut learning as an incorrect inductive bias, focusing on cases where models rely on spurious attributes such as image background or shallow lexical overlap. Waterbirds is used as the primary empirical setting: the target label is bird type, but land and water backgrounds are correlated with the label during training. I propose Persistent Gradient Disagreement Reweighting (PGDR), an approach that does not use training group labels to construct its selected set or training loss, and instead uses last-layer gradient geometry to identify examples that may conflict with the learned shortcut. The report evaluates gradient disagreement as a pseudo-group discovery signal, compares PGDR with ERM, JTT, and GroupDRO, and analyzes how warmup quality and refresh affect the reliability of the signal.

02 Resources

Full text

pgdr.pdf PDF · 8 pages

Implementation

github.com/hellonish/PGDR Code and experiments

03 Citation

Sharma, Nishant. “Persistent Gradient Disagreement Reweighting: Detecting Shortcut Biases Without Training Group Labels.” Technical report, April 30, 2026.

@misc{sharma2026pgdr,
  author       = {Nishant Sharma},
  title        = {Persistent Gradient Disagreement Reweighting:
                  Detecting Shortcut Biases Without Training Group Labels},
  year         = {2026},
  month        = apr,
  howpublished = {Technical report},
  url          = {https://www.hellonish.dev/research/pgdr/}
}