A glossary of all the ways ML models fail to train

• Vanishing gradient

  • Rounding to 0 in mixed precision
  • [ ]

• Over and under flows

• Exploding gradients

• Data issues

• Numerical issues

• Dead neurons

• Correlations

  • Tennis player

• Compression

• ml-engineering/training/dtype.md at master · stas00/ml-engineering · GitHub