Deep Learning Tricks of the Trade

Data Sampling and Curation

Augmentation

Batch

Mixup

• HMix and GMix

Augmentation Schedules

FixRes

Initialization

muP

Maximal Update Parametrization (μP)

Weight Averaging

SWA

Model Soup

Learning Rates

Learning Rate Range Test

Learning Rate Schedules

Warmup-Stable-Decay (WSD)

• [2408.13359v1] Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler

• Allows continued pretraining from checkpoint before decay

”Rewarming” for continued pretraining

• [2308.04014] Continual Pre-Training of Large Language Models: How to (re)warm your model?

Optimization

Fine Tuning and Distillation

Test Time

Multi Crop