Distributed Training

• Slaying OOMs with PyTorch FSDP and torchao - YouTube

• LLM-Training-Puzzles/Distributed.ipynb at main · srush/LLM-Training-Puzzles · GitHub

• DeepSpeed

• Torch FSDP

• GitHub - pytorch/torchtitan: A native PyTorch Library for large model training

• Fairscale

• Megatron

Data Parallel

Model Parallel

Tensor Parallel

Split Tensors

Pipeline Parallelism

Split by layer

Expert Parallelism for MoEs

Activation Checkpointing

Gradient Accumulation

CPU Offloading

• How Fully Sharded Data Parallel (FSDP) works? - YouTube
• Lecture 12 (Part2): Maximize GPU Utilization - YouTube

Torch FSDP (Fully Sharded Data Parallel)

• torchtitan/docs/fsdp.md at main · pytorch/torchtitan · GitHub

Device Mesh

DeepSpeed

• Training Overview and Features - DeepSpeed

• Latest News - DeepSpeed

• Stanford CS25: V4 I From Large Language Models to Large Multimodal Models - YouTube

ZeRO1

ZeRO2

ZeRO3

1. Most memory taken up by optimizer states (adam moments need to be stored in full precision)

Multi Datacenter / Federated

• Multi-Datacenter Training: OpenAI’s Ambitious Plan To Beat Google’s Infrastructure

• Arthur Douillard on X: “@dylan522p You may want to read into the latest advance of dist/fed learning for LLMs, e.g. DiLoCo, OpenDiLoCo, Flower. You could summarize those as the Branch-Train-Merge you mention, but on steroids. https://t.co/MrRrIOkj8A” / X

Async SGD

Links

• Distributed Training Of Deep Learning Models : Part ~ 1distributed
• How to train a model on 10k H100 GPUs?distributed
• Self contained example of how pipeline parallel works (AFAB and 1F1B) in 200 LOC · GitHub