Long Context Transformers

transformers

• Long-Context LLM Extension - YouTube

• Everything About Long Context Fine-tuning

• [2309.12307] LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models

• YouTube

RoPE Expansion

• Gradient Blog: Scaling Rotational Embeddings for Long-Context Language Models
  • rope theta scaling + fine tuning on longer context data
• Extending the RoPE | EleutherAI Blog

Sliding Window Attention

Ring Attention

• RING Attention explained: 1 Mio Context Length - YouTube

Tree Attention

• [2408.04093] Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters

StreamingLLM

• [2309.17453] Efficient Streaming Language Models with Attention Sinks
  • GitHub - mit-han-lab/streaming-llm: [ICLR 2024] Efficient Streaming Language Models with Attention Sinks
• StreamingLLM - Efficient Streaming Language Models with Attention Sinks Explained - YouTube

GitHub - princeton-nlp/ProLong: Homepage for ProLong (Princeton long-context language models) and paper “How to Train Long-Context Language Models (Effectively)”

x.com x.com

Longformer

Linformer

Reformer

Blockwise Attention

Adaptive Attention Span

Infini-Attention

• A failed experiment: Infini-Attention, and why we should keep trying?