2024-10-07

Models

• rhymes.ai/blog-details/aria-first-open-multimodal-native-moe-model

Papers

• [2410.05258] Differential Transformer
• [2410.02884] LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoning
• [2410.04739] TableRAG: Million-Token Table Understanding with Language Models
• [2410.07073] Pixtral 12B
• [2410.05993] Aria: An Open Multimodal Native Mixture-of-Experts Model
• [2410.07170] One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation
  • GitHub - ml-jku/EVA: One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation
• [2410.06205] Round and Round We Go! What makes Rotary Positional Encodings useful?
• [2409.20566] MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
• [2410.05954] Pyramidal Flow Matching for Efficient Video Generative Modeling
• GitHub - TIGER-AI-Lab/VLM2Vec: This repo contains the code and data for “VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks”

Code

• [ ]

Articles

• Nixiesearch: running Lucene over S3, and why we’re building a new serverless search enginestorage

Videos

• [M2L 2024] Mixture of Experts - Diego de Las Casas - YouTube
• Inference Optimization Tutorial (KDD) - Making models run faster - Part 1 - YouTube
• Databases In-Depth – Complete Course - YouTube

Other

Tweets

• Yuchen Jin on X: “It just scales! I trained GPT-2 (350M) using @kellerjordan0’s new optimizer and achieved a Fineweb validation loss of 3.05 with just 5.2B tokens—only 52% of the tokens required compared to AdamW. This means you can effectively speed up the GPT-2 training by almost 2X! When https://t.co/4uhCS8TbGq” / X