2023 NeurIPS

NeurIPS 2023 | OpenReview

neurips2023.vizhub.ai

Paper Digest: NeurIPS 2023 Highlights – Paper Digest

Papers

Computer Vision

• Image Captioners Are Scalable Vision Learners Too | OpenReview
• GitHub - bytedance/fc-clip: [NeurIPS 2023] This repo contains the code for our paper Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP

Transformers

• GitHub - JeanKaddour/NoTrainNoGain: Revisiting Efficient Training Algorithms For Transformer-based Language Models (NeurIPS 2023)

2023-12-10 - Sunday

Schedule

SMALLER MODELS CAN PACK A PUNCH IN THE ERA OF LARGE LANGUAGE MODELS

11am - 11:50am | Room 206 - 207

• smoothquant
  • repo
• multiLORA
• text2sql demo

OPTIMIZING AND REASONING ABOUT LLM INFERENCE: FROM FIRST PRINCIPLES TO SOTA TECHNIQUES

12-12:50 | Room 206 - 207

• memory bound

• kv cache

• prompt in parallel

  • compute bound

• decode memory bound

• ideas

  • decompose attention into two steps, block sparse, read less
  • more lower dim tokens
  • predict multiple tokens at a time
    • use position encoding in decoder step, predict token at step x

CONVERSATIONAL RECOMMENDATIONS: PRESENT AND FUTURE

1pm | Room 208 - 210

• user history / profile summarized with LLM to include context in conversation

BUILDING USING LLAMA 2

2pm - 4pm | Room 208 - 210

• intro

REINFORCEMENT LEARNING: TRENDS, APPLICATIONS, AND CHALLENGES

3-3:50pm

THE FUTURE IS HERE: A DEEP DIVE INTO AUTONOMOUS AGENTS

4pm-4:50 | Room R06-R09 (level 2)

COMMUNICATION WITHOUT LANGUAGE BARRIERS: RECENT ADVANCES IN TRANSLATION FOUNDATION MODELS

4-6pm

EMPIRICAL RIGOR IN ML AS A MASSIVELY PARALLELIZABLE CHALLENGE

5pm - 5:50 | Room R06-R09 (level 2)

INDUSTRIAL DEMONSTRATION OF POPULAR BACKBONES FOR TIME SERIES FOUNDATION MODELS

Ballroom A - C (level 2) Sun 10 Dec 12:30 p.m. - 2 p.m. CST
Sun 10 Dec 3:30 p.m. - 5 p.m. CST

2023-12-11 - Monday

Language Models Meet World Models

9:45 a.m. CST — 12:15 p.m. CST

https://sites.google.com/view/neurips2023law

• theory of mind
• simulators
  • next frame prediction

Latent Diffusion Models

9:45 - 12:15 | Hall E2 https://neurips2023-ldm-tutorial.github.io/

Contributing to an Efficient and Democratized Large Model Era

9:45 - 12:15

Application Development using Large Language Models

1:45 p.m. CST — 4:15 p.m | Hall E2

Reconsidering Overfitting in the Age of Overparameterized Models

1:45 p.m. CST — 4:15 p.m | Hall D2 https://sml.inf.ethz.ch/gml23/neuripstut-blank.html

NextGenAI: The Delusion of Scaling and the Future of Generative AI

Hall F (level 1) | 5:25 p.m. CST — 6:15 p.m. CST

Welcome Reception

Hall C & D (level 1) | 6:15 p.m. CST — 8:30 p.m. CS

2023-12-12 - Tuesday

The Many Faces of Responsible AI by Lora Aroyo

8:30 a.m. CST — 9:20 a.m. CST

• data quality
  • sources of disagreements
    • ambiguities
    • context (cultures, differences by country / location)
    • diversity of real world

Oral 1D DL Theory

Room R06-R09 (level 2) 10 a.m. CST — 10:45 a.m

Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalization

Abide by the law and follow the flow: conservation laws for gradient flows

A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learning

Oral 1B Datasets & Benchmarks

La Nouvelle Orleans Ballroom A-C (level 2) 10 a.m. CST — 10:45 a.m.

Poster Session 1

• Species196: A One-Million Semi-supervised Dataset for Fine-grained Species Recognition Poster 204
• Topological RANSAC for instance verification and retrieval without fine-tuning Poster 216
• Learning to Parameterize Visual Attributes for Open-set Fine-grained Retrieval Poster 229
• Rethinking the Role of Token Retrieval in Multi-Vector Retrieval Poster408
• Improving multimodal datasets with image captioning Poster 544
• Analyzing Vision Transformers for Image Classification in Class Embedding Space Poster 714
• An Inverse Scaling Law for CLIP Training Poster 718

• Cola: A Benchmark for Compositional Text-to-image Retrieval 723

• DataPerf: Benchmarks for Data-Centric AI Development 914

Test of Time Award Talk:Distributed Representations of Words and Phrases and their Compositionality

Tomas Mikolov · Ilya Sutskever · Kai Chen · Greg Corrado · Jeff Dean Hall F (level 1) 3:05 p.m. CST — 3:25 p.m. CST

Oral 2A Efficient Learning

Hall C2 (level 1 gate 9 south of food court) 3:40 p.m. CST — 4:40 p.m

Poster Session 2

Great Hall & Hall B1+B2 (level 1) 5:15 p.m. CST — 7:15 p.m.

2023-12-14 - Thursday

ML for Systems #star

• SSMs #TODO

#TODO mamba 4 rec

Oral 5D Vision

Room R06-R09 (level 2) Thu 14 Dec 10 a.m. CST — 10:45 a.m. CST

Visual Instruction Tuning (Poster 229)

• LLaVA. #star

DataComp: In search of the next generation of multimodal datasets #star

• check CLIP embeddings provided #TODO

• check eval pipeline #TODO

• baselines #star #TODO

  • clip filtering (threshold on CLIP score)
  • image cluster based filtering (matching images in target dataset)

• smaller filtered datasets work better than larger noisy ones

• Leaderboard

Oral 6C Vision

Room R02-R05 (level 2) Thu 14 Dec 3:20 p.m. CST — 4:20 p.m. CST

Siamese Masked Autoencoders

Image Captioners Are Scalable Vision Learners Too

The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation

Spatial-frequency channels, shape bias, and adversarial robustness

2023-12-15 - Friday

Foundation Models for Decision Making — Hall E2 (level 1)

• Fri 9:30 a.m. - 10:00 a.m. - Percy Liang ( Invited Talk )
  • memory stream
• Fri 10:00 a.m. - 10:30 a.m. - Ruslan Salakhutdinov ( Invited Talk )
• Fri 10:30 a.m. - 11:00 a.m. - Jürgen Schmidhuber ( Invited Talk ) %% - Fri 2:00 p.m. - 2:30 p.m. - Chelsea Finn ( Invited Talk ) %%

Table Representation Learning Workshop — Room 235 - 236

• 11:00 - 12:00 - Talks

  • CAAFE
  • TabPFN #star
    • #TODO try tuning on MFP data

• 12:00 - 12:20 - Check Posters

• GitHub - AI-sandbox/HyperFast

Workshop on robustness of zero/few-shot learning in foundation models (R0-FoMo) — La Nouvelle Orleans Ballroom A+B (level 2)

• Fri 1:30 p.m. - 2:30 p.m. Poster Session
  • CLIPA-v2: Scaling CLIP Training with 81.1% Zero-shot ImageNet Accuracy within a $10,000 Budget

NeurIPS Large Language Model Efficiency Challenge: 1 LLM + 1GPU + 1Day — Room 356

• 1:30 - 4:30

NeurIPS Large Language Model Efficiency Challenge:1 LLM + 1GPU + 1Day | NeurIPS 2023 Challenge

• Invited Speaker: Jeremy Howard-Lessons from 25 years of machine learning competitions

• Invited Speaker: Sebastian Raschka (lightning.ai) - LoRA in Action: Insights from Finetuning LLMs with Low-Rank Adaptation
• Unveiling Success: A100 track Team percent_bdf’s Winning Strategies
• Invited Speaker: Sourab Mangrulka — Generative AI for Al: 🤗 PEFT: Finetuning made simple, efficient and extendable
• Unveiling Success: 4090 Track winning team’s strategies
• Invited Speaker: Keming Lu (Alibaba Research) - Qwen: Towards a Generalist Model
• Invited Speaker: Mojan Javaheripi (Microsoft Research) - Unleashing the power of Small Language Models
• Invited Speaker: Leshem Choshen (IBM Research) - Efficient Evaluation for Efficient Training

Instruction Tuning and Instruction Following — Room 220 - 222

NeurIPS 2023 Workshop on Diffusion Models — Hall B1 (level 1)

OPT 2023: Optimization for Machine Learning — Hall D2 (level 1)

2023-12-16 - Saturday

Practical Vector Search (Big ANN) Challenge 2023 — Room 356

• [OOD & Sparse track] PyANNS
• 11:00-11:30 - [Keynote] Approximate Nearest Neighbor Search in Recommender Systems

4th Workshop on Self-Supervised Learning: Theory and Practice — Room 217 - 219

• Schedule | NeurIPS 2023 Workshop: Self-Supervised Learning - Theory and Practice

• 9:45 - Diane Larlus - Visual representations that transfer

  • stable diffusion for synthetic datasets (imagenet)
    • not close to supervised
  • DINO + NeRF = N3F

• 10:15 - Alyosha Efros - The Revolution Will not Be Supervised… 8 year later

  • should move away from data augmentation
  • MAE / MIM seems like the current winner (thanks to transformers)
  • visual prompting via image inpainting
  • LVM - large visual model (new paper) #star

• 11:00 - Abhinav Gupta - Self-supervised Learning: Towards Rich Representations?

• 11:30 - Chelsea Finn - Updating Language Models without Human Supervision

• 12:00 - 1:00 - Posters

Third Workshop on Efficient Natural Language and Speech Processing (ENLSP-III): Towards the Future of Large Language Models and their Emerging Descendants — Room 206 - 207

Machine Learning for Systems — Room 211 - 213

Workshop on Advancing Neural Network Training (WANT): Computational Efficiency, Scalability, and Resource Optimization — Room 243 - 245

Synthetic Data Generation with Generative AI — Hall E2 (level 1)

• scaling
  • foundational models for X
• LLMs
• diffusion models
• efficiency
• multi modality
  • CLIP
  • seq 2 seq
• transformers