speech

Speech Recognition

asr

• Open ASR Leaderboard - a Hugging Face Space by hf-audio

• https://github.com/openai/whisper

• https://github.com/ggerganov/whisper.cpp

• ylacombe/whisper-large-v3-turbo · Hugging Face

• nvidia/canary-1b · Hugging Face

• Whisper - a mlx-community Collection

• GitHub - sindresorhus/awesome-whisper: 🔊 Awesome list for Whisper — an open-source AI-powered speech recognition system developed by OpenAI

Transducers - RNN-T

Separate source encoder for input sequence and “predictor” model that only predicts next token in output space (LLM), with a “Joiner” module that takes the encoder and predictor outputs and combines them to predict the next output.

• good for streaming

• + can pretrain the predictor in self supervised fashion on next token prediction

• Sequence-to-sequence learning with Transducers - Loren Lugosch

• In Depth Explaination Of RNN-T based Automatic Speech Recognition Systems (ASR) - YouTube

  • sites.cc.gatech.edu/classes/AY2021/cs7643_spring/assets/L24_rnnt_asr_tutorial_gt.pdf

HuBERT

• [2106.07447] HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units

CTC

Decoding

Greedy

Beam Search

Conformer

• [2005.08100] Conformer: Convolution-augmented Transformer for Speech Recognition

Whisper

Mamba

TTS - Text to Speech

• TTS Arena - a Hugging Face Space by TTS-AGI

F5-TTS

• [2410.06885] F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching

End to End Multimodal Speech to Speech

Spirit LM

• GitHub - facebookresearch/spiritlm: Inference code for the paper “Spirit-LM Interleaved Spoken and Written Language Model”.
• SpiRit-LM, an Interleaved Spoken and Written Language Model | Multimodal Weekly 47 - YouTube

Links

• Olewave - YouTube
• WAVLab - YouTube