Audio & Speech

TLDR

Audio and speech processing serves as the acoustic interface for modern AI, bridging the gap between continuous physical waveforms and discrete digital tokens. The field is anchored by two primary technologies: Automatic Speech Recognition (ASR), which transcribes audio into text, and Text-to-Speech (TTS), which synthesizes human-like speech from text. Modern systems have moved beyond modular Hidden Markov Models (HMMs) toward end-to-end neural architectures like Conformers and Transformers. In the context of Multi-Modal RAG, audio is no longer just a transcription target but a first-class data citizen, where audio embeddings enable semantic retrieval of non-textual cues like emotion, speaker identity, and environmental context.

Conceptual Overview

The engineering of audio systems requires reconciling the high-dimensional, continuous nature of sound with the discrete requirements of computational models.

1. The Physics and Digitalization of Sound

Sound is a longitudinal pressure wave. To process it digitally, we must adhere to the Nyquist-Shannon Sampling Theorem, which states that to accurately reconstruct a signal, the sampling rate must be at least twice the highest frequency present.

• Sampling Rate: While 44.1 kHz is standard for music, most ASR models (e.g., Whisper, wav2vec 2.0) utilize 16 kHz. This captures the essential frequencies for human speech (up to 8 kHz) while minimizing computational overhead.
• Bit Depth: Usually 16-bit PCM, providing a dynamic range of 96 dB, sufficient for capturing the nuances between a whisper and a shout.

2. Feature Extraction: From Time to Frequency

Raw audio (time-domain) is difficult for neural networks to process directly due to its high sample rate (16,000 data points per second). We transform it into the frequency domain:

• STFT (Short-Time Fourier Transform): We apply a sliding window (e.g., 25ms) to the audio, calculating the frequency spectrum for each window.
• Mel-Spectrograms: Human hearing is non-linear; we are better at distinguishing small changes in low frequencies than high ones. The Mel Scale warps the frequency axis to match human perception. Most modern ASR and TTS models use Log-Mel Filterbanks as their primary input/output feature.

3. ASR Architectures: Decoding the Signal

The goal of ASR is to find the most likely sequence of words $W$ given an acoustic signal $O$.

• Connectionist Temporal Classification (CTC): A loss function that allows the model to output character probabilities at every time step without requiring a manual alignment between audio and text. It introduces a "blank" token to handle transitions.
• RNN-Transducer (RNN-T): Popular in low-latency applications (like mobile assistants), it processes audio frames and predicts tokens in a streaming fashion.
• Attention-based Encoder-Decoder (AED): Models like Whisper use a Transformer architecture to look at the entire audio sequence at once, providing superior accuracy for long-form transcription but higher latency.

4. TTS Architectures: Synthesizing the Waveform

TTS is typically a two-stage process:

• Acoustic Model: Converts text (or phonemes) into a Mel-spectrogram. FastSpeech 2 is a notable non-autoregressive model that uses a "Duration Predictor" to ensure stable speech rhythms.
• Vocoder: The "engine" that turns the spectrogram back into a raw waveform. Neural Vocoders like HiFi-GAN or WaveNet use deep learning to predict the exact amplitude of every sample, resulting in high-fidelity, natural-sounding audio.

. The 'Acoustic Encoder' (Conformer/Transformer) processes these features. For ASR, it flows to a 'Decoder' producing 'Text'. For TTS, 'Text' enters an 'Acoustic Model' (FastSpeech 2) to create a 'Mel-Spectrogram', which is then processed by a 'Neural Vocoder' (HiFi-GAN) to produce 'Synthesized Audio'. A central 'Vector DB' shows 'Audio Embeddings' being stored for RAG retrieval.)

Practical Implementations

1. Optimization through "A (Comparing prompt variants)"

In a Multi-Modal RAG system, the quality of the synthesized speech is heavily dependent on the text generated by the LLM. This is where A (Comparing prompt variants) becomes a critical engineering step.

When an LLM generates a response intended for TTS, the prompt must be engineered to produce "prosody-friendly" text. For instance:

• Standard Prompt: "Explain the process of photosynthesis." -> Result: "Photosynthesis is a process used by plants..." (Can sound dry/robotic).
• TTS-Optimized Prompt: "Explain photosynthesis as if you are a friendly teacher, using short sentences and natural pauses." -> Result: "So, photosynthesis... it's basically how plants eat. First, they take in sunlight..." (Results in much higher Mean Opinion Score (MOS) from the TTS engine).

By A (Comparing prompt variants), developers can identify which system instructions lead to text that avoids "tongue-twisters," complex acronyms, or overly long subordinate clauses that might cause a neural vocoder to hallucinate or glitch.

2. Performance Metrics

• Word Error Rate (WER): The industry standard for ASR. $WER = (S + D + I) / N$, where S is substitutions, D is deletions, I is insertions, and N is the number of words in the reference.
• Real-Time Factor (RTF): $RTF = \text{Processing Time} / \text{Audio Duration}$. An RTF < 1.0 is required for real-time applications.
• Mean Opinion Score (MOS): A subjective 1-5 scale used to evaluate the naturalness of TTS.

3. Handling Long-Form Audio

For RAG systems indexing podcasts or meetings, simple transcription is insufficient.

• VAD (Voice Activity Detection): Stripping silence to save compute.
• Chunking with Overlap: Processing 30-second windows with a 5-second overlap to ensure words aren't cut off at the boundaries.
• Timestamp Alignment: Using models like WhisperX to provide word-level timestamps, allowing the RAG system to link a specific text snippet back to the exact millisecond in the source audio.

Advanced Techniques

1. Speaker Diarization

Diarization answers the question "Who spoke when?".

• Embedding Extraction: Each speaker segment is mapped to a high-dimensional vector (e.g., X-vectors or ECAPA-TDNN embeddings).
• Clustering: Algorithms like Spectral Clustering group these embeddings. In a RAG context, this allows the system to filter information by speaker (e.g., "What did the Doctor say about the medication?").

2. Self-Supervised Learning (SSL) with wav2vec 2.0

The breakthrough in low-resource ASR came from SSL. Models are pre-trained on massive amounts of unlabeled audio by solving a "masked prediction" task—predicting the latent representation of a hidden audio segment. This allows a model to learn the "phonetics" of human speech before being fine-tuned on a small amount of labeled text data.

3. Zero-Shot Voice Cloning

Modern TTS systems like VALL-E use Neural Codecs (like EnCodec). By representing audio as a sequence of discrete codes, the model can treat voice cloning as a language modeling task. Given a 3-second "prompt" of a target speaker, the model can synthesize any text in that speaker's voice, preserving their unique timbre and prosody.

4. Audio-Text Cross-Modal Embeddings (CLAP)

CLAP (Contrastive Language-Audio Pretraining) is the audio equivalent of CLIP. It trains a model to map audio clips and their textual descriptions into the same vector space.

• Application in RAG: This allows for searching audio databases using natural language queries that describe sounds, not just speech (e.g., "Find the part of the recording where a dog is barking in the background").

Research and Future Directions

1. Native Multimodal LLMs

The industry is shifting from "cascaded" systems (ASR -> LLM -> TTS) to Native Multimodal architectures. Models like GPT-4o or Gemini 1.5 are trained on raw audio tokens. This eliminates the "information bottleneck" of text, allowing the model to:

• Hear sarcasm, urgency, or hesitation.
• Respond with synchronized emotional tone.
• Reduce latency by removing the intermediate transcription step.

2. Audio RAG and Semantic Indexing

Future RAG systems will index audio using multi-vector strategies:

1. Text Vector: The semantic meaning of the transcript.
2. Acoustic Vector: The "vibe" or environment (e.g., noisy cafe vs. quiet office).
3. Speaker Vector: The identity and authority of the speaker. This enables queries like: "Find the segment where the speaker sounded most uncertain about the quarterly projections."

3. On-Device and Privacy-Preserving Speech

With the rise of "Ambient Computing," there is a massive push for On-Device ASR/TTS. This involves:

• 4-bit Quantization: Running models like Whisper on mobile NPUs (Neural Processing Units).
• Federated Learning: Training speech models on user data without the audio ever leaving the device, ensuring privacy for sensitive conversations.

Frequently Asked Questions

Q: What is the difference between ASR and Speech-to-Text (STT)?

While often used interchangeably, ASR (Automatic Speech Recognition) is the broader technical field focused on the recognition of spoken language by computers. STT (Speech-to-Text) is the specific application or product of that technology.

Q: Why does my TTS sound "robotic" even with high-quality models?

This is often a "prosody" issue. If the input text lacks punctuation or natural phrasing, the model cannot predict where to place emphasis or pauses. Utilizing A (Comparing prompt variants) to refine the LLM's output style can significantly improve the naturalness of the resulting audio.

Q: How does background noise affect ASR accuracy?

Noise lowers the Signal-to-Noise Ratio (SNR). Modern models use Multi-condition Training, where they are exposed to synthetic noise (rain, traffic, babble) during training. For production, using a pre-processing "Denoiser" (like Meta's Demucs) can improve WER in harsh environments.

Q: Can I use RAG on audio without transcribing it?

Yes, using models like CLAP, you can generate embeddings directly from the audio signal. These embeddings can be stored in a vector database and retrieved using a text query, though the "granularity" of retrieval is currently lower than text-based search.

Q: What is a "Phoneme" and why does it matter for TTS?

A Phoneme is the smallest unit of sound in a language. Because English is not a phonetic language (e.g., "tough" vs. "though"), TTS systems often convert text into a phonemic representation (using a Grapheme-to-Phoneme or G2P model) before generating the audio to ensure correct pronunciation.