Speech Research Intern-2

📋 Description

• • Design and evaluate speech-first models, with a focus on Spoken Language Models (SLMs) that reason over audio and engage in conversational interactions.
• • Develop end-to-end speech dialogue systems that process speech input and generate speech output.
• • Align speech encoders with text backbones using lightweight adapters to enable multimodal reasoning.
• • Design efficient speech tokenization and temporal compression techniques optimized for long-form audio processing.
• • Build and execute reliable evaluation frameworks covering speech recognition, understanding, and generation tasks, including robustness and safety metrics.
• • Optimize inference pipelines for low-latency, streaming, and real-time user experiences.
• • Prototype conversational SLMs using self-supervised learning (SSL) speech encoders and compact adapters on existing LLMs, comparing against established baselines.
• • Create data recipes that combine conversational speech datasets with instruction-following corpora, conducting targeted ablations and documenting findings.
• • Construct an evaluation harness that measures performance across ASR, ST, SLU, and speech QA tasks, including streaming-specific metrics like latency, stability, and endpointing.
• • Ship minimal production-ready demos with streaming inference and logging capabilities, documenting setup procedures, metrics, and reliability checks.
• • Author clear internal technical write-ups outlining project goals, design decisions, results, and actionable next steps for productionization.
• • Work closely with applied scientists and engineers to translate research prototypes into practical, scalable AI solutions.
• • Utilize PyTorch, CUDA, torchaudio, and librosa for model development and training on GPU infrastructure.
• • Implement experiment tracking using tools such as Weights & Biases to ensure reproducibility and iterative improvement.
• • Integrate LLM backbones with lightweight adapters and apply neural audio codecs or vocoders as needed for speech generation.
• • Deploy efficient inference systems using FastAPI/gRPC, ONNX, TensorRT, and quantization techniques.
• • Operate in a remote-first environment with flexible scheduling, aligned to team collaboration needs.
• • Contribute to a research culture focused on fast, responsible experimentation with access to cutting-edge AI infrastructure.
• • Engage in mentorship opportunities with senior scientists and engineers, with potential pathways to publish findings or present at internal/external forums.