I am a Ph.D. student at LV-Lab, Nanyang Technological University, advised by Prof. Shuicheng Yan and Prof. Chunyan Miao. I am also a two-time founder and CEO: I built and led two startups before and during my Ph.D., shipping DreamFactory as CEO of my first company and later founding Pask.ai, where our proactive agent work grew out of real product needs. Before that I received my bachelor's degree from Beijing Institute of Technology, studied at Tsinghua University, and interned at the Qwen Team under the supervision of Dr. Jin Xu.
My research focuses on large audio language models and real-time speech interaction — building models that hear, think and speak at the same time. I led the Mini-Omni series, the first open-source end-to-end speech-to-speech conversational models, whose architecture has since been adopted as a starting point by a generation of omni models, together with Audio-Reasoner, Mini-Omni-Reasoner, Mega-ASR, the Audio Interaction Model and VoiceMem. I hold research to one bar: it has to be worth doing. Going forward I keep pushing on always-on multimodal interaction, with a focus on interactivity, robustness, reasoning and proactivity.
Outside the lab I advise industry teams on speech technology, have played in symphony orchestras for 16 years, and love animals — many of my projects use a Shiba Inu avatar in memory of Guagua, who grew up with me.
New
Factual memory and emotion/persona memory are kept apart and retrieved while the user is still speaking — persistent personalization without giving up real-time interaction.
On real-time audio interaction and where large audio language models go next — from streaming perception to models that decide when to speak.
The far-field speech recognition benchmark run by Treble Technologies and Hugging Face, measuring ASR in noisy, reverberant rooms.
NewVoiceMem makes long-term memory native to the real-time voice loop. Its factual “left brain” and emotion/persona “right brain” retrieve compact memories while the user is still speaking, combining personalization with near-zero interaction overhead. The result is a practical path from stateless voice assistants to agents that remember, adapt and build continuity across conversations.
Most voice assistants only react after a user speaks. Audio Interaction Model turns continuous sound into an always-on perceive–decide–respond loop, so a model can decide when to stay silent or proactively intervene. SoundFlow, StreamAudio-2M and Proactive-Sound-Bench make real-time proactive audio interaction a trainable and measurable research problem.
MMAE provides the missing evaluation foundation for general-purpose instruction-based audio editing: 7 modalities, 6 complexity levels, 8 edit types and 2,000 curated samples. With current systems still below 5% exact-match on its hardest setting, the benchmark exposes how far audio editing remains from reliable, compositional control.
Mega-ASR targets the gap between clean ASR benchmarks and deployment in noisy, reverberant, moving environments. By scaling physically grounded compound acoustic simulation, it cuts WER by more than 30% versus strong baselines in complex scenes and tops the FFASR far-field leaderboard — a step toward ASR that remains dependable in the real world.
Deep-Reporter addresses a core weakness of research agents: long reports drift from evidence and usually treat figures as afterthoughts. Its multimodal search, checklist-guided synthesis and recurrent context management keep text, images and citations grounded together. M²LongBench makes this harder, more realistic form of long-form generation reproducibly measurable.
PASK moves assistants from answering explicit requests toward anticipating latent needs under real latency constraints. Its demand detection, long-term memory and streaming IntentFlow components form an end-to-end proactive-agent stack, while LatentNeeds-Bench evaluates the setting on user-consented data. The work reframes proactivity as a deployable systems problem rather than a demo feature.
Mini-Omni-Reasoner removes the usual trade-off between reasoning quality and spoken latency by interleaving silent reasoning tokens with speech tokens in one stream. It improves Spoken-MQA arithmetic by +19.1% with no extra decoding latency, establishing thinking-in-speaking as a practical paradigm for reasoning-capable real-time voice models.
Audio-Reasoner brought deliberate reasoning to large audio-language models, showing that structured reasoning training transfers beyond text to sound, music and speech. With the 1.2M-sample CoTA dataset, it achieved strong gains across MMAU-mini, AIR-Bench and MELD, helping establish audio reasoning as a distinct and reproducible research direction.
Cited by Qwen3-Omni · Audio Flamingo 3 · MMAR (NeurIPS 2025) · MMAU-Pro · SonicBench · referenced by the Interspeech 2026 Audio Reasoning Challenge
Mini-Omni2 was an early open-source reference for GPT-4o-style interaction: a single model that can see, hear and speak in real time, including mid-sentence interruption. Its duplex, end-to-end design showed that natural multimodal conversation did not have to remain exclusive to closed frontier systems.
Cited by Qwen2.5-Omni · Baichuan-Omni-1.5 · MiniCPM-o · VITA-1.5 · Step-Audio · EMOVA
Mini-Omni made direct end-to-end speech-to-speech interaction openly reproducible. Parallel speech generation lets the model hear, think and talk in one streaming pipeline instead of chaining ASR, an LLM and TTS. The work became a widely used architectural reference for the open omni-model wave and was later cited as a basis for Qwen2.5-Omni.
Cited by Qwen2.5-Omni · Kimi-Audio · Step-Audio · GLM-4-Voice · Baichuan-Omni-1.5 · MiniCPM-o · LLaMA-Omni2 · Freeze-Omni
DreamFactory reframed long-video generation as coordinated production rather than isolated clip sampling. Its multi-agent “film crew” separates scripting, storyboarding, direction and continuity so multiple scenes can stay coherent over time. The work was an early demonstration that agent orchestration could extend generative video from short clips toward structured, multi-scene narratives.
