The Soul of Voice AI
Talk to it once. It knows you next time.
Memory for your voice AI, in a few lines.
# 1. Install VoiceMem (ASR / voiceprint / scene / emotion / embedding built in)pip install voicememfrom voicemem import VoiceMemvm = VoiceMem(mode="normal", openai_key="api_xxx", top_k=5)vm.warmup() # local models are lazy-loaded# 2. Ingest audio — ASR, voiceprint, scene, emotion and embedding run insidevm.ingest(audio="assets/input.wav") # "I'm vegetarian, allergic to nuts."# 3. Search — pure vector retrieval, decoupled from ingestionresult = vm.search("What are my dietary restrictions?")print(result.result_leftbrain, result.result_rightbrain)# 1. The streaming interface behaves like a VAD that keeps eating audiostream = vm.stream(src_rate=sr, vad_threshold=0.5, on_partial=on_partial)step = int(sr * .032)for i in range(0, len(pcm), step): st = await stream.feed(pcm[i:i + step].tobytes()) if st.state != "turn_over": continue # 2. Memory was already retrieved while the user was still speaking print(st.transcript, st.result_leftbrain, st.result_rightbrain) print(st.speaker_id, st.emotion, st.entity, st.schema) # 3. Every turn goes through one ingestion decision res = vm.ingest(st.transcript)# 1. Clone the repo — the demo lives here, pip ships the library onlygit clone https://github.com/xzf-thu/VoiceMem.gitcd VoiceMem# 2. Pull the default modelspip install -U huggingface_hubhf download zhifeixie/VoiceMem_Default_Models_Env --local-dir ./models# 3. Start the interactive web demoexport OPENAI_API_KEY=sk-...python web/run.py# 4. Then open the local UIhttp://localhost:8787Two brains. One voice. Nothing to wait for.
The left brain organizes factual memory through schemas and entities, while the right brain models personality and emotion with independent and cross-entity nodes. The entire pipeline is streaming: while the user is still speaking, audio is segmented, transcribed, extracted, and written into the graph. At query time, VoiceMem routes first, ranks second, and injects only the Top-K memories into context.
ChatMem-400K is produced by a three-stage pipeline of memory-world construction, SLM-validated online on-policy distillation, and human refinement; the same pipeline also produces the ChatMem-Bench evaluation set. After OPD distillation, Qwen2.5-Omni, Qwen3-Omni, and Step-Audio2-Mini learn to proactively invoke memory without sacrificing their original speech capabilities.
More accurate. Fewer tokens. Near-zero latency inside the voice turn.
VoiceMemK=5 deployment point430 mean memory tokens on LoCoMo
Build it with VoiceMem.
'/%3E%3Cellipse%20cx='38.5'%20cy='42'%20rx='4.6'%20ry='6'%20fill='%2332343D'/%3E%3Cellipse%20cx='61.5'%20cy='42'%20rx='4.6'%20ry='6'%20fill='%2332343D'/%3E%3Cpath%20d='M35%2057c4.2%208.4%2010.6%2012%2015%2012s10.8-3.6%2015-12'%20fill='none'%20stroke='%2332343D'%20stroke-width='5.4'%20stroke-linecap='round'%20stroke-linejoin='round'/%3E%3Cg%20fill='url(%23g)'%20stroke='%23E9A400'%20stroke-width='2.2'%20stroke-linejoin='round'%3E%3Cpath%20d='M22%2064c-7%201-12%206-13.5%2012.5-0.8%203.6%202%206.5%205.4%205.6L27%2078c3-0.8%204.4-4.2%202.7-6.8L25.6%2065c-0.8-1-2-1.4-3.6-1z'/%3E%3Cpath%20d='M78%2064c7%201%2012%206%2013.5%2012.5%200.8%203.6-2%206.5-5.4%205.6L73%2078c-3-0.8-4.4-4.2-2.7-6.8L74.4%2065c0.8-1%202-1.4%203.6-1z'/%3E%3C/g%3E%3C/svg%3E){kind=small}