We introduce CoVoMix2: a fully non-autoregressive framework for zero-shot multi-talker dialogue generation. It directly predicts mel-spectrograms from multi-stream transcriptions using a flow-matching-based generative model, eliminating the reliance on intermediate token representations. To better capture realistic conversational dynamics, we propose transcription-level speaker disentanglement, sentence-level alignment, and prompt-level random masking strategies. CoVoMix2 operates without requiring transcriptions for the prompt and supports controllable dialogue generation, including overlapping speech and precise timing control, demonstrating strong generalizability to real-world speech generation scenarios.
Paper’s Link (opens in new tab)
System Overview
Samples
All prompt audios are from the demo page of Moonshot’s MoonCast and Google’s Soundstorm and NotebookLM.
Zero-shot Dialogue Generation Samples
Transcription
Sun-set hotel. May I help you? | Yes, I have booked a room for 24th. It’s a double room. | Hold on, please. Let me check it for you. Yes, you’re right. You will keep it for 3 days. | Well, now I want to change the date from 24th to 28th. | OK, that shall be arranged.
Prompts
Generated Speech
Transcription
Why’d you pull me over? | Are you aware that you drove through a red light? | I ran a red light? | Yes, you did. | I apologize, but I didn’t realize that I did. | Weren’t you taught that yellow means slow down, not speed up? | I did learn that. | So, then why did you speed up? | I don’t know what to tell you. | I’m going to have to write you a ticket. | I understand. | Here you go. Don’t do that again.
Prompts
Generated Speech
Transcription
Hey man, you wanna buy some weed? | Some what? | Weed! You know? Pot, Ganja, Mary Jane some chronic! | Oh, umm, no thanks. | I also have blow if you prefer to do a few lines. | No, I am ok, really. | Come on man! I even got dope and acid! Try some! | Do you really have all of these drugs? Where do you get them from? | I got my connections! Just tell me what you want and I’ll even give you one ounce for free. | Sounds good! Let’s see, I want. | Yeah? | I want you to put your hands behind your head! You are under arrest!
Prompts
Generated Speech
Cross-lingual Samples
Transcription
Can you tell me where the pots and pans are? | Pots and pans are right over there. | Oh, thank you. | Could I interest you in our store credit card? | No, thanks. I already have credit cards. | But our credit card saves you 10%. | That’s a nice discount. | Here. Let me give you an application form. | Thank you, but I’m just browsing today. | Okay. Enjoy your browsing.
Prompts
Generated Speech
Transcription
Here is the fish counter. Look at the lobsters and crabs. Shall we have some? | I’m allergic to these things, you know. | Sorry, I forgot. I don’t like seafood, neither. | Let’s go over there and get some milk, a couple dozen eggs and some orange juice. | Let’s get frozen juice. It is really good. We have got enough food. Let’ s go over to the check-out stand. | OK. But just let me pick up a bottle of cooking wine and oil as we go by.
Prompts
Generated Speech
Overlapping Samples
Transcription
Seriously, Jamie? You didn’t think to tell me about this? I had to find out from someone else | Oh! come on! It’s not that big of a deal.
Prompts
Generated Speech
Transcription
Hey, guys! I think we should focus on the marketing strategy first. If we can get the word out there, we’ll have a better chance of attracting more customers. | yeah yeah I agree with you