Annoynomous authors
Autoregressive diffusion models (ARDMs), which generate continuous latent sequences, have recently achieved state-of-the-art zero-shot text-to-speech (TTS) performance. However, fine-tuning these models with reinforcement learning (RL) to directly optimize user-defined reward functions remains an open challenge. In this work, we propose Value-Guided Policy Optimization (VGPO), an actor-critic RL algorithm tailored to ARDMs. We train a causal value model to predict expected future rewards and update the ARDM using gradients from this value model. To validate VGPO, we fine-tune the recently introduced DiTAR model and evaluate it on two tasks: improving F0 variance to enhance expressiveness; and optimizing text log-probability to improve the model's robustness to challenging long text. VGPO can achieve significant improvement in zero-shot TTS expressiveness and robustness, while maintaining naturalness and speaker similarity.
Scroll the table horizontally to find more audio samples.
| # | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Text |
Hugh MacConnell came with his sister, and stood about, managing his tea-cup awkwardly and watching every one out of his deep-set, faded eyes. |
They constitute two different orders of facts which correspond to each other, which are always interlaced, and which often bring forth results. |
In an instant Servadac mounted the side-work, laid himself down in the gap, and thus filling up the breach by his own body, shouted, "March on!" |
My remark pleases him, but I soon prove to him that it is not the right way to speak, however perfect may have been the language of that ancient writer. |
"See!" said Uncas, pointing north and south, at the evident marks of the broad trail on either side of him, "the dark-hair has gone toward the forest." |
"Then, dear," said Mrs. Whitney, "you must be kinder to her than ever; think what it would be for one of you to be away from home even among friends." |
"I quite agree--in regard to Griffin's ghost, or whatever it was--that its appearing first to the little boy, at so tender an age, adds a particular touch." |
As a result we have this paradoxical situation: The Gospel supplies the world with the salvation of Jesus Christ, peace of conscience, and every blessing. |
Then there was a little pause in the conversation, and I felt myself bound to say something as to the violent interruption to which I had this morning been subjected. |
But Rodolfo had been struck by the great beauty of Leocadia, the hidalgo's daughter, and presently he began to entertain the idea of enjoying it at all hazards. |
The queens had taken their seats upon a magnificent dias or platform, erected upon the borders of the lake, in a theater of wonderful elegance of construction. |
| Prompt | |||||||||||
| VGPO | |||||||||||
| DPO | |||||||||||
| Base Model | |||||||||||
| Ground Truth |
Scroll the table horizontally to find more audio samples.
| # | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Text |
这些花粉被抖出来抖出来抖出来抖出来抖出来抖出来抖出来，再筛选一番，然后酿成酿成酿成酿成酿成酿成酿成蜜，做成蜡。 |
近日，除了葛洲坝葛洲坝葛洲坝葛洲坝葛洲坝股价下跌外，其余其余其余其余其余三家均均均均均有不同程度的的的的的的的上涨。 |
好吧，我们别别别别别别别耽搁耽搁耽搁耽搁耽搁耽搁时间了，收拾收拾收拾收拾收拾收拾收拾东西，干干干干干干正经事吧。 |
涮墩布墩布墩布墩布墩布墩布拖地拖地拖地拖地拖地拖地拖地也是用洗完洗完洗完洗完洗完洗完衣服，倒在在在在在桶里的的的的的水。 |
我等不及等不及等不及等不及等不及去去去去去玩水，我站站站站站在沙上沙上沙上沙上沙上的时候，脚快脚快脚快脚快脚快烧起来了。 |
老太太青面獠牙，姑娘一见心惊胆战心惊胆战心惊胆战心惊胆战心惊胆战心惊胆战心惊胆战，打算打算打算打算打算打算打算赶快逃走。 |
同时同时同时同时同时同时肥胖者腹部腹部腹部腹部腹部腹部脂肪堆积，又限制限制限制限制限制了肺的呼吸运动运动运动运动运动运动运动。 |
难道是有哪个幸运小家伙小家伙小家伙小家伙小家伙小家伙小家伙，被他看中看中看中看中看中看中了了了了了了不成不成不成不成不成。 |
好久没去幼稚园看小朋友们了，不能两手空空呀。不能两手空空呀。不能两手空空呀。不能两手空空呀。不能两手空空呀。不能两手空空呀。不能两手空空呀。 |
刚进刚进刚进刚进刚进房间，陈缵光不明白明白明白明白明白明白明白，三个人怎么挤得上双层双层双层双层双层双层的的的的的的的单人床。 |
共同建设面向未来面向未来面向未来面向未来面向未来面向未来的的的的的的的交通，和出行服务服务服务服务服务新生态生态生态生态生态生态生态。 |
| Prompt | |||||||||||
| VGPO | |||||||||||
| DPO | |||||||||||
| Base Model |