Formant-Guided Speech Repair for Enhanced Comprehension of Dysarthric Speech Demo

View the code on GitHub

Framework

Speech Repair Framework

We present speech repair results for dysarthric speakers. The table below compares Input, Input+FAST, ASR+TTS, Ablation studies, Our Proposed Full Model, and Baselines.

Chinese Dysarthria Speech Database

High intelligibility

Method	Input	Input+FAST	ASR+TTS	Ours (w/o SPK)	Ours (w/o FAST)	Ours (Full)	Oracle TTS	DiffDSR^[1]	RnV^[2]	Liu et al.^[3]
成功處置深航 chéng gōng chǔ zhì shēn háng
首屈一指的超大企業 shǒu qū yī zhǐ de chāo dà qǐ yè
身體素質指數偏低 shēn tǐ sù zhì zhǐ shù piān dī

Moderate intelligibility

Method	Input	Input+FAST	ASR+TTS	Ours (w/o SPK)	Ours (w/o FAST)	Ours (Full)	Oracle TTS	DiffDSR^[1]	RnV^[2]	Liu et al.^[3]
附加補貼 fù jiā bǔ tiē
宜減不宜加 yí jiǎn bù yí jiā
進行合理分攤 jìn xíng hé lǐ fēn tān

Low intelligibility

Method	Input	Input+FAST	ASR+TTS	Ours (w/o SPK)	Ours (w/o FAST)	Ours (Full)	Oracle TTS	DiffDSR^[1]	RnV^[2]	Liu et al.^[3]
發行利率也有較大幅度上升 fā xíng lì lǜ yě yǒu jiào dà fú dù shàng shēng
排位均均在二十位之內 pái wèi jūn jūn zài èr shí wèi zhī nèi
城投債券的發行規模 chéng tóu zhài quàn de fā xíng guī mó

Very low intelligibility

Method	Input	Input+FAST	ASR+TTS	Ours (w/o SPK)	Ours (w/o FAST)	Ours (Full)	Oracle TTS	DiffDSR^[1]	RnV^[2]	Liu et al.^[3]
近期銷售模式的 jìn qī xiāo shòu mó shì de
未查出舊案 wèi chá chū jiù àn
兩地的生產總值 liǎng dì de shēng chǎn zǒng zhí

English Dataset (TORGO)

Speaker: M05

Method	Input	Input+FAST	ASR+TTS	Ours (w/o SPK)	Ours (w/o FAST)	Ours (Full)	Oracle TTS	DiffDSR^[1]	RnV^[2]	Liu et al.^[3]
It also provides for funds to clear slums and help colleges build dormitories
Don't ask me to carry an oily rag like that
usually minus several buttons

Speaker: M04

Method	Input	Input+FAST	ASR+TTS	Ours (w/o SPK)	Ours (w/o FAST)	Ours (Full)	Oracle TTS	DiffDSR^[1]	RnV^[2]	Liu et al.^[3]
The train approached the depot slowly
Nothing is as offensive as innocence
a long flowing beard clings to his chin

References

[1] X. Chen, D. Yang, W. Wu, M. Wu, J. Xu, X. Wu, Z. Wu, and H. Meng, “Diffdsr: Dysarthric speech reconstruction using latent diffusion model,” arXiv preprint arXiv:2506.00350, 2025.
[2] Karl El Hajal and Enno Hermann and Sevada Hovsepyan and Mathew Magimai Doss, “Unsupervised Rhythm and Voice Conversion to Improve ASR on Dysarthric Speech,” in Interspeech 2025, 2025, pp. 2760–2764.
[3] D. Liu, Y. Lin, H. Bu, and M. Li, “Two-stage and self-supervised voice conversion for zero-shot dysarthric speech reconstruction,” in 2024 International Conference on Asian Language Processing (IALP). IEEE, 2024, pp. 423–427.