AI语音克隆

安装

下载安装github代码库

git clone https://github.com/Plachtaa/VITS-fast-fine-tuning.git

安装文档
中日语言模型网站
目前支持的任务:

从 10条以上的短音频 克隆角色声音从 3分钟以上的长音频（单个音频只能包含单说话人） 克隆角色声音从 3分钟以上的视频（单个视频只能包含单说话人） 克隆角色声音通过输入 bilibili视频链接（单个视频只能包含单说话人） 克隆角色声音

本地运行和推理

python VC_inference.py --model_dir ./OUTPUT_MODEL/G_latest.pth --share True

这个时候在本地的浏览器打开网址

http://localhost:7860

就可以看到语音tts的使用界面，但这只能在本地电脑能看到，如果要在远程的电脑上访问，可以使用cpolar

cpolar http 7860

这个时候就会出现一个访问的网址链接。

本地训练

1.创建conda运行环境

conda create -n tts python=3.8

2.安装环境依赖

pip install -r requirements.txt

在这个过程中，有一部分安装包，比如OpenAI的whisper代码包，可能因为网络问题，而无法访问，无法使用pip进行网络安装。可以在其它地方，单独下载好代码包，然后使用pip单独安装本地包。
3.安装GPU版本的PyTorch

# CUDA 11.6pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu116# CUDA 11.7pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu117

4.安装视频模块包

pip install imageio==2.4.1pip install moviepy

5.构建预处理模块

cd monotonic_alignmkdir monotonic_alignpython setup.py build_ext --inplacecd ..

6.下载辅助数据包

mkdir pretrained_models# download data for fine-tuningwget https://huggingface.co/datasets/Plachta/sampled_audio4ft/resolve/main/sampled_audio4ft_v2.zipunzip sampled_audio4ft_v2.zip# create necessary directoriesmkdir video_datamkdir raw_audiomkdir denoised_audiomkdir custom_character_voicemkdir segmented_character_voice

7.下载预训练模型

CJE: Trilingual (Chinese, Japanese, English)CJ: Dualigual (Chinese, Japanese)C: Chinese only

wget https://huggingface.co/spaces/Plachta/VITS-Umamusume-voice-synthesizer/resolve/main/pretrained_models/D_trilingual.pth -O ./pretrained_models/D_0.pthwget https://huggingface.co/spaces/Plachta/VITS-Umamusume-voice-synthesizer/resolve/main/pretrained_models/G_trilingual.pth -O ./pretrained_models/G_0.pthwget https://huggingface.co/spaces/Plachta/VITS-Umamusume-voice-synthesizer/resolve/main/configs/uma_trilingual.json -O ./configs/finetune_speaker.json

wget https://huggingface.co/spaces/sayashi/vits-uma-genshin-honkai/resolve/main/model/D_0-p.pth -O ./pretrained_models/D_0.pthwget https://huggingface.co/spaces/sayashi/vits-uma-genshin-honkai/resolve/main/model/G_0-p.pth -O ./pretrained_models/G_0.pthwget https://huggingface.co/spaces/sayashi/vits-uma-genshin-honkai/resolve/main/model/config.json -O ./configs/finetune_speaker.json

wget https://huggingface.co/datasets/Plachta/sampled_audio4ft/resolve/main/VITS-Chinese/D_0.pth -O ./pretrained_models/D_0.pthwget https://huggingface.co/datasets/Plachta/sampled_audio4ft/resolve/main/VITS-Chinese/G_0.pth -O ./pretrained_models/G_0.pthwget https://huggingface.co/datasets/Plachta/sampled_audio4ft/resolve/main/VITS-Chinese/config.json -O ./configs/finetune_speaker.json

8.将语音数据放置在对应的文件目录

短语音
将多段语音打包成zip文件，文件结构为

Your-zip-file.zip├───Character_name_1├   ├───xxx.wav├   ├───...├   ├───yyy.mp3├   └───zzz.wav├───Character_name_2├   ├───xxx.wav├   ├───...├   ├───yyy.mp3├   └───zzz.wav├───...├└───Character_name_n    ├───xxx.wav    ├───...    ├───yyy.mp3    └───zzz.wav

将打包文件放置在./custom_character_voice/
运行

unzip ./custom_character_voice/custom_character_voice.zip -d ./custom_character_voice/

长语音
将wav格式的语音命名为Diana_234135.wav，放置在./raw_audio/视频
将视频命名为Taffy_332452.mp4，放置在./video_data/

9.处理音频

python scripts/video2audio.pypython scripts/denoise_audio.pypython scripts/long_audio_transcribe.py --languages "{PRETRAINED_MODEL}" --whisper_size largepython scripts/short_audio_transcribe.py --languages "{PRETRAINED_MODEL}" --whisper_size largepython scripts/resample.py

注意将"{PRETRAINED_MODEL}"替换为"C"，如果GPU内存没有12GB，将whisper_size替换为medium或small。

10.处理文本数据
选择对应的辅助数据包，运行

python preprocess_v2.py --add_auxiliary_data True --languages "C"

如果不选择辅助数据包，运行

python preprocess_v2.py --languages "{PRETRAINED_MODEL}"

11.开始训练
运行命令，开始训练

python finetune_speaker_v2.py -m ./OUTPUT_MODEL --max_epochs "{Maximum_epochs}" --drop_speaker_embed True

如果是从一个训练过的模型，开始继续训练

python finetune_speaker_v2.py -m ./OUTPUT_MODEL --max_epochs "{Maximum_epochs}" --drop_speaker_embed False --cont True

12.清除语音数据

rm -rf ./custom_character_voice/* ./video_data/* ./raw_audio/* ./denoised_audio/* ./segmented_character_voice/* ./separated/* long_character_anno.txt short_character_anno.txt

del /Q /S .\custom_character_voice\* .\video_data\* .\raw_audio\* .\denoised_audio\* .\segmented_character_voice\* .\separated\* long_character_anno.txt short_character_anno.txt