README.md

---
license: mit
language:
  - ru
pipeline_tag: audio-to-audio
tags:
  - VoiceConversion
---

# DINO-HuVITS

## Info
Разработка данной модели вдохновлена статьёй [DINO-VITS](https://arxiv.org/abs/2311.09770).
В основе лежит архитектура VITS, в которой оригинальный `PosteriorEncoder` был заменён на модель [HuBERT Base](https://arxiv.org/abs/2106.07447), а обучение `SpeakerEncoder` происходило с помощью функции потерь [DINO](https://arxiv.org/abs/2304.05754).

## Quick start

```python
import librosa
import torch

from dino_huvits import DinoHuVits


model = DinoHuVits.from_pretrained("SazerLife/DINO-HuVITS")
model = model.eval()

content, _ = librosa.load("<content-path>", sr=16000)
reference, _ = librosa.load("<reference-paht>", sr=16000)

content = torch.from_numpy(content).unsqueeze(0)
lengths = torch.tensor([content.shape[1]], dtype=torch.long)
reference = torch.from_numpy(reference).unsqueeze(0)

with torch.no_grad():
    output, _ = model(content, lengths, reference)
```

## Datasets
- [Common Voice](https://commonvoice.mozilla.org/ru)
- [VoxForge](https://github.com/vlomme/Multi-Tacotron-Voice-Cloning)
- [M-AILABS](https://github.com/vlomme/Multi-Tacotron-Voice-Cloning)
- [VoxTube](https://github.com/IDRnD/VoxTube)
- [Golos](https://github.com/sberdevices/golos)
- [OpenSTT](https://github.com/snakers4/open_stt)
- [Sova](https://github.com/sovaai/sova-dataset)