How I Wasted Millions

WARNING: A memoir of bad financial decisions and GPU abuse.

You want to know where my money went? Don’t check my Google Colab billing history. I stopped checking after the first million VND because it stopped being funny. It just became… data I didn’t want to look at.

It didn’t start like that.

At the beginning, it felt easy. I loaded up XTTS v2, scraped a couple of stream videos, turned them into a dataset, and hit train. Nothing fancy. Just enough to see what would happen.

And it worked.

The first result wasn’t perfect, but it was good enough to feel like progress. So I did what everyone does when something works the first time. I assumed I understood it. I added more data, trained again, and the model improved. Not dramatically, but enough to make me think I was getting closer to something real.

That should have been the stopping point.

But “good enough” is a dangerous place to stop when you’ve already seen improvement. It creates this quiet assumption that perfection is just a few tweaks away.

That’s where things started to go wrong.

The model had small issues. Occasional word errors. Slight inconsistencies. Nothing that would matter to most people. But once you hear them, you can’t unhear them. So I decided to fix it properly.

That meant touching the DVAE.

Looking back, that was the moment the system stopped behaving like something I understood. I tried extending the vocabulary, thinking it would give the model more flexibility. Instead, it introduced more errors. Not new types of errors. Just more of them, appearing more often, in places that used to be stable.

At that point, I had a choice. Step back, or go deeper.

I went deeper.

I started feeding it more data, hoping scale would smooth things out. It didn’t. The voice didn’t degrade, but it didn’t improve either. It just plateaued. Same mistakes, same tone, no movement. Like the model had decided it was done learning.

So I started changing everything around it instead.

I shortened audio chunks. Cleaned the dataset again. Ran everything through Demucs, then ClearVoice. Split segments, ranked them by quality, filtered out anything that looked even slightly off. I even tried clustering embeddings, thinking maybe I could isolate the “cleanest” voice patterns.

None of it fixed the core problem.

The model could reproduce structure. It could follow rhythm. But it couldn’t capture the small variations that make a voice feel real. It sounded correct, but not alive. Like it was reading from memory instead of speaking.

At some point, I stopped expecting it to get better. I just kept trying different things to see if anything would change.

Eventually, I gave up on XTTS.

Not because it was bad, but because I didn’t know how to push it further without breaking it again. So I switched to IndexTTS, hoping a different architecture would reset the problem.

The first output was almost convincing. The similarity was high enough to feel uncomfortable. For a moment, it felt like I had skipped all the previous mistakes.

Then I looked at the inference time.

It was slow in a way that didn’t feel fixable. Not “needs optimization” slow. More like “this is how it works” slow. Generating anything longer than a short sentence felt impractical. It solved one problem and replaced it with another.

So I moved again.

By the time I got to Chatterbox, I wasn’t experimenting anymore. I was just trying to get something usable out of everything I had already collected.

The dataset helped. Training was faster, more predictable. I tried a full fine-tune first, mostly out of habit. That failed in a way I should have expected. My data was too small compared to the base model. It didn’t adapt; it collapsed.

LoRA was different.

It didn’t try to overwrite everything. It just nudged the model enough to shift the voice in the right direction. That was the first time in a while something felt stable again.

Not perfect. Just stable.

There are still issues. Speed control is inconsistent. Short sentences sometimes break in ways that are hard to predict. At one point, I had to go into the code and manually adjust how the model handles end-of-sentence tokens, because it kept stopping too early for no clear reason.

But it works.

Not in the way I imagined at the beginning. Not clean, not elegant, not fully understood. But it produces a voice that is close enough to be usable.

Seven million VND later, that’s what I have.

Something that works, and a long list of things that didn’t.

🎧 The Results

Synthesized (My Model)

Reference Audio (Emotion Replication)

Real Voice (Ground Truth)

author: Ope tag: #tts links: