Drama-3-Japanese

Built with Fish Audio

Drama-3-Japanese is a Japanese fine-tune of Fish Audio Drama 3. It focuses on single-speaker text-to-speech with optional reference speech, descriptive captions, and Japanese inline tags for expressive delivery.

🌟 Key Features

• Japanese TTS: Generate speech from Japanese text, with or without additional conditioning.
• Caption-based voice design: Describe voice characteristics, emotion, or speaking style in Japanese.
• Voice cloning: Clone a voice using up to three reference clips from the same speaker.
• Japanese inline tags: Control emotion and delivery with tags such as [笑いながら], [驚いて], or [怒りを込めて]. See the tag list for more examples.
• Upstream compatibility: Use the same inference code as Drama 3; replace the model path with this checkpoint.

🎧 Audio Samples

1. Voice Design — Text + Caption

Text	Caption	Generated Audio
本日はお越しいただき、誠にありがとうございます。どうぞごゆっくりお過ごしください。	落ち着いた大人の男性。フォーマルな場で、深く響く声で丁寧かつ歓迎の意を込めて話している。
本日はお越しいただき、誠にありがとうございます。どうぞごゆっくりお過ごしください。	若く元気な女性の声。カフェの店員のように、明るくハキハキとした少し高めのトーンで話している。
すみません！この近くにコンビニってありますか？ちょっと急いでて、道に迷っちゃったみたいで	低めの声の男性が、丁寧に道を尋ねている。穏やかで礼儀正しく、余裕のある口調。
すみません！この近くにコンビニってありますか？ちょっと急いでて、道に迷っちゃったみたいで	若い女性が、慌てた様子で早口に話している。焦りと不安が声ににじんでいる。

2. Expressive Generation — Text + Caption + Reference Speech + Tags

Text with Tags	Reference Audio	Caption	Generated Audio
あははっ[笑い]、それ本当に言ってるの？…[呆れたように]まぁ、君らしいけどね。		余裕のある大人の男性。親しい相手に対して、くだけた雰囲気で呆れながらも楽しそうに話している。
ゲホッ、ゲホッ[咳き込みながら]…ごめん、少し休ませて。[弱々しく]今日はもう無理みたい。		体調が悪く、非常に苦しそうな若い女性。息も絶え絶えに、申し訳なさそうに弱々しい声で話している。

🚀 Usage

Follow the original Fish Speech instructions or the SGLang-Omni Drama 3 cookbook, using this model in place of fishaudio/drama-3.

After installing SGLang-Omni with Drama 3 support:

sgl-omni serve --model-path Aratako/Drama-3-Japanese --port 8000

Generate a single-speaker utterance without a reference or caption:

curl http://localhost:8000/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{"input":"こんにちは。今日はいい天気ですね。","response_format":"wav"}' \
  --output output.wav

🛠️ Training

Drama-3-Japanese was fine-tuned from Fish Audio Drama 3 using LoRA on approximately 70,000 hours of Japanese speech data for one epoch. The token embeddings and audio decoder were also trained.

Hyperparameter	Value
LoRA rank / alpha	32 / 32
Optimizer	AdamW
Learning rate (LoRA / embeddings / audio decoder)	2e-4 / 2e-5 / 5e-5

📊 Benchmarks

Both models were evaluated with SGLang-Omni at commit b658fa671d6a8a01d9b973754300837dff49d22d, using temperature 1.0, top-p 0.9, top-k 30, and a maximum of 2,048 acoustic frames (approximately 95.1 seconds). Each item was generated five times with independent random sampling.

To exclude likely generation failures, pairs are excluded if either model produces an empty output or audio outside the specified duration range.

Japanese Reading — Joyo Kanji Yomi Benchmark: Parakeet Edition

We used JKYB-Parakeet, with 13,536 texts × 5 repeats, without reference audio or captions. We treated outputs lasting 1–10 seconds as successful generations and retained pairs where both models met this criterion (66,423 / 67,680 pairs).

Model	Target Reading Accuracy ↑	Kana-CER ↓	Kana-CER clipped ↓	Sentence Kana-CER ↓	Standard Text CER ↓
Fish Audio Drama 3	92.021%	7.474%	6.285%	1.421%	5.699%
Drama-3-Japanese	93.188%	6.548%	5.511%	2.087%	5.572%

Scores are pooled across five runs using the official Parakeet Edition evaluator.

Voice Cloning — JVS

We evaluated 100 JVS speakers × 5 texts × 5 repeats, with one or three reference clips and no captions. Both models used the same references and transcripts.

We treated outputs lasting 1–20 seconds as successful generations and retained pairs where both models met this criterion (2,492 / 2,500 pairs with one reference and 2,429 / 2,500 pairs with three references).

Model	References	CAM++ Similarity ↑	ERes2Net Similarity ↑	CAM++ Top-1 ↑	ERes2Net Top-1 ↑	Whisper CER ↓
Fish Audio Drama 3	1	0.7333	0.6988	97.43%	95.83%	2.886%
Drama-3-Japanese	1	0.7280	0.6919	97.11%	96.67%	3.082%
Fish Audio Drama 3	3	0.7769	0.7402	99.75%	99.71%	3.023%
Drama-3-Japanese	3	0.7688	0.7336	99.30%	99.42%	3.008%

Scores are means across five repeats. The evaluation follows the Irodori-TTS voice cloning benchmark procedure, adapted to compare one versus three reference clips.

⚠️ Limitations

• Japanese-only fine-tuning: Performance in other languages has not been evaluated.
• Single-speaker TTS: Fine-tuning covers single-speaker utterances, not multi-speaker conversations, audio editing, multi-turn dialogue, or director's notes.
• Up to three references: Fine-tuning used at most three same-speaker reference clips. Larger reference sets are outside the training setup.
• Generation stability: As with the base model, generation may stop mid-sentence, include long periods of silence, or occasionally produce no audio.
• Caption and tag control: Instructions may be applied inconsistently, especially when reference speech, captions, and tags conflict. Automatically annotated training tags can also contain errors.

📜 License

This model inherits the Fish Audio Community License, Version 1.0 from original model.

🙏 Acknowledgments

• Fish Audio for Drama 3, the codec, and the official inference implementation.
• SGLang-Omni for benchmark inference.

Special thanks to gabrielclark3330 and kikouousya for supporting this project.