Panga-Azazia/LFM2.5-350M-TTS-v4

Model Card for LFM2.5-350M-TTS

This model is our finetuned TTS model from Panga-Azazia/LFM2.5-350M-TTS, on 21 languages, 19 African languages, English and French.

Supported Languages

• Bambara (bm)
• Bomu (bmq)
• Jula (dyu)
• Dogon (dts)
• Minianka(Mamara) (myk)
• Fulfulde (ffm)
• Konabere (bbo)
• Songhay-Timbuktu (thq)
• Moore (mos)
• Khassonke (kao)
• Senoufo-Shempire (spp)
• Senoufo-Sicite (sep)
• Senoufo-Nyarafolo (sev)
• Senoufo-Djimini (dyi)
• Malinke (mwk)
• Wolof (wol)
• Pulaar (fuc)
• Jola-Fonyi (dyo)
• Tamasheq (taq)
• English (en)
• French (fr)

Training Data

The model was finetuned on dataset built from Bible, Kppwdfgu1/Fula-pular and galsenai/anta_women_tts .

Usage

Installation

pip install unsloth snac
pip install git+https://github.com/sudoping01/bambara-text-normalization.git

Using unsloth

from unsloth import FastLanguageModel
import torch

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="Panga-Azazia/LFM2.5-350M-TTS-v4",
    max_seq_length=2048,
    dtype=None,
    load_in_4bit=False,
)

FastLanguageModel.for_inference(model)

Using transformers

tokenizer = AutoTokenizer.from_pretrained("Panga-Azazia/LFM2.5-350M-TTS-v4", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    "Panga-Azazia/LFM2.5-350M-TTS-v4",
    trust_remote_code=True
)

Normalisation du Bambara

The bambara_normalizer module is used only for Bambara. It is used to normalize certain parts of the text before generation.

For more information on how to use it, see: bambara-text-normalizer

import torch
from snac import SNAC
import soundfile as sf
import numpy as np
from unsloth import FastLanguageModel
from bambara_normalizer import normalize_dates_in_text, normalize_measurements_in_text, normalize_numbers_in_text, normalize_times_in_text


CODE_END_TOKEN_ID = 7

SNAC_MIN_ID = 64402
CODE_TOKEN_OFFSET = 64402
SNAC_MAX_ID = 93073
SNAC_TOKENS_PER_FRAME = 7



def build_prompt(text: str, lang : str) -> str:

  text = normalize_dates_in_text(text)
  text = normalize_times_in_text(text)
  text = normalize_measurements_in_text(text)
  text = normalize_numbers_in_text(text)

  messages = [
      #{"role" : "system", "content" : prompt},
      {"role": "user", "content": f"<description=Read the following text in {lang}> {text}"}
      ]

  return messages



def extract_snac_codes(token_ids: list) -> list:
    """Extract SNAC codes from generated tokens."""
    try:
        eos_idx = token_ids.index(CODE_END_TOKEN_ID)
    except ValueError:
        eos_idx = len(token_ids)

    snac_codes = [
        token_id for token_id in token_ids[:eos_idx]
        if SNAC_MIN_ID <= token_id <= SNAC_MAX_ID
    ]

    return snac_codes


def unpack_snac_from_7(snac_tokens: list) -> list:
    """Unpack 7-token SNAC frames to 3 hierarchical levels."""

    if snac_tokens and snac_tokens[-1] == CODE_END_TOKEN_ID:
      snac_tokens = snac_tokens[:-1]

    frames = len(snac_tokens) // SNAC_TOKENS_PER_FRAME
    snac_tokens = snac_tokens[:frames * SNAC_TOKENS_PER_FRAME]

    if frames == 0:
        return [[], [], []]

    l1, l2, l3 = [], [], []

    for i in range(frames):
        slots = snac_tokens[i*7:(i+1)*7]
        l1.append((slots[0] - CODE_TOKEN_OFFSET) % 4096)
        l2.extend([
            (slots[1] - CODE_TOKEN_OFFSET) % 4096,
            (slots[4] - CODE_TOKEN_OFFSET) % 4096,
        ])
        l3.extend([
            (slots[2] - CODE_TOKEN_OFFSET) % 4096,
            (slots[3] - CODE_TOKEN_OFFSET) % 4096,
            (slots[5] - CODE_TOKEN_OFFSET) % 4096,
            (slots[6] - CODE_TOKEN_OFFSET) % 4096,
        ])

    return [l1, l2, l3]


snac_model = SNAC.from_pretrained("hubertsiuzdak/snac_24khz").eval()

def main(text = "An me kɛrɛlamana dɔw don a tigilamɔgɔ la." ,lang = "Bambara", temp = .1, top_p = 1.05, top_k = 50,
         max_tokens = 2048, min_tokens = 28, rp = 1.25):

    messages = build_prompt(text, lang)

    chat = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

    print(chat)

    inputs = tokenizer(chat, return_tensors="pt").to("cuda")

    print(inputs)


    with torch.inference_mode():
        outputs = model.generate(
            **inputs,
            max_new_tokens=max_tokens, 
            min_new_tokens=min_tokens, 
            temperature=temp,
            top_p=top_p,
            top_k = top_k,
            repetition_penalty=rp,  
            do_sample=True,
            eos_token_id=tokenizer.eos_token_id,  
            pad_token_id=tokenizer.pad_token_id,
            use_cache=True,
        )


    generated_ids = outputs[0, inputs['input_ids'].shape[1]:].tolist()

    print(generated_ids)
    print(len(generated_ids))
    generated_ids = generated_ids[1:-1]
    print(len(generated_ids))


    # Unpack SNAC tokens to 3 hierarchical levels
    levels = unpack_snac_from_7(generated_ids)
    frames = len(levels[0])

    print(f"Unpacked to {frames} frames")
    print(f"   L1: {len(levels[0])} codes")
    print(f"   L2: {len(levels[1])} codes")
    print(f"   L3: {len(levels[2])} codes")

    # Convert to tensors
    device = "cuda" if torch.cuda.is_available() else "cpu"
    codes_tensor = [
        torch.tensor(level, dtype=torch.long, device=device).unsqueeze(0)
        for level in levels
    ]

    # Generate final audio with SNAC decoder
    print("\n[4/4] Decoding to audio...")
    with torch.inference_mode():
        z_q = snac_model.quantizer.from_codes(codes_tensor)
        audio = snac_model.decoder(z_q)[0, 0].cpu().numpy()

    # Trim warmup samples (first 2048 samples)
    if len(audio) > max_tokens:
        audio = audio[max_tokens:]

    duration_sec = len(audio) / 24000
    print(f"Audio generated: {len(audio)} samples ({duration_sec:.2f}s)")

    # Save your emotional voice output
    output_file = f"{text}-{lang}.wav"
    sf.write(output_file, audio, 24000)

Examples

A Bomu audio sample.

main(
    lang = "Bomu",
    text="Ho lana a sinciri cinnu, ho a ziya berobe nɛ zeni be nɛ mi, a nucoro yire ɓɛɛ da mi."
)

A Fulfulde audio sample.

main(
    lang = "fulfulde",
    text = "Goonɗinal yo taƴorde heɓan ko fawi jikke e mum e hoolaade ko yiyaali ana woodi."
  )

A Wolof audio sample.

main(
    lang = "wolof",
    text= "iniwersite bu ndar, ëmb na juróorum-ñaari bërëbu tàggat ak gëstu.",
  )

A Bambara audio sample.

main(
    lang = "Bambara",
    text="Ayiwa, an jigi bɛ fɛn minnu kan, dannaya ye olu jɛyali ye. Dannaya bɛ fɛn yebaliw seereya kɛ."
  )

A Konabere audio sample.

main(
    lang = "konabere",
    text="Laɩnapɩ tɩɛɛ n’ɩ kɔ nʋ: ŋma daŋ ma ma tɩgɛɩmapɩ ti, ŋmɔ na jisigefaa, ŋma fɩrɛɛ ma m’ɩ jaa ko, ŋmɔ na sɩɛrɩpɩ.",
  )

A Khassonke audio sample.

main(
    lang = "khassonke",
    text="Lannaya mu ɲin ne ti xo: Í gigo be fenŋo men xan, wo soto be xoyiyanin í ma; walla í ɲa nte fenŋo men na, í b'a yitanin xo í ɲa be wo la.",
  )

A English audio sample.

main(
    lang = "english",
    text="Now faith is the substance of things hoped for, the evidence of things not seen.",
  )

A Songhay-timbuktu audio sample.

main(
    lang = "songhay-timbuktu",
    text="Naanay ci, ka kow sika hayadiyo ra ka ni go tammaha, nda ka tabatindi hayadiyo ka ni si guna.",
  )

A Moore audio sample.

main(
    lang = "moore",
    text="Wẽnd sẽn pʋlem tõnd vʋʋsgã, tɩ b kãabga ket n yɛsẽ wã, d maan dabeem, tɩ yãmb nin-yend na n wa yet tɩ b basa yẽ kɛɛnga.",
  )

A French audio sample.

main(
    lang = "french",
    text="Or la foi est une ferme assurance des choses qu’on espère, une démonstration de celles qu’on ne voit pas.",
  )

A Fulfulde audio sample.

main(
    lang = "fulfulde",
    text = "Goonɗinal yo taƴorde heɓan ko fawi jikke e mum e hoolaade ko yiyaali ana woodi."
  )

A Dogon audio sample.

main(
    lang = "Dogon",
    text="Kijɛ emɛ dɔmɔ bɛbɛɛdɛin ga diyɛ daanu ye bii gɔ, gagawu gɔ woi. Gagawu, kijɛ iyemɔɔlɛ gɔbe yɔɔin ga diyɛ daanu ye bii.",
  )

A Minianka audio sample.

main(
    lang = "Minianka",
    text="Ayiwa, nʼa daa wʼa wèe wo tadaŋa ki wo yaŋmuyɔ yi keree fiinŋɛ, na wèe pye wèe ya tiin ni li ni na yi wa wèe mu na xɔ. Nʼa daa wʼa wèe pye wèe ya sɔɔ li na na can wa yaŋmuɲabaaya yi wo keree ki ni.",
  )

A jula audio sample.

main(
    lang = "jula",
    text="Lanaya ye min ye, o ye nin ye i be la fɛɛn dɔ la, k'i jigi la a kan k'a sɔrɔ i m'a ye ka ye. Fɔlɔfɔlɔ mɔgɔw ye tɔgɔɲuman sɔrɔ Ala fɛ u ka lanaya kosɔn.",
  )

A Senoufo-Djimini audio sample.

main(
    lang = "Senoufo-Djimini",
    text="Na we kaga jɛn mbe yo we jigi wi yɛn yaraga ŋga na we yaa ki ta, ko ki yɛn tagawa we. Tagawa po pi yɛn yaraga ŋga we woro na yaan, we taga ki na we yo ki yɛn kaselege.",
  )

A Senoufo-Shempire audio sample.

main(
    lang = "Senoufo-Shempire ",
    text="Ɲaha ku ɲyɛ dániyaŋi yɛ? Mà ma sɔ̀nŋɔre taha yaaga na maa ntɛ̀ɛn ná l'e mu sí ku ta, lire ɲyɛ mɛ yaage ku ɲyɛ mu ɲyɛ na ku ɲaa mɛ, maa dá ku na mu à jwo mu ɲyii na ɲyɛ ku na ke, lire li ɲyɛ dániyaŋi.",
  )

A Senoufo-Nyarafolo audio sample.

main(
    lang = "Senoufo-Nyarafolo",
    text="Tɛ́ngɛlɛ ni kɔri wire yē bè tɛ́ngɛ ki nɛ̄ gìi nɛ̄ muɔ nyɛ́nì muɔ sɔ̀ngirɔ taʔa dè, muɔ bé ki taa, bè ki cɛ́n gàa muɔ wè nɛ nyaʔa gèʔ, kiī kányiʔɛ.",
  )

A Malinke audio sample.

main(
    lang = "malinke",
    text="Lannaya dun ye le xo ilan jigi ye hɛn mɛnnu xan, olu jusigilan ne ye, ilan ɲa ye ntɛ hɛn mɛnnu la, olu seeriyali le ye.",
  )

A tamasheq audio sample.

main(
    lang = "tamasheq",
    text="Immun, ălmăɣna-net a-s a-wa dăɣ-ijăr ăwadəm ăṭṭăma, ad-has-umas šund a-s insa data-s; meɣ daɣ, a-wa-dăɣ wăr-ihənnəy s-tiṭṭawen-net, ad-has-umas šund ăsiḍălalăk dăɣ-s."
  )

Other examples

You can also use one language's voice to read text written in another.

A English audio sample. Here the model are reading English using Wolof accent.

main(
    lang = "wolof",
    text="Now faith is the substance of things hoped for, the evidence of things not seen.",
  )

Ressources

If you want to finetune or to know how to transform an LLM to a TTS model you can use notebooks bellow.

Name	Description	Notebook
PT (Unsloth)	Pre-Training using Unsloth for TTS.
SFT (Unsloth)	Supervised Fine-Tuning with LoRA using Unsloth.

Limiations

We are noticed some misalignment on Moore dataset, so the speech generated could probably not reflect the given Text.

The french and English still not yet at the point, the model still struggle to generate clear speech in them.

Citation

The Fula-pular dataset used is from Kppwdfgu1/Fula-pular . The Wolof dataset used is from galsenai/anta_women_tts

@dataset{galsenai_anta,
  author       = {GalsenAI-Lab},
  abstract     = {Anta is a cleaned version of the Baamtu Wolof TTS dataset. We extracted the female voice, denoised it and enhanced it with the Resemble Enhance library. We also cleaned up the annotations by removing special characters, emojis, Arabic and Russian characters. We've corrected a few annotation errors, but there are potentially many more to come. Some lines and audios judged not qualitative enough have been removed from the dataset, reducing its size to 19947 annotated audios.},
  title        = {Anta: Wolof female-voice Text-to-Speech dataset},
  howpublished = {\url{https://huggingface.co/datasets/galsenai/anta_women_tts}},
  year         = {2024}
}