indictrans: English words in Indian-language text, written in the request's script

Runtime assets of the code-mix transliteration layer of the Finesse TTS (utils/codemix_xlit): English words inside Indic text are rewritten in the language's script, spelled the way an Indian-English speaker says them, so an Indic voice reads them correctly; text typed entirely in Latin is read word by word, the language's own words back into its script. 22 languages (Dogri and Santali through the Hindi model). From 1.5, word lists for the non-Indic languages too (intl/).

Version 1.5.8+666503b124aa. Built by tools/codemix_xlit/build_assets.py; manifest.json lists every file with its size and SHA-256.

Contents

• LICENSE, NOTICE: the licences and attributions below, inside the bundle, so a copy of the folder keeps them
• model/: the IndicXlit en->indic v1.0 transformer converted to CTranslate2
• lexicon/<code>.reviewed.json, lexicon/<code>.generated.json: reviewed and generated spellings
• rescore/: pruned IndicXlit word-frequency dictionaries
• cmudict.json, english_*: English pronunciations, vocabulary and reviewed corrections
• indic_words.json: Indian words read natively; ranker.json: candidate ranker
• roman/<code>.json: how people type each language's words in Latin, how often, and the other words a typing can be (alts)
• native_freq/<token>.json: word frequencies of the eleven languages without a rescore/ list, for the language's own words typed in Latin
• roman_ctx/<code>.json: for typings that are two different words, how often each is seen next to the words that tell them apart (sentence context)
• names/<code>.json: Indian given names, surnames and place names typed in Latin and how the language writes them (tools/codemix_xlit/build_names.py), read for a capitalised word (Sneha स्नेहा, not स्नेह)
• intl/<code>/lexicon.tsv: for the non-Indic languages (utils/codemix_xlit/intl), English words and the spelling each language writes them in (Japanese katakana; Standard, Egyptian, Gulf, Najdi and Hijazi Arabic; Spanish), with the source of every row; built by tools/codemix_xlit/intl/build_<lang>.py
• intl/en/<script>.tsv: for English requests, words in another script and their Latin spelling (names, places, loanwords back to English, everyday words as people type them, kanji readings)

Sources and licences

license: other because the files carry different licences; each is listed here with its notice.

• model/, rescore/, native_freq/: the IndicXlit model and word-frequency dictionaries, AI4Bharat, MIT licence (https://github.com/AI4Bharat/IndicXlit). Notice:

  MIT License

  Copyright (c) 2022 AI4Bhārat

  Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

  The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

• cmudict.json: the CMU Pronouncing Dictionary (https://github.com/cmusphinx/cmudict), BSD-style licence. Notice:

  Copyright (C) 1993-2015 Carnegie Mellon University. All rights reserved.

  Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. The contents of this file are deemed to be source code.
  2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

  This work was supported in part by funding from the Defense Advanced Research Projects Agency, the Office of Naval Research and the National Science Foundation of the United States of America, and by member companies of the Carnegie Mellon Sphinx Speech Consortium. We acknowledge the contributions of many volunteers to the expansion and improvement of this dictionary.

  THIS SOFTWARE IS PROVIDED BY CARNEGIE MELLON UNIVERSITY ``AS IS'' AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL CARNEGIE MELLON UNIVERSITY NOR ITS EMPLOYEES BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

• english_vocab.txt, english_pron_vocab.txt: English word lists derived from wordfreq (https://github.com/rspeer/wordfreq), whose data is CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/); these lists are shared under the same licence.

• roman/ of bn, gu, hi, kn, ml, mr, pa, sd, ta, te, ur: counts derived from the romanized sentences of the Dakshina dataset (Roark et al. 2020, https://github.com/google-research-datasets/dakshina), CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/); these tables are shared under the same licence.

• roman_ctx/: neighbour counts from AI4Bharat Sangraha, verified subset (Khan et al. 2024, https://huggingface.co/datasets/ai4bharat/sangraha), CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/).

• names/: the English and native-script labels of people with Indian citizenship, of the name items they carry and of India's states, union territories and places of 100,000 people, from Wikidata (https://www.wikidata.org), CC0 1.0 (https://creativecommons.org/publicdomain/zero/1.0/), checked against the IndicXlit model above.

• Lexicons, corrections, ranker and the other romanized tables: derived from the above as described below.

• intl/ja/: UniDic-cwj 2023.02 (NINJAL, BSD-3-Clause of its GPL/LGPL/BSD choice) and SudachiDict (Works Applications, Apache-2.0) loanword entries, with forms derived through CMUdict

• intl/arb/, intl/arz/, intl/afb/, ...: Wikidata labels (CC0) and spellings from Gemma 4 E2B (Google, Apache-2.0) where two prompts agree, checked against the CMUdict renderer and reviewed

• intl/arz/arabizi_table.json, intl/arz/arabizi_freq.tsv: the Egyptian Arabizi everyday-word table, written for this project (tools/codemix_xlit/intl/arabizi/conv_arz.json), and word counts from Tatoeba sentences in Egyptian and Standard Arabic (https://tatoeba.org, by the Tatoeba contributors, CC BY 2.0 FR, https://creativecommons.org/licenses/by/2.0/fr/), the Common Voice Arabic sentence collector (CC0 1.0) and Wikidata lexemes (CC0 1.0); only the counts are included

• intl/es/: words measured on the Finesse Spanish voice, respelled from CMUdict

• intl/en/: Wikidata labels (CC0), SudachiDict kanji readings (Apache-2.0), this bundle's own lists reversed; rows marked roman come from the romanized tables (roman/), which use Dakshina counts and are shared under CC BY-SA 4.0 like them

Who wrote the generated parts

• lexicon/<code>.generated.json: written offline by the layer's own pipeline (build_assets.py lexicon: IndicXlit's candidates, spellings rendered from the pronunciation, the ranker). Its entries that did not read back as the English word, among each language's 5,000 most frequent, were reviewed by the LLM graders below and the errors corrected in the reviewed lexicon.
• english_extra_prons.json (Indian-English readings of words the CMU dictionary lacks: names, brands, foreign words), english_pron_overrides.json, indic_words.json, the gold spellings in lexicon/<code>.reviewed.json, and the everyday words in roman/: written by a large language model, run as per-language writer and grader agents under fixed spelling conventions. They were reviewed by further agents of the same model (a reading is read back against the English pronunciation; a spelling is graded C/A/W blind against plain IndicXlit's). Two independent graders agreed on 96.2-97.7% of labels.
• ranker.json: fitted on those graders' labels.
• The accuracy figures published for this bundle come from the same model graders. No native speaker has reviewed any of these files yet; treat them as unverified for external claims.