A Survey on Transfer Learning for Low-Resource Language Technologies: A Focused Survey of Transformer-Based Code-Mixed Language Identification and Zero-Shot Multilingual Speech Synthesis

Abstract

Most progress in natural language and speech processing has been achieved for a small group of languages with abundant labelled data, while languages with limited corpora and tooling continue to be underserved. This survey examines how transfer learning is used to narrow that gap in two complementary settings: word-level language identification of romanized Malayalam–English code-mixed social media text, and zero-shot multilingual multi-speaker text-to-speech synthesis for Indonesian, Javanese, and Sundanese. Two peer-reviewed studies are analysed in depth and placed in context using the literature they cite. A taxonomy of transfer strategies is proposed that distinguishes data-centric corpus construction, full fine-tuning of pretrained transformer encoders, shape-matched and partial parameter transfer, frozen pretrained modules used as feature extractors or perceptual critics, and explicit conditioning on language, speaker, and style. The historical development of word-level language identification and of speaker adaptation in neural speech synthesis is traced, and the evaluation practices of both lines of work are compared. The analysis shows that both studies report large gains over their in-study baselines, but that these gains depend on heuristically labelled or pooled small datasets, single evaluation splits, and, in the speech case, an objective metric that shares its encoder with the training loss. Open problems include reliable gold-standard annotation, class imbalance, orthographic variation, decoupled evaluation, and responsible use of voice cloning. Research directions such as code-mixed speech synthesis driven by token-level language tags are outlined.

Introduction

The text discusses the challenges of developing language technologies for low-resource languages and explains how transfer learning can improve both text processing and speech synthesis when training data are limited.

The survey focuses on two main applications. The first is word-level language identification (WLLI), which identifies the language of each word in code-mixed text, such as Malayalam–English social media messages written in the Latin script. This task is challenging because of spelling variations, mixed-language words, ambiguous terms, and limited annotated datasets. The reviewed study addresses these issues by fine-tuning pretrained transformer models, including BERT-family architectures, on a newly collected and labelled Malayalam–English dataset.

The second application is neural text-to-speech (TTS), which generates natural speech in different languages and voices. The reviewed study transfers knowledge from an English speech-synthesis model to three low-resource Indonesian languages: Indonesian, Javanese, and Sundanese. It uses pretrained speaker-verification models, style encoders, and speaker-level training objectives to improve voice similarity and support zero-shot synthesis for speakers not encountered during training.

The survey identifies several transfer-learning strategies, including collecting and labelling new datasets, combining data from related languages, fine-tuning pretrained models, transferring selected neural network parameters, and using speaker embeddings and style representations. These techniques help reduce dependence on large target-language datasets, although their effectiveness depends on data quality, model compatibility, and evaluation methods.

The review uses a focused narrative approach based on two peer-reviewed studies rather than a systematic literature search. It compares their methodologies, datasets, evaluation metrics, reported results, and limitations. It also highlights unresolved challenges, including insufficient annotated data, inconsistent spelling, limited generalization to unseen languages and speakers, computational costs, and the reliability of evaluation protocols.

Conclusion

This survey examined transfer learning for low-resource language technologies through two complementary studies: transformer-based word-level language identification for romanized Malayalam–English text and zero-shot multilingual multi-speaker speech synthesis for Indonesian, Javanese, and Sundanese. A taxonomy of transfer strategies was proposed, ranging from data-centric corpus construction to full fine-tuning, partial parameter transfer, frozen pretrained modules, and explicit conditioning. Both studies show that knowledge from high-resource sources can make low-resource tasks tractable: pretrained encoders reached very high agreement with the corpus labels, and partial transfer combined with style conditioning and a speaker-level loss enabled intelligible speech with markedly higher speaker similarity for unseen voices. At the same time, the analysis identified factors that limit how far these results generalise, including heuristically derived labels, single evaluation splits, coupling between the training objective and an evaluation metric, and narrow linguistic and platform coverage. Progress will depend as much on better benchmarks and evaluation practice as on new architectures, and the combination of code-mixed language identification with multilingual speech synthesis offers a natural next step.

References

[1] M. A. Hedderich, L. Lange, H. Adel, J. Strötgen, and D. Klakow, “A survey on recent approaches for natural language processing in low-resource scenarios,” in Proc. Conf. North Amer. Chapter Assoc. Comput. Linguistics: Human Lang. Technol. (NAACL-HLT), 2021, pp. 2545–2568. [2] S. Thara and P. Poornachandran, “Transformer based language identification for Malayalam-English code-mixed text,” IEEE Access, vol. 9, pp. 118?837–118?850, 2021. [3] U. Barman, A. Das, J. Wagner, and J. Foster, “Code mixing: A challenge for language identification in the language of social media,” in Proc. 1st Workshop Comput. Approaches Code Switching, 2014, pp. 13–23. [4] K. Azizah and W. Jatmiko, “Transfer learning, style control, and speaker reconstruction loss for zero-shot multilingual multi-speaker text-to-speech on low-resource languages,” IEEE Access, vol. 10, pp. 5895–5911, 2022. [5] S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Trans. Knowl. Data Eng., vol. 22, no. 10, pp. 1345–1359, 2010. [6] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, ?. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2017, pp. 5998–6008. [7] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” arXiv:1810.04805, 2018. [8] L. Martin, B. Müller, P. J. O. Suárez, Y. Dupont, L. Romary, É. V. de la Clergerie, D. Seddah, and B. Sagot, “CamemBERT: A tasty French language model,” arXiv:1911.03894, 2019. [9] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, “Unsupervised cross-lingual representation learning at scale,” arXiv:1911.02116, 2019. [10] J. Shen et al., “Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), 2018, pp. 4779–4783. [11] Y. Wang, D. Stanton, Y. Zhang, R. J. Skerry-Ryan, E. Battenberg, J. Shor, Y. Xiao, Y. Jia, F. Ren, and R. A. Saurous, “Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis,” in Proc. 35th Int. Conf. Mach. Learn. (ICML), 2018, pp. 5180–5189. [12] D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur, “X-vectors: Robust DNN embeddings for speaker recognition,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), 2018, pp. 5329–5333. [13] L. Wan, Q. Wang, A. Papir, and I. L. Moreno, “Generalized end-to-end loss for speaker verification,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), 2018, pp. 4879–4883. [14] G. Chittaranjan, Y. Vyas, K. Bali, and M. Choudhury, “Word-level language identification using CRF: Code-switching shared task report of MSR India system,” in Proc. 1st Workshop Comput. Approaches Code Switching, 2014, pp. 73–79. [15] G. Molina, F. AlGhamdi, M. Ghoneim, A. Hawwari, N. Rey-Villamizar, M. Diab, and T. Solorio, “Overview for the second shared task on language identification in code-switched data,” in Proc. 2nd Workshop Comput. Approaches Code Switching, 2016, pp. 40–49. [16] A. Jaech, G. Mulcaire, M. Ostendorf, and N. A. Smith, “A neural model for language identification in code-switched tweets,” in Proc. 2nd Workshop Comput. Approaches Code Switching, 2016, pp. 60–64. [17] Y. Jia, Y. Zhang, R. J. Weiss, Q. Wang, J. Shen, F. Ren et al., “Transfer learning from speaker verification to multispeaker text-to-speech synthesis,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2018, pp. 4485–4495. [18] E. Cooper, C.-I. Lai, Y. Yasuda, F. Fang, X. Wang, N. Chen, and J. Yamagishi, “Zero-shot multi-speaker text-to-speech with state-of-the-art neural speaker embeddings,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), 2020, pp. 6184–6188. [19] Z. Cai, C. Zhang, and M. Li, “From speaker verification to multispeaker speech synthesis, deep transfer with feedback constraint,” in Proc. INTERSPEECH, 2020, pp. 3974–3978. [20] D. Mave, S. Maharjan, and T. Solorio, “Language identification and analysis of code-switched social media text,” in Proc. 3rd Workshop Comput. Approaches Linguistic Code-Switching, 2018, pp. 51–61.

Copyright

Copyright © 2026 Bhuvana S R, Niranjana S S, Mrs. Jipsy Jose, Mrs. Viji C, Mrs. Sagara M R. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.