Authors:
Sauvik Bal,Lopa Mandal,DOI NO:
https://doi.org/10.26782/jmcms.2025.07.00007Keywords:
Chatbots,Dialog processing system,LLM,Low resource languages,Transformer model,Abstract
This study investigates the performance of dialogue processing systems in low-resource languages, specifically Bengali and Hindi, using advanced transformer-based models. English, a high-resource language, is used as a benchmark for comparison. Transformer models such as BERT, RoBERTa, FLAN-T5, DistilBERT, and GPT-2 are fine-tuned for question answering tasks across these languages. The evaluation includes metrics like F1 Score, Precision, Recall, and Exact Match to assess language-specific performance. The experiment reveals that GPT-2 delivers the highest exact match scores in Bengali and Hindi, while RoBERTa achieves superior F1 scores, indicating balanced performance. The study emphasizes the importance of monitoring training and validation losses to ensure effective model convergence and to identify overfitting. These findings highlight the potential of transformer models in improving dialogue systems for low-resource linguistic contexts.Refference:
I. Banerjee, Somnath, Sudip Kumar Naskar, Paolo Rosso, and Sivaji Bndyopadhyay. “Classifier combination approach for question classification for Bengali question answering system.” Sādhanā 44 (2019): 1-14. 10.1007/s12046-019-1224-8.
II. Baykara, Batuhan, and Tunga Güngör. 2023. : ‘Turkish Abstractive Text Summarization Using Pretrained Sequence-to-Sequence Models’. Natural Language Engineering 29(5): 1275–1304. 10.1017/S1351324922000195.
III. Cao K., Cheng W., Hao Y., Gan Y., Gao R., Zhu J. and Wu J., 2024.: ‘DMSeqNet-mBART: a state-of-the-art adaptive-DropMessage enhanced mBART architecture for superior Chinese short news text summarization’. Expert Systems with Applications, 257, p.125095. 10.1016/j.eswa.2024.125095.
IV. Chouhan, Sanjay, Shubha Brata Nath, and Aparajita Dutta. : ‘HindiLLM: Large Language Model for Hindi’. In International Conference on Pattern Recognition, pp. 255-270. Springer, Cham, 2025, 10.1007/978-3-031-78172-8.
V. Dabre Raj, Shrotriya Himani, Kunchukuttan Anoop, Puduppully Ratish, Khapra Mitesh and Kumar Pratyush. 2022.: ‘IndicBART: A Pre-trained Model for Indic Natural Language Generation’. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland. Association for Computational Linguistics, pp. 1849–1863, 10.18653/v1/2022.findings-acl.145.
VI. Das, Arijit, and Diganta Saha. “Question Answering System Using Deep Learning in the Low Resource Language Bengali.” Convergence of Deep Learning In Cyber‐IoT Systems and Security (2022): 207-230. 10.1002/9781119857686.ch10.
VII. Das, Mithun; Pandey, Saurabh Kumar; Sethi, Shivansh; Saha, Punyajoy; Mukherjee, Animesh.: ‘Low-Resource Counter speech Generation for Indic Languages: The Case of Bengali and Hindi’. arXiv preprint arXiv:2402.07262 (2024). 10.48550/arXiv.2402.07262.
VIII. Ghosh, Koyel, and Apurbalal Senapati. 2025. : ‘Hate Speech Detection in Low-Resourced Indian Languages: An Analysis of Transformer-Based Monolingual and Multilingual Models with Cross-Lingual Experiments’. Natural Language Processing 31(2): 393–414. 10.1017/nlp.2024.28.
IX. Glass M, Gliozzo A, Chakravarti R, Ferritto A, Pan L, Bhargav G P S, Garg D and Sil A. 2020.: ‘Span Selection Pre-training for Question Answering’. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 10.18653/v1/2020.acl-main.247.
X. Gruetzemacher, Ross, and David Paradice.: ‘Deep transfer learning & beyond: Transformer language models in information systems research’. ACM Computing Surveys (CSUR) 54, no. 10s (2022): 1-35. 10.1145/3505245.
XI. Han, Lifeng, Serge Gladkoff, Gleb Erofeev, Irina Sorokina, Betty Galiano, and Goran Nenadic.: ‘Neural machine translation of clinical text: an empirical investigation into multilingual pre-trained language models and transfer-learning’. Frontiers in Digital Health 6 (2024): 1211564. 10.3389/fdgth.2024.1211564.
XII. Haque, Rejwanul, Chao-Hong Liu, and Andy Way.: ‘Recent advances of low-resource neural machine translation’. Machine Translation 35, no. 4 (2021): 451-474. 10.1007/s10590-021-09281-1.
XIII. Hsu, T.-Y., Liu, C.-L., & Lee, H.-Y. (2019). : ‘Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model’. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, 5933–5940. 10.18653/v1/D19-1607.
XIV. Hwang M.H., Shin J., Seo H., Im J.S., Cho H. and Lee C.K., 2023.: ‘Ensemble-nqg-t5: Ensemble neural question generation model based on text-to-text transfer transformer’. Applied Sciences, 13(2), p.903, 10.3390/app13020903.
XV. Itsnaini, Qurrota A’yuna, Mardhiya Hayaty, Andriyan Dwi Putra, and Nidal AM Jabari.: ‘Abstractive Text Summarization using Pre-Trained Language Model” Text-to-Text Transfer Transformer (T5)’. ILKOM Jurnal Ilmiah 15, no. 1 (2023): 124-131. 10.33096/ilkom.v15i1.1532.124-131.
XVI. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim , Donghyeon Kim, Sunkyu Kim, Chan Ho So , Jaewoo Kang, 2020.: ‘BioBERT: a pre-trained biomedical language representation model for biomedical text mining’. Bioinformatics, Volume 36, Issue 4, February 2020, Pages 1234–1240, 10.1093/bioinformatics/btz682.
XVII. K. Pipalia, R. Bhadja and M. Shukla, ‘Comparative Analysis of Different Transformer Based Architectures Used in Sentiment Analysis’.” 2020 9th International Conference System Modeling and Advancement in Research Trends (SMART), Moradabad, India, 2020, pp. 411-415, 10.1109/SMART50582.2020.9337081.
XVIII. Katib I., Assiri F.Y., Abdushkour H.A., Hamed D. and Ragab M., 2023.: ‘Differentiating chat generative pretrained transformer from humans: detecting ChatGPT-generated text and human text using machine learning’. Mathematics, 11(15), p.3400, 10.3390/math11153400.
XIX. Kumari, N. and Singh, P., 2023.: ‘Hindi Text Summarization Using Sequence to Sequence Neural Network’. ACM Transactions on Asian and Low-Resource Language Information Processing, 22(10), pp.1-18, 10.1145/3624013.
XX. L. Zhou,: ‘LongT5-Mulla: LongT5 With Multi-Level Local Attention for a Longer Sequence’, in IEEE Access, vol. 11, pp. 138433-138444, 2023, doi: 10.1109/ACCESS.2023.3340854.
XXI. La Quatra, Moreno, and Luca Cagliero. 2023.: ‘BART-IT: An Efficient Sequence-to-Sequence Model for Italian Text Summarization’, Future Internet 15, no. 1: 15. 10.3390/fi15010015.
XXII. Lakew, Surafel M., Marcello Federico, Matteo Negri, and Marco Turchi.: ‘Multilingual neural machine translation for low-resource languages’. IJCoL. Italian Journal of Computational Linguistics 4, no. 4-1 (2018): 11-25. 10.4000/ijcol.531.
XXIII. Mastropaolo, Antonio, Nathan Cooper, David Nader Palacio, Simone Scalabrino, Denys Poshyvanyk, Rocco Oliveto, and Gabriele Bavota.: ‘Using transfer learning for code-related tasks’. IEEE Transactions on Software Engineering, Volume 49, Issue 4, Pages 1580 – 1598, 10.1109/TSE.2022.318329.
XXIV. Pakray, Partha, Alexander Gelbukh, and Sivaji Bandyopadhyay. 2025.: ‘Natural Language Processing Applications for Low-Resource Languages’. Natural Language Processing 31(2): 183–97. 10.1017/nlp.2024.33.
XXV. Pang, Jianhui, Fanghua Ye, Derek Fai Wong, Dian Yu, Shuming Shi, Zhaopeng Tu, and Longyue Wang. “Salute the classic: Revisiting challenges of machine translation in the age of large language models.” Transactions of the Association for Computational Linguistics 13 (2025): 73-95. 10.1162/tacl_a_00730.
XXVI. Romim N., Ahmed M., Talukder H. and Saiful Islam M., 2021.: ‘Hate speech detection in the bengali language: A dataset and its baseline evaluation’. In Proceedings of International Joint Conference on Advances in Computational Intelligence: IJCACI 2020 (pp. 457-468). Springer Singapore. 10.1007/978-981-16-0586-4_37.
XXVII. Roy P.K., Bhawal S and Subalalitha C.N., 2022. : ‘Hate speech and offensive language detection in Dravidian languages using deep ensemble framework’. Computer Speech & Language, 75, p.101386. 10.1016/j.csl.2022.101386.
XXVIII. H. Amanda Tan, E. S. Aung and H. YAMANA,: ‘Two-stage fine-tuning for Low-resource English-based Creole with Pre-Trained LLMs’, 2023 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE), Nadi, Fiji, 2023, pp. 1-6, 10.1109/CSDE59766.2023.10487143.
XXIX. Tahsin Mayeesha, Tasmiah, Abdullah Md Sarwar, and Rashedur M. Rahman. 2020.: ‘Deep Learning Based Question Answering System in Bengali’. Journal of Information and Telecommunication 5 (2): 145–78. 10.1080/24751839.2020.1833136.
XXX. Zhang H., Song H., Li S., Zhou M. and Song D., 2023.: ‘A survey of controllable text generation using transformer-based pre-trained language models’. ACM Computing Surveys, Volume 56, Issue 3, Article No.: 64, Pages 1 – 37. 10.1145/3617680.