Fine-Tuning Transformer Models for Indonesian Text Summarization and Translation: A Comparative Study of PEGASUS, T5, and Komodo-7B

Abstract

Bahasa Indonesia ranks among the most widely spoken languages in the world. However, Natural Language Processing (NLP) research for this language remains relatively limited. This study evaluates the effectiveness of three Transformer-based models namely PEGASUS, T5, and Komodo-7B on two Indonesian-language NLP tasks: text summarization and machine translation. Fine-tuning was conducted using the IndoSum dataset for summarization and a parallel Indonesian-English corpus for translation. Performance was assessed using ROUGE metrics for summarization and BLEU scores for translation, supplemented by qualitative analysis of model outputs. Experimental results indicate that PEGASUS achieved the highest ROUGE-L score (0.652) in summarization when fine-tuned with a learning rate of 1e-4 and batch size of 4. In the translation task, Komodo-7B yielded the best performance with a BLEU score of 49.33, producing translations closely aligned with reference outputs. The T5 model exhibited consistent and competitive results, particularly in resource-constrained settings. Qualitative findings further reinforce Komodo-7B's semantic precision in translation and PEGASUS's contextual adequacy in summarization. These results highlight the potential of Transformer models for Indonesian NLP tasks and provide valuable insights into model fine-tuning strategies in low-resource linguistic environments.