ITEBIS PGRI Dewantara Academic Service Chatbot Based on IndoBERT and Retrieval-Augmented Generation (RAG)
DOI:
https://doi.org/10.15294/rji.v4i2.62938Keywords:
Chatbot, IndoBERT, Retrieval-Augmented Generation, Natural Language Processing, Academic ServicesAbstract
Abstract. Academic administrative services must stay responsive beyond regular hours, yet repetitive student inquiries add to staff workload. Retrieval-Augmented Generation (RAG) with a fine-tuned language model offers a way to automate accurate, round the clock responses in Indonesian.
Purpose: This study develops a prototype academic-service chatbot for ITEBIS PGRI Dewantara Jombang, integrating a fine-tuned IndoBERT model with RAG architecture to address limited-service hours and repetitive handling of routine student inquiries.
Methods/Study design/approach: The system used 2,517 SQuAD-format question-answer pairs, split into 72% training (1,812), 8% validation (201), and 20% test (504). IndoBERT (indobenchmark/indobert-base-p1) was fine-tuned as an extractive reader for five epochs (batch size 8, learning rate 3 × 10⁻⁵) with early stopping, then converted to ONNX format and INT8-quantized for CPU-based local deployment. The prototype separates a Laravel web interface from a FastAPI backend running the RAG pipeline, retrieving passages from a MySQL knowledge base via FAISS search (top-k = 5, min score 0.30) and catching frequent query-answer pairs in Redis to cut redundant database access and inference latency. Evaluation used black-box testing across five scenarios.
Result/Findings: On the 504-pair test set, the model achieved an Exact Match of 71.32% and an F1-Score of 86.63%. All five black-box scenarios passed, including the system’s ability to give database-grounded responses within the tested scenarios rather than fabricated answers, though retrieval for categories with fewer passages (e.g., finance) was more sensitive to question phrasing
Novelty/Originality/Value: The novelty lies not in any single element but in combining four components: a fine-tuned IndoBERT reader for Indonesian, an academic-domain RAG architecture, a fully functional end-to-end web prototype rather than model evaluation alone, and an ONNX INT8-quantized reader paired with a Redis caching layer that jointly reduce inference latency and database load a combination not jointly addressed in prior IndoBERT or RAG-based chatbot studies.
References
[1] K. F. Hew, W. Huang, J. Du, and C. Jia, “Using chatbots to support student goal setting and social presence in fully online activities: learner engagement and perceptions,” J Comput High Educ, vol. 35, no. 1, pp. 40–68, Apr. 2023, doi: 10.1007/s12528-022-09338-x.
[2] M. Kremantzis, A. Chondrogianni, and A. Essien, “Evaluating the impact of AI chatbots on student support and engagement in UK higher education,” Journal of Further and Higher Education, pp. 1–23, Jul. 2026, doi: 10.1080/0309877X.2026.2693084.
[3] I. Engeness, M. Nohr, and T. Fossland, “Investigating AI Chatbots’ Role in Online Learning and Digital Agency Development,” Education Sciences, vol. 15, no. 6, p. 674, May 2025, doi: 10.3390/educsci15060674.
[4] G. Caldarini, S. Jaf, and K. McGarry, “A Literature Survey of Recent Advances in Chatbots,” Information, vol. 13, no. 1, p. 41, Jan. 2022, doi: 10.3390/info13010041.
[5] N. S. Amarnath and R. Nagarajan, “An Intelligent Retrieval Augmented Generation Chatbot for Contextually-Aware Conversations to Guide High School Students,” in 2024 4th International Conference on Sustainable Expert Systems (ICSES), Kaski, Nepal: IEEE, Oct. 2024, pp. 1393–1398. doi: 10.1109/ICSES63445.2024.10762977.
[6] T. I. Ramadhan, A. Supriatman, and T. R. Kurniawan, “Evaluasi dan Implementasi Indobert Question Answering (QA) pada Domain Spesifik Menggunakan Mean Reciprocal Rank,” Jurnal Algoritma, vol. 21, no. 1, pp. 180–188, May 2024, doi: 10.33364/algoritma/v.21-1.1542.
[7] Anugerah Simanjuntak et al., “Research and Analysis of IndoBERT Hyperparameter Tuning in Fake News Detection,” Jurnal Nasional Teknik Elektro dan Teknologi Informasi, vol. 13, no. 1, pp. 60–67, Feb. 2024, doi: 10.22146/jnteti.v13i1.8532.
[8] R. I. Perwira, V. A. Permadi, D. I. Purnamasari, and R. P. Agusdin, “Domain-Specific Fine-Tuning of IndoBERT for Aspect-Based Sentiment Analysis in Indonesian Travel User-Generated Content,” J. Inf. Syst. Eng. Bus. Intell., vol. 11, no. 1, pp. 30–40, Mar. 2025, doi: 10.20473/jisebi.11.1.30-40.
[9] B. V. Kartika, M. J. Alfredo, and G. P. Kusuma, “Fine-Tuned IndoBERT Based Model and Data Augmentation for Indonesian Language Paraphrase Identification,” RIA, vol. 37, no. 3, pp. 733–743, Jun. 2023, doi: 10.18280/ria.370322.
[10] R. Santosa, A. B. Nusantara, and S. Imron, “Comparative Analysis of SVM and IndoBERT for Intent Classification in Indonesian Overtime Chatbots,” JSCE, vol. 6, no. 3, pp. 258–270, Aug. 2025, doi: 10.61628/jsce.v6i3.2058.
[11] D. Suhartono, M. R. N. Majiid, and R. Fredyan, “Towards automatic question generation using pre-trained model in academic field for Bahasa Indonesia,” Educ Inf Technol, vol. 29, no. 16, pp. 21295–21330, Nov. 2024, doi: 10.1007/s10639-024-12717-9.
[12] T. Zheng, S. Shen, and C. Zeng, “A Retrieval-Augmented Generation Method for Question Answering on Airworthiness Regulations,” Electronics, vol. 14, no. 16, p. 3314, Aug. 2025, doi: 10.3390/electronics14163314.
[13] E. Karakurt and A. Akbulut, “Retrieval-Augmented Generation (RAG) and Large Language Models (LLMs) for Enterprise Knowledge Management and Document Automation: A Systematic Literature Review,” Applied Sciences, vol. 16, no. 1, p. 368, Dec. 2025, doi: 10.3390/app16010368.
[14] K. Olawore, M. McTear, and Y. Bi, “Development and Evaluation of a University Chatbot Using Deep Learning: A RAG-Based Approach,” in Chatbots and Human-Centered AI, A. Følstad et al., Eds. Cham: Springer Nature Switzerland, 2025, pp. 96–111, doi: 10.1007/978-3-031-88045-2_7.
[15] L. Danuarta, V. C. Mawardi, and V. Lee, “Retrieval-Augmented Generation (RAG) Large Language Model For Educational Chatbot,” in 2024 Ninth International Conference on Informatics and Computing (ICIC), Medan, Indonesia, 2024, pp. 1–6, doi: 10.1109/ICIC64337.2024.10957676.
[16] N. S. Amarnath and R. Nagarajan, “An Intelligent Retrieval Augmented Generation Chatbot for Contextually-Aware Conversations to Guide High School Students,” in Proc. 2024 4th Int. Conf. Sustainable Expert Syst. (ICSES), Kaski, Nepal, 2024, pp. 1393–1398, doi: 10.1109/ICSES63445.2024.10762977.
[17] M. Reuter et al., “Towards Reliable Retrieval in RAG Systems for Large Legal Datasets,” in Proceedings of the Natural Legal Language Processing Workshop 2025, Suzhou, China: Association for Computational Linguistics, Nov. 2025, pp. 17–30, doi: 10.18653/v1/2025.nllp-1.3.
[18] Y. A. Prasetyo, E. Utami, and A. Yaqin, “Pengaruh Komposisi Split Data Terhadap Performa Akurasi Analisis Sentimen Algoritma Naïve Bayes dan SVM,” JEECOM, vol. 6, no. 2, pp. 382–390, Oct. 2024, doi: 10.33650/jeecom.v6i2.9188.
[19] C. A. Indranie, A. Ramadhan, J. Raharjo, R. R. N. Ikhsan, M. F. Rohadi, and D. I. Anandaputra, “Impact of Train-Test Splitting Ratios on Machine Learning Models for Daily Electricity Load Time Series Forecasting,” in 2025 2nd International Conference on Information System and Information Technology (ICISIT), Yogyakarta, Indonesia, 2025, pp. 1–6, doi: 10.1109/ICISIT66233.2025.11402981.
[20] R. Alruwaithi and S. AlHumoud, “SHIFA: SBERT-Based Healthcare Information Focused Arabic Question Answering,” IEEE Access, vol. 13, pp. 86344–86355, 2025, doi: 10.1109/ACCESS.2025.3570637.
[21] Z. Han, C. Gao, J. Liu, J. Zhang, and S. Q. Zhang, “Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey,” Sep. 16, 2024, arXiv: arXiv:2403.14608. doi: 10.48550/arXiv.2403.14608.
[22] W. Suwarningsih, R. A. Pramata, F. Y. Rahadika, and M. H. A. Purnomo, “RoBERTa: language modelling in building Indonesian question-answering systems,” TELKOMNIKA, vol. 20, no. 6, p. 1248, Dec. 2022, doi: 10.12928/telkomnika.v20i6.24248.
[23] S. Park, A. Kim, S. Lee, H. Lee, C. Kamyod, and C. G. Kim, “Design of REST API Client for Conversational Agent using Large Language Model with Open API System,” in 2024 IEEE/ACIS 22nd International Conference on Software Engineering Research, Management and Applications (SERA), Honolulu, HI, USA: IEEE, May 2024, pp. 55–58. doi: 10.1109/SERA61261.2024.10685639.
[24] Y. Song et al., “RestGPT: Connecting Large Language Models with Real-World RESTful APIs,” Aug. 27, 2023, arXiv: arXiv:2306.06624. doi: 10.48550/arXiv.2306.06624.
[25] A. Rahmatulloh, A. Ginanjar, I. Darmawan, N. Kurniati, and E. Haerani, "Chatbot for Diagnosis of Pregnancy Disorders using Artificial Intelligence Markup Language (AIML)," JOIV: International Journal on Informatics Visualization, vol. 7, no. 1, pp. 77–83, 2023, doi: 10.30630/joiv.7.1.1595.
