A Hybrid Approach of Aspect-Based Sentiment Analysis and Knowledge Extraction for Evaluating Security Perceptions in Digital Payment Applications
DOI:
https://doi.org/10.15294/sji.v12i4.31557Keywords:
Aspect-Based Sentiment Analysis (ABSA), Security, Sentiment Classification, Machine Learning, Deep Learning, Knowledge ExtractionAbstract
Purpose: The rapid expansion of digital wallets in Indonesia has heightened concerns regarding user security and trust. This study evaluates user sentiment toward the security features of the DANA digital payment application using Aspect Sentiment Classification (ASC), a subtask of Aspect-Based Sentiment Analysis (ABSA). It aims to compare multiple classification models and generate structured, machine-readable sentiment outputs to support knowledge extraction and system integration.
Methods: A total of 4,846 security-related reviews were collected from the Google Play Store using keyword-based filtering, supplemented by 3,000 unfiltered reviews for robustness evaluation. Sentiment labeling was performed using a hybrid rule-based and manual annotation approach. From 300 proportionally sampled reviews (150 positive and 150 negative), the validation achieved 0.8504 accuracy and a Cohen’s κ of 0.951, indicating near-perfect agreement. Five models—Support Vector Machine (SVM), Random Forest (RF), Convolutional Neural Network (CNN), Bidirectional Long Short-Term Memory (BiLSTM), and IndoBERT—were evaluated using 5-fold stratified cross-validation with random oversampling to address class imbalance.
Results: IndoBERT achieved the highest performance with 98% accuracy, an F1-score of 0.974, and an AUC-ROC of 0.996, followed by CNN and BiLSTM. Robustness testing across temporal (DANA June–October) and cross-domain (GoPay) datasets confirmed IndoBERT’s strong generalization with minimal F1-score variation.
Novelty: Unlike previous ABSA studies that addressed multiple aspects, this research focuses exclusively on the security aspect, providing fine-grained insights into user trust. The integration of XML-based structured output enhances interpretability and interoperability in digital financial sentiment analysis, contributing to the development of more secure and transparent fintech ecosystems.
