An Ablation Study on Stacked LSTM and SMOTE for Government Policy Sentiment Classification

Abstract

Purpose: We compare four deep learning configurations of Standard LSTM without SMOTE, Standard LSTM with SMOTE, Stacked LSTM without SMOTE, and Stacked LSTM with SMOTE used for multi-class sentiment classification of Indonesian public comments on the government’s Free Nutritious Meals Program (MBG) policy posted on Instagram. Most existing research on this topic relies on single-layer LSTM models that fail to address class imbalance and the layered, often sarcastic semantic structure commonly found in informal Indonesian social media writing at the same time. We test whether a Stacked LSTM architecture combined with SMOTE can simultaneously model sequential word dependencies and high-level semantic patterns, while correcting the bias that class imbalance tends to introduce.

Methods: We collected 1,000 Instagram comments from the account @ferryirwandi using the Instaloader library and labeled them into three sentiment classes with TextBlob. Preprocessing was carried out through six steps: cleaning, case folding, slang normalization with a custom dictionary, stopword removal, tokenization, and stemming via Sastrawi after which TF-IDF was used for feature extraction. The data was split 80:20 into training and testing sets before applying any oversampling; SMOTE was applied only to the training data, so the test set remained intact and free of data leakage. Performance was measured using Accuracy, Macro-Precision, Macro-Recall, and Macro-F1 Score, obtained from a multiclass confusion matrix based on a One-vs-Rest scheme.

Result: Scenario 3 (Stacked LSTM without SMOTE) produced the top result, reaching 95.00% Accuracy, Macro-Precision of 0.9276, Macro-Recall of 0.8752, and Macro-F1 Score of 0.8973 the best outcome among the four configurations. In contrast, Scenario 4 (Stacked LSTM with SMOTE) performed worse, with accuracy dropping to 92.50% and the F1-Score falling to 0.8178. We attribute this decline to feature space overlap: the synthetic minority samples generated by SMOTE landed too close to the dominant neutral class within the high-dimensional, sparse TF-IDF feature space.

Novelty: Our findings provide empirical support for the idea that hierarchical representations built by stacked recurrent layers are well suited to capturing the complex context, sarcasm, and long-range semantic relationships found in Indonesian political discourse. We also propose the four-scenario ablation design as a reusable method for disentangling how much architectural depth and data balancing each contribute, both independently and jointly, when working with imbalanced Indonesian social media sentiment data.

Practical Implications: For practitioners working with TF-IDF features on narrow-topic datasets, our findings suggest that Stacked LSTM without oversampling is the more reliable configuration for multi-class sentiment analysis of Indonesian government-policy discourse on social media.