Hybrid IndoBERT and Support Vector Machine for Multi-class Emotion Classification of Indonesian Tourism Reviews
DOI:
https://doi.org/10.35194/mji.v18i1.6377Keywords:
Emotion Classification , IndoBERT, Support Vector Machine, SMOTE, Class imbalanceAbstract
Online reviews hold emotional nuances that binary sentiment analysis cannot adequately capture for targeted tourism management. Indonesian reviews pose additional computational challenges due to informal language, Sundanese vernacular, and severe class imbalance. Objective: This study develops a hybrid classification framework using IndoBERT as a frozen feature extractor and a Support Vector Machine (SVM) across five emotional classes. It investigates integrating Principal Component Analysis (PCA) and SMOTE within a strict cross-validation pipeline to mitigate extreme minority class scarcity while preventing data leakage. The duplicate-free dataset comprises 446 manually annotated reviews from agro-tourism destinations in Rancakalong. Annotations followed Ekman’s emotions plus a neutral category, cross-validated by a Large Language Model (Cohen's Kappa = 0.7475). To satisfy oversampling constraints, three extreme minority classes (fear, surprise, disgust) were consolidated into an 'OTHER' class. Three configurations were evaluated via 5-Fold Stratified Cross-Validation: TF-IDF + SVM (M1 baseline), IndoBERT + SVM (M2), and IndoBERT + PCA + SMOTE + SVM (M3), utilizing Macro F1 as the primary metric. Results: The M1 baseline yielded a Macro F1 of 0.3920. By capturing contextual semantics, M2 improved accuracy to 0.7131 and Macro F1 to 0.4133. The proposed M3 architecture achieved the highest Macro F1 (0.4321), demonstrating that combining dimensionality reduction and oversampling strengthens minority class decision boundaries. However, erratic performance on the synthetic 'OTHER' class confirms that merging distinct emotions disrupts cohesive semantic signatures. Integrating frozen IndoBERT embeddings with PCA and SMOTE within a cross-validated SVM architecture significantly outperforms traditional baseline models on highly imbalanced, low-resource Indonesian text data. This study contributes an empirically validated emotion corpus and establishes a foundational, data-driven behavioral modeling framework to guide targeted managerial interventions in local agro-tourism.
References
[1] Y. Mao, Q. Liu, and Y. Zhang, “Sentiment analysis methods, applications, and challenges: A systematic literature review,” Journal of King Saud University - Computer and Information Sciences, vol. 36, no. 4, p. 102048, Apr. 2024, doi: 10.1016/j.jksuci.2024.102048.
[2] P. Ekman, “An argument for basic emotions,” Cognition and Emotion, vol. 6, no. 3–4, pp. 169–200, 1992.
[3] R. Plutchik, “A general psychoevolutionary theory of emotion,” in Emotion: Theory, Research, and Experience, vol. 1, R. Plutchik and H. Kellerman, Eds., New York: Academic Press, 1980, pp. 3–33.
[4] Y. Kurniawati, R. B. Hamid, D. I. Sensuse, S. Lusa, P. A. W. Putro, and S. Indriasari, “Analysis of Public Sentiment Indonesia’s Personal Data Protection Law: A Comparison of SVM and IndoBERT on X Platform,” J. Tek. Inform. (JUTIF), vol. 7, no. 2, pp. 1007–1027, Apr. 2026, doi: 10.52436/1.jutif.2026.7.2.5415.
[5] A. Vaswani et al., “Attention Is All You Need,” 2017, arXiv. doi: 10.48550/ARXIV.1706.03762.
[6] B. Wilie et al., “IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding,” in Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing (AACL-IJCNLP), 2020, pp. 843–857.
[7] F. Koto, A. Rahimi, J. H. Lau, and T. Baldwin, “IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP,” in Proceedings of the 28th International Conference on Computational Linguistics (COLING), 2020, pp. 757–770.
[8] F. Koto, J. H. Lau, and T. Baldwin, “IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initialization,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021, pp. 10660–10668.
[9] A. R. Setyawan, L. H. Suadaa, and B. Yuniarto, “Aspect-Based Sentiment Analysis using Adaptive Aspect on Tourist Reviews in Jakarta,” SISTEMASI, vol. 13, no. 6, p. 2456, Nov. 2024, doi: 10.32520/stmsi.v13i6.4585.
[10] E. Junianto, M. Puspitasari, S. I. Zakaria, T. Arifin, and I. W. P. Agung, “Emotion Detection in Indonesian Text Using the Logistic Regression Method,” media j. inform., vol. 17, no. 2, pp. 305–316, Dec. 2025, doi: 10.35194/mji.v17i2.5927.
[11] E. C. Garrido-Merchan, R. Gozalo-Brizuela, and S. Gonzalez-Carvajal, “Comparing BERT Against Traditional Machine Learning Models in Text Classification,” JCCE, vol. 2, no. 4, pp. 352–356, Apr. 2023, doi: 10.47852/bonviewJCCE3202838.
[12] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: Synthetic Minority Over-sampling Technique,” Journal of Artificial Intelligence Research, vol. 16, pp. 321–357, 2002.
[13] A. A. Lestari, Ahmad Faqih, and Gifthera Dwilestari, “Improving Sentiment Analysis Performance of Tokopedia Reviews Using Principal Component Analysis and Naïve Bayes Algorithm,” j. of artif. intell. and eng. appl., vol. 4, no. 2, pp. 758–763, Feb. 2025, doi: 10.59934/jaiea.v4i2.743.
[14] S. Syah Putra and D. Riminarsih, “Evaluating Machine Learning Models Across Feature Extraction and Data Balancing Scenarios for Coretax Sentiment Analysis,” media j. inform., vol. 17, no. 2, pp. 379–396, Dec. 2025, doi: 10.35194/mji.v17i2.5968.
[15] F. Nurpandi, F. S. Sulaeman, and A. Hermawan, “Analisis Sentimen Terhadap Kinerja Kepolisian Indonesia Menggunakan Metode Multinomial Naive Bayes, Long Short-Term Memory, dan Lexicon-Based,” MJI, vol. 16, no. 1, p. 1, Jun. 2024, doi: 10.35194/mji.v16i1.4165.
[16] T. B. Brown et al., “Language Models are Few-Shot Learners,” in Advances in Neural Information Processing Systems (NeurIPS), 2020, pp. 1877–1901.
[17] L. Ouyang et al., “Training language models to follow instructions with human feedback,” in Advances in Neural Information Processing Systems (NeurIPS), 2022, pp. 27730–27744.
[18] F. Gilardi, M. Alizadeh, and M. Kubli, “ChatGPT outperforms crowd workers for text-annotation tasks,” Proceedings of the National Academy of Sciences, vol. 120, no. 30, p. e2305016120, 2023.
[19] T. Wolf et al., “Transformers: State-of-the-Art Natural Language Processing,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP), 2020, pp. 38–45.
[20] F. Pedregosa et al., “Scikit-learn: Machine Learning in Python,” Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011.
[21] G. Lemaître, F. Nogueira, and C. K. Aridas, “Imbalanced-learn: A Python Toolbox to Tackle the Curse of Imbalanced Datasets in Machine Learning,” Journal of Machine Learning Research, vol. 18, no. 17, pp. 1–5, 2017.
[22] J. R. Landis and G. G. Koch, “The measurement of observer agreement for categorical data,” Biometrics, vol. 33, no. 1, pp. 159–174, 1977.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Firas Atqiya, Afrida Helen, Muhammad Rizqi Sholahuddin
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
