The Consent Paradox: Evaluating the Validity of Sensitive Data Collection by AI Chatbots

Marianne Charleen Sabini; Fajar Sugianto; Kentarou  Kaneda; Vincensia  EP Sari; Jerry  Shalmont

doi:10.15294/ijals.v8i1.37509

Authors

Marianne Charleen Sabini Faculty of Law, Universitas Pelita Harapan Author https://orcid.org/0009-0009-4553-9580
Fajar Sugianto Faculty of Law, Universitas Pelita Harapan Author https://orcid.org/0000-0002-5969-4495
Kentarou Kaneda Ginzaseiwa Law Office, Tokyo Author https://orcid.org/0009-0005-1684-2899
Vincensia EP Sari Faculty of Law, Universitas Pelita Harapan Author https://orcid.org/0009-0007-1713-8195
Jerry Shalmont Faculty of Law, Universitas Pelita Harapan Author https://orcid.org/0009-0003-0176-7529

DOI:

https://doi.org/10.15294/ijals.v8i1.37509

Keywords:

Sensitive Data Collection, AI Chatbots, Automated Processing, Personal Data Protection

Abstract

This paper explores the legal friction between user consent and data collection practices, investigating whether the current Indonesian legal landscape provides sufficient safeguards for AI chatbot users. It employs a comparative legal analysis to evaluate the regulatory alignment between the Indonesian PDP Law and the EU GDPR, specifically regarding the protection of sensitive data. The analysis reveals that while the two regulations share foundational data protection principles, the PDP Law lacks necessary specificity regarding the definition and processing of sensitive data, risk assessment mechanisms, and the authority of supervisory bodies. The first analysis elucidates how these regulatory gaps hinder the PDP Law’s ability to effectively govern AI-based environments and establish accountability for data controllers and processors. Subsequently, the second analysis proposes necessary reforms, including more robust governance mechanisms, strengthened administrative oversight, and industry-specific guidelines to ensure uniform enforcement. This paper concludes that the PDP Law requires substantive refinement to remain effective and responsive to the rapid evolution of AI and automated systems.

References

Ademokun, Oluwatosin Victoria. “AI And the GDPR: Understanding the Foundations of Compliance.” TechGDPR, 2025. techgdpr.com/blog/ai-and-the-gdpr-understanding-the-foundations-of-compliance.

Aisy, Firsta Rahadatul, Muhammad Azil Maskur, and A.M. Adzkiya' Amiruddin. "Establishing Indonesia’s Personal Data Protection Agency: Comparative Administration Sanctions Enforcement from Ireland, Australia, and Singapore." Journal of Indonesian Legal Studies 10, no. 1 (2025): 431–482. https://doi.org/10.15294/jils.v10i2.10025.

Aminy, Yudhi PrasetyoReza. “Introduction of the Official Personal Data Protection Act (UU PDP).” BDO, 2024. https://www.bdo.co.id/en-gb/insights/introduction-of-the-official-personal-data-protection-act-(uu-pdp).

Andreotta, Adam John, and Björn Lundgren. “Automated Informed Consent.” Big Data & Society 11, no. 4 (October 18, 2024): 20539517241289440. https://doi.org/10.1177/20539517241289439.

Asiean Breafing. “Indonesia’s Comprehensive Personal Data Protection Law Guide - Indonesia Guide.” Doing Business in Indonesia, 2025. https://www.aseanbriefing.com/doing-business-guide/indonesia/company-establishment/personal-data-protection-law.

Bagdasarian, Eugene, Ren Yi, Sahra Ghalebikesabi, Peter Kairouz, Marco Gruteser, Sewoong Oh, Borja Balle, and Daniel Ramage. “AirGapAgent: Protecting Privacy-Conscious Conversational Agents.” In Proceedings of the ACM Conference on Computer and Communications Security (CCS ’24), December 9–13, 2024. DOI:10.1145/3658644.3690350.

Benseghir, Mourad, Aouatef Zerara, Maamar Bentria, Halima Bendriss, and Mohamad Hidayat Muhtar. "Legal Aspects of Patient Data Governance in Digital Health: A Comparative Analytical Study of UAE and Indonesian Legislation." Journal of Indonesian Legal Studies 10, no. 2 (2025): 773–808. https://doi.org/10.15294/jils.v10i2.10025

Burmagina, Kseniia. “ChatGPT Usage: Statistics and Facts.” Elfsight, 2025. https://elfsight.com/blog/chatgpt-usage-statistics/.

Carmichael, Laurie, Sara-Maude Poirier, Constantinos K. Coursaris, Pierre-Majorique Léger, and Sylvain Sénécal. “Users’ Information Disclosure Behaviors during Interactions with Chatbots: The Effect of Information Disclosure Nudges.” Applied Sciences 12, no. 24 (2022): 12660. https://doi.org/10.3390/app122412660.

Chin-Rothmann, Caitlin. “Protecting Data Privacy as a Baseline for Responsible AI.” Protecting Data Privacy as a Baseline for Responsible AI, 2024. https://www.csis.org/analysis/protecting-data-privacy-baseline-responsible-ai.

Derico, Ben. “ChatGPT Bug Leaked Users’ Conversation Histories.” BBC News, 2023. https://www.bbc.com/news/technology-65047304.

European Commission. “Data Protection Explained.” European Commission, 2025. https://commission.europa.eu/law/law-topic/data-protection/data-protection-explained_en.

Exabeam. “The Intersection of GDPR and AI and 6 Compliance Best Practices.” Exabeam, 2025. www.exabeam.com/explainers/gdpr-compliance/the-intersection-of-gdpr-and-ai-and-6-compliance-best-practices/.

General Data Protection Regulation (GDPR). “Consent - General Data Protection Regulation (GDPR),” 2024. https://gdpr-info.eu/issues/consent/.

GROUP-IB. “Group-IB Discovers 100K+ Compromised ChatGPT Accounts on Dark Web Marketplaces; Asia-Pacific Region Tops the List.” GROUP-IB, 2023. www.group-ib.com/media-center/press-releases/stealers-chatgpt-credentials.

Gumusel, Ece. “A Literature Review of User Privacy Concerns in Conversational Chatbots: A Social Informatics Approach.” Journal of the Association for Information Science and Technology 76, no. 1 (2024): 121–54. https://doi.org/10.1002/asi.24898.

Hasal, Martin, Jana Nowaková, Khalifa Ahmed Saghair, Hussam Abdulla, Václav Snášel, and Lidia Ogiela. “Chatbots: Security, Privacy, Data Protection, and Social Aspects.” Concurrency and Computation: Practice and Experience 33, no. 19 (October 10, 2021): e6426. https://doi.org/10.1002/cpe.6426.

ICO. “A Guide to the Data Protection Principles.” ICO, 2025. https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/data-protection-principles/a-guide-to-the-data-protection-principles/.

ICO. “What Is Valid Consent?,” 2025. https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/lawful-basis/consent/what-is-valid-consent/.

Ive, Julia, Vishal Yadav, Mariia Ignashina, Matthew Rand, and Paulina Bondaronek. “Privacy-Preserving Behaviour of Chatbot Users: Steering Through Trust Dynamics.” arXiv preprint, 26 Nov 2024. arXiv:2411.17589.

John, Beauden. “User Perceptions of AI Chatbots: Analyzing Fear, Skepticism, and Acceptance.” ResearchGate, 2025. https://www.researchgate.net/publication/389283323_User_Perceptions_of_AI_Chatbots_Analyzing_Fear_Skepticism_and_Acceptance.

Kamarinou, Dimitra, Christopher Millard, Jatinder Singh, and R Leenes. “Machine Learning with Personal Data.” In Data Protection and Privacy: The Age of Intelligent Machines, 89–114. Hart Publishing Oxford, 2017. https://dokumen.pub/data-protection-and-privacy-the-age-of-intelligent-machines-9781509919345-9781509919376-9781509919352.html.

Koops, Bert-Jaap, and Ronald Leenes. “Code and the Slow Erosion of Privacy.” Michigan Telecommunications and Technology Law Review 12, no. 1 (2019): 115. https://repository.law.umich.edu/cgi/viewcontent.cgi?article=1114&context=mttlr.

Leschanowsky, Anna, Silas Rech, Birgit Popp, and Tom Bäckström. “Evaluating Privacy, Security, and Trust Perceptions in Conversational AI: A Systematic Review.” Computers in Human Behavior 159 (2024): 108344. https://doi.org/10.1016/j.chb.2024.108344.

LII (Legal Information Institute). “Right to Privacy.” LII, 2025.

Meyer, Linda. “ChatGPT and the Developer’s Ethical Responsibility: A Literature Study of Chatbot-Related Ethical Dilemmas from the Developer’s Perspective.” Thesis, Sweden: Linnaeus University, Faculty of Technology, Department of computer science and media technology (CM), 2023.

Neufeld-Wall Maya, and Trinity University. “Being Real: Gen-Z, Self-Presentation, and Authenticity on Social Media.” Digital Commons @ Trinity, 2023. digitalcommons.trinity.edu/cgi/viewcontent.cgi?article=1026&context=comm_honors.

NewsGP. “Increase in Patients Turning to AI for Therapy.” NewsGP, 2025. www1.racgp.org.au/newsgp/professional/increase-in-patients-turning-to-ai-for-therapy.

Ok, Emmanuel. “Transparency and Explainability in AI Chatbots: How They Shape User Confidence.” Thesis, Ogbomoso: Ladoke Akintola University of Technology, Department of Computer Science and Engineering, 2025.

Pujianti, Sri. “Govt: Law on Personal Data Protection Provides Legal Protection.” Constitutional Court of the Republic of Indonesia, 2023. https://en.mkri.id/news/details/2023-02-13/Govt:_Law_on_Personal_Data_Protection_Provides_Legal_Protection.

Regulation (EU). Regulation (EU) 2016/679 (General Data Protection Regulation) (2016).

Republic of Indonesia. Law of the Republic of Indonesia Number 27 of 2022 on Personal Data Protection (2022).

Roivainen, Mika. “Automated Data Processing: Definition and Examples,” 2025. https://www.esystems.fi/en/blog/automated-data-processing-definition-and-examples.

Ruane, Elayne, Abeba Birhane, and Anthony Ventresque. “Conversational AI: Social and Ethical Considerations.” The Irish Software Research Centre 2563 (2019): 104–15. https://ceur-ws.org/Vol-2563/aics_12.pdf.

Sebastian, Glorin. “Privacy and Data Protection in Chatgpt and Other Ai Chatbots: Strategies for Securing User Information.” International Journal of Security and Privacy in Pervasive Computing (IJSPPC) 15, no. 1 (2023): 1–14. https://doi.org/10.4018/IJSPPC.325475.

Secure Privacy Team. “Adaptive Consent Frequency: Using AI to Combat Consent Fatigue,” 2025. https://secureprivacy.ai/.

Simbolon, Valentina Ancillia, and Vishnu Juwono. “Comparative Review of Personal Data Protection Policy in Indonesia and the European Union General Data Protection Regulation.” Publik (Jurnal Ilmu Administrasi) 11, no. 2 (2022): 178–90. https://doi.org/10.31314/pjia.11.2.178-190.2022.

SISA MKTG. “A Guide to Data Classification Types and Levels.” SISA, 2020. https://www.sisainfosec.com/blogs/a-guide-to-data-classification-types-and-levels/.

Solo, Alex. “GDPR ‘Implied’ Consent – Why Guesswork Spells Trouble.” Sprintlaw, 2025. https://sprintlaw.co.uk/articles/gdpr-implied-consent-why-guesswork-spells-trouble/.

Sugianto, Fajar. OECD Guidelines on the Protection of Privacy and Transborder Flows of Personal Data; An Overview. Slide Show, 2025.

Susento, Jovan Rafael Aurelio. “Perbandingan Hukum Peraturan Pelindungan Data Pribadi Indonesia Dengan Uni-Eropa Mengenai Kebocoran Data Pribadi Lingkup Instansi Pemerintahan.” Thesis, Jakarta: Program Studi Hukum Fakultas Hukum Universitas Pelita Harapan, 2024.

Susser, Daniel, Beate Roessler, and Helen Nissenbaum. “Online Manipulation: Hidden Influences in a Digital World.” Georgetown Law Technology Review 4 (2019): 1. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3306006.

Taufiq, Muchamad, and Ananda Salsabila Kenyo. “The Legal Protection of Personal Data in the Digital Era: A Comparative Study of Indonesian Law and the GDPR.” International Journal of Business, Law, and Education 6, no. 2 (2025): 1260–68. http://ijble.com/index.php/journal/article/view/1178.

TermsFeed. “Personal Vs. Sensitive Personal Information.” Termly, 2025. https://termly.io/resources/articles/sensitive-personal-information/.

United Nations - CEB. “Principles on Personal Data Protection and Privacy.” United Nations - CEB, 2025. https://unsceb.org/principles-personal-data-protection-and-privacy-listing.

Verma, Karishma. "Digital Deception: The Impact of Deepfakes on Privacy Rights." Lex Scientia Law Review 8, no. 2 (December 2024): 859–896. https://doi.org/10.15294/lslr.v8i2.13749.

Witcher, Rob. “OECD Privacy Guidelines & Security | CISSP Study Guide.” Destination Certification, 2025. https://destcert.com/resources/oecd-privacy-principles-global-security/.

Yang, Jing, Yen-Lin Chen, Lip Yee Por, and Chin Soon Ku. “A Systematic Literature Review of Information Security in Chatbots.” Applied Sciences 13, no. 11 (2023): 6355. https://doi.org/10.3390/app13116355.