Validity and Reliability of Online Socio-Scientific Issue-Based Critical Reasoning Instrument
DOI:
https://doi.org/10.15294/2hnw4674Keywords:
critical reasoning; instrument; online; reliability; socio-scientific issue; validityAbstract
Information regarding the evaluation of reasoning skills related to online socio-scientific issues is still limited due to limited instruments. This research aims to determine the validity and reliability of an online socio-scientific issue-based critical reasoning instrument. Content validity was proven by the assessment method by five experts. Confirmatory factor analysis was carried out with the Lisrel 8.8 program to prove construct validity. Instrument testing was carried out on 145 grade 10 high school students. Convergent validity was determined through the Average Variance Extracted (AVE) calculation, and reliability was determined through Construct Reliability (CR). There are 12 items from four dimensions of critical reasoning based on online socio-scientific issues: (1) evaluating SSI information, (2) identifying SSI complexity, (3) checking the credibility of information sources, and (4) multiperspective. The results of the content validity test obtained the Aiken coefficient value in the valid category. Based on the analysis results, the GFI, RMSEA, and p-value of the instrument fit the model and have loading factors that meet the minimum criteria. In addition, the instrument meets convergent validity and internal consistency. Thus, it is concluded that the instrument is declared valid and reliable and can be used to measure critical reasoning based on online socio-scientific issues.
References
Aiken, L. R. (1980). Content validity and reliability of single items or questionnaires. Educational and Psychological Measurement, 40(4), 955–959.
Aiken, L. R. (1985). Three coefficients for analyzing the reliability and validity of ratings, Educational and Psychological Measurement. Journal Articles; Reports - Research; Numerical/Quantitative Data, 45(1), 131–142.
Akhmaddhian, S., Budiman, H., & Bhandari, R. (2023). The Strengthening Government Policies on Mineral and Coal Mining to Achieve Environmental Sustainability in Indonesia, Africa and Germany. Bestuur, 11(1), 95–120.
Allcott, H., Gentzkow, M., & Yu, C. (2019). Trends in the diffusion of misinformation on social media. Research and Politics, 6(2), 1–8.
Alvarez-Risco, A., Del-Aguila-Arcentales, S., Rosen, M. A., & Yáñez, J. A. (2022). Social cognitive theory to assess the intention to participate in the facebook metaverse by citizens in Peru during the COVID-19 pandemic. Journal of Open Innovation: Technology, Market, and Complexity, 8(3), 142.
Asori, M., Mpobi, R. K. J., Morgan, A. K., Apoanaba, T. A., Katey, D., Ampofo, S. T., ... & Appiah, D. O. (2023). Is illegal mining socio-politically entrenched? An opinion piece of the interaction between formal politics and chief dominance in mineral governance, and its influence on fighting Galamsey in Ghana. GeoJournal, 88(2), 1953-1963.
Azumah, F. D., Baah, E., & Nachinaab, J. O. (2020). Causes and effects of Illegal Gold Mining (Galamsey) activities on school dropout and residents at the Tutuka Central Circuit in Obuasi Municipality in Ashanti Region, Ghana. Journal of Education, 201(3), 162–173.
Baharuddin, M. F., Masrek, M. N., & Shuhidan, S. M. (2020). Content validity of assessment instrument for innovative work behaviour of Malaysian school teachers. International Journal of Scientific and Technology Research, 9(4), 1940–1946. https://www.ijstr.org/final-print/apr2020/Content-Validity-Of-Assessment-Instrument-For-Innovative-Work-Behaviour-Of-Malaysian-School-Teachers.pdf
Banerjee, M., Zlatkin-Troitschanskaia, O., & Roeper, J. (2020). Narratives and their impact on students’ information seeking and critical online reasoning in higher education economics and medicine. Frontiers in Education, 5(570625).
Bernard, M.-C., & Albert, M. (2018). Integrating socio-scientific issues in biology classrooms: The point of view of Québec’s college teachers. Recherches en Didactiques des Sciences et des Technologies, 18, 79–102.
Bizzotto, N., De Bruijn, G.-J., & Schulz, P. J. (2023). Buffering Against Exposure to Mental Health Misinformation in Online Communities: The Interplay of Depression Literacy and Expert Moderation. BMC Public Health, 23(1577), 1–12.
Brown, T. A. (2003). Confirmatory factor analysis of the Penn State Worry Questionnaire: Multiple factors or method effects? Behaviour Research and Therapy, 41(12), 1411–1426.
Cebesoy, U. B., & Rundgren, S. N. C. (2023). Embracing socio-scientific issues-based teaching and decision-making in teacher professional development. Educational Review, 75(3), 507–534.
Chen, H., Lewis, H., & Myhill, D. (2021). Fostering critical reasoning: Developing argumentative competence in early and middle primary years. Australian Journal of Language and Literacy, 44(2), 46–61.
Cho, H., Cannon, J., Lopez, R., & Li, W. (2022). Social media literacy: A conceptual framework. New Media and Society, 00(0).
Chu, J., Zhu, Y., & Ji, J. (2023). Characterizing the semantic features of climate change misinformation on Chinese social media. Public Understanding of Science, 0(0).
Cian, H. (2020). The influence of context: comparing high school students’ socio-scientific reasoning by socio-scientific topic. International Journal of Science Education, 42(9), 1503–1521.
Cohen, A. S., Lutzke, L., Otten, C. D., & Árvai, J. (2022). I Think, Therefore I Act: The Influence of Critical Reasoning Ability on Trust and Behavior During the COVID-19 Pandemic. Risk Analysis, 42(5), 1073–1085.
Dallo, I., Elroy, O., Fallou, L., Komendantova, N., & Yosipof, A. (2023). Dynamics and characteristics of misinformation related to earthquake predictions on Twitter. Scientific Reports, 13(1), 13391.
dos Santos, P. M., & Cirillo, M. Â. (2021). Construction of the average variance extracted index for construct validation in structural equation models with adaptive regressions. Communications in Statistics: Simulation and Computation, 0(0), 1–13.
Elastika, R. W., Sukono, & Dewanto, S. P. (2021). Analysis of factors affecting students’ mathematics learning difficulties using sem as information for teaching improvement. International Journal of Instruction, 14(4), 281–300.
Espin, J., & Perz, S. (2021). Environmental crimes in extractive activities: Explanations for low enforcement effectiveness in the case of illegal gold mining in Madre de Dios, Peru. Extractive Industries and Society, 8(1), 331–339.
Freiling, I., & Matthes, J. (2023). Correcting climate change misinformation on social media: Reciprocal relationships between correcting others, anger, and environmental activism. Computers in Human Behavior, 145(107769).
Garrecht, C., Reiss, M. J., & Harms, U. (2021). “I wouldn’t want to be the animal in use nor the patient in need’–the role of issue familiarity in students” socio-scientific argumentation. International Journal of Science Education, 43(12), 2065–2086.
Goretzko, D., Siemund, K., & Sterner, P. (2023). Evaluating model fit of measurement models in confirmatory factor analysis. Educational and Psychological Measurement, 00(0), 1–22.
Hair, J. F., Black, W. C., J.Babin, B., & Anderson, R. E. (2010). Multivariate Data Analysis (7th Edition). Pearson Prentice Hall.
Hashemi, M. (2023). Geographical visualization of tweets, misinformation, and extremism during the USA 2020 presidential election using LSTM, NLP, and GIS. Journal of Big Data, 10(1).
Herman, B. C., Clough, M. P., & Rao, A. (2022). Socio-scientific Issues Thinking and Action in the Midst of Science-in-the-Making. Science and Education, 31(5), 1105–1139.
Hox, J. J. (2021). Confirmatory Factor Analysis. In The Encyclopedia of Research Methods in Criminology and Criminal Justice: Volume II: Parts 5-8.
Hu, L. T., & Bentler, P. M. (1999). Cut-off criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55.
Keshavarz, H. (2021). Evaluating credibility of social media information: current challenges, research directions and practical criteria. Information Discovery and Delivery, 49(4), 269–279.
Khampirat, B. (2021). Validation of Motivated Strategies for Learning Questionnaire: Comparison of Three Competing Models. International Journal of Instruction, 14(2), 609–626.
Kim, S., Lin, T. J., Glassman, M., Ha, S. Y., Wen, Z., Nagpal, M., ... & Kraatz, E. (2024). Linking knowledge justification with peers to the learning of social perspective taking. Journal of Moral Education, 53(2), 321-341.
Lai, C., Toriumi, F., & Yoshida, M. (2023). A cross-lingual analysis on the spread of misinformation using the case of Ivermectin as a treatment for COVID-19. Scientific Reports, 13(14686), 1–15.
Laing, T., & Moonsammy, S. (2021). Evaluating the impact of small-scale mining on the achievement of the sustainable development goals in Guyana. Environmental Science and Policy, 116(November 2020), pp. 147–159.
List, A. (2023). Social justice reasoning when students learn about social issues using multiple texts. Discourse Processes, 60(4–5), 244–272.
Lombardi, A. R., Rifenbark, G. G., Rogers, H. J., Swaminathan, H., Taconet, A., Mazzotti, V. L., ... & Langdon, S. (2023). Establishing construct validity of a measure of adolescent perceptions of college and career readiness. Career Development and Transition for Exceptional Individuals, 46(1), 4-14.
Marttunen, M., Salminen, T., & Utriainen, J. (2021). Student evaluations of the credibility and argumentation of online sources. Journal of Educational Research, 114(3), 294–305.
Maulida, N. H., Mattiro, S., Nur, R., P, R., & Syaharuddin, S. (2022). Dampak sosial ekonomi penambang emas tanpa izin (Illegal) pada masyarakat Binawara. PAKIS (Publikasi Berkala Pendidikan Ilmu Sosial), 2(2), 54–65.
McGrew, S., Smith, M., Breakstone, J., Ortega, T., & Wineburg, S. (2019). Improving university students’ web savvy: An intervention study. British Journal of Educational Psychology, 89(3), 485–500.
Moreira, C. dos S. (2020). Contributions and limits of the socio-scientific issues Phosphoethanolamine in Science classes. Investigacoes Em Ensino de Ciencias, 25(3), 344–360.
Nagel, M. T., Zlatkin-Troitschanskaia, O., & Fischer, J. (2022). Validation of newly developed tasks for the assessment of generic Critical Online Reasoning (COR) of university students and graduates. Frontiers in Education, 7.
Namdar, B., & Namdar, A. O. (2022). Role Reversal to Facilitate Social & Moral Compassion: A Case for Climate Change as an Ethics Dilemma. American Biology Teacher, 84(4), 242–245.
Nunn, R., Brandt, C., & Deveci, T. (2016). Project-based learning as a holistic learning framework: Integrating 10 principles of critical reasoning and argumentation. Asian ESP Journal, 12(2), 9–53.
Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory. McGraw-Hill Inc: New York.
Pallant, J. (2020). Checking the reliability of a scale. SPSS Survival Manual, pp. 102–106.
Parker, W., Muelier, M., & Laura, W. (1989). Critical Reasoning on Civic Issues. Theory and Research in Social Education, XVII(1), 7–32.
Retnawati, H. (2016a). Analisis Kuantitatif Instrumen Penelitian (Panduan Peneliti, Mahasiswa, dan Psikometrian). Parama Publishing.
Retnawati, H. (2016b). Proving Content Validity of Self-Regulated Learning Scale (The Comparison of Aiken Index and Expanded Gregory Index). Research and Evaluation in Education, 2(2), 155–164. http://journal.uny.ac.id/index.php/reid
Ridwan, M. R., Hadi, S., Jailani, J., & Retnawati, H. (2023). The instrument development to measure the verbal ability of prospective high school students. International Journal of Evaluation and Research in Education, 12(1), 357–368.
Roldán, J. L., & Sánchez-Franco, M. J. (2012). Variance-based structural equation modeling: Guidelines for using partial least squares in information systems research. Research Methodologies, Innovations and Philosophies in Software Systems Engineering and Information Systems, May, 193–221.
Romine, W. L., Sadler, T. D., & Kinslow, A. T. (2017). Assessment of scientific literacy: Development and validation of the Quantitative Assessment of Socio-Scientific Reasoning (QuASSR). Journal of Research in Science Teaching, 54(2), 274–295.
Rosli, M. S. et al. (2021). Improving Questionnaire Reliability using Construct Reliability for Researches in Educational Technology. International Journal of Interactive Mobile Technologies, 15(4), 109–116.
Saye, J. W., & Brush, T. (2002). Scaffolding critical reasoning about history and social issues in multimedia-supported learning environments. Educational Technology Research and Development, 50(3), 77–96.
Subiantoro, A. W., & Treagust, D. F. (2021). Development and validation of an instrument for assessing high-school students’ perceptions of socio-scientific issues-based learning in biology. Learning Environments Research, 24(2), 223–237.
Tam, J., Porter, E. K., & Lee, U. J. (2022). Examination of Information and Misinformation about Urinary Tract Infections on TikTok and YouTube. Urology, 168, 35–40.
Valentini, F., & Damásio, B. F. (2016). Average Variance Extracted and Composite Reliability: Reliability Coefficients. Psicologia: Teoria e Pesquisa, 32(2), 1–7.
Weber, B. A., Derrico, D. J., Yoon, S. L., & Sherwill-Navarro, P. (2010). Educating patients to evaluate web-based health care information: The GATOR approach to healthy surfing. Journal of Clinical Nursing, 19(9–10), 1371–1377.
Wiblom, J., Rundgren, C. J., & Andrée, M. (2019). Developing Students’ Critical Reasoning About Online Health Information: A Capabilities Approach. Research in Science Education, 49(6), 1759–1782.
Xiao, S., & Sandoval, W. A. (2017). Associations Between Attitudes Towards Science and Children’s Evaluation of Information About Socio-scientific Issues. Science and Education, 26(3–4), 247–269.
