A Digital System-Based Assessment for Collaborative Problem-Solving Skills among Pre-service Science Teachers: Preliminary Evidence of Quality and Implementation
DOI:
https://doi.org/10.15294/jpii.v15i3.54201Keywords:
collaborative problem-solving, digital assessment, quality science educationAbstract
Collaborative problem-solving (CPS) is central to science teacher preparation, yet existing assessments often emphasize individual outcomes or final products and provide limited evidence of how CPS develops across iterative stages of group problem-solving. This study developed and preliminarily evaluated a digital assessment system for science teachers’ CPS across four stages: identifying and representing problems, planning and developing solutions, testing and refining solutions, and communicating evidence-based results. A multiphase, quantitative-dominant mixed-methods design involved three expert validators and 118 prospective science teachers from four intact classes. Quantitative evidence included content relevance (Aiken’s V; expanded Gregory), internal structure (CFA), expert-judgment consistency (ICC), and multiclass score patterns; expert comments, revision records, and user feedback provided qualitative evidence integrated through a joint display. Content relevance was mixed, with CFA showing three dimensions loading above 0.30 and communicating results loading at 0.286. Expert-judgment consistency was inadequate, and student-score reliability was not established. Class scores differed significantly; however, intact-class observations preclude causal inference. Thus, the system provides preliminary stage-specific CPS score patterns rather than dependable individual-level profiles. Its contribution is integrating stage-specific assessment across an authentic design–test–redesign–retest cycle using analytic-rubric scoring with measurement and implementation evidence. Further refinement should prioritize communication descriptors, validator calibration, and student-score reliability.
References
Adom, D., Adu-Mensah, J., & Dake, D. A. (2020). Test, measurement, and evaluation: Understanding and use of the concepts in education. International Journal of Evaluation and Research in Education (IJERE), 9(1), 109–119.
Aiken, L. R. (1985). Three Coefficients for Analyzing the Reliability and Validity of Rating. Educational and Psychological Measurement, 45(1), 131–142.
Alenezi, M., Wardat, S., & Akour, M. (2023). The Need of Integrating Digital Education in Higher Education: Challenges and Opportunities. Sustainability, 15(6), 4782.
Alfin, J., Fuad, A. Z., Nur, M., Yuanita, L., & Prahani, B. K. (2019). Development of Group Science Learning (GSL) Model to Improve the Skills of Collaborative Problem Solving, Science Process, and Self-Confidence of Primary Schools Teacher Candidates. International Journal of Instruction, 12(1), 147–164.
Almanasreh, E., Moles, R. J., & Chen, T. F. (2022). A practical approach to the assessment and quantification of content validity. In Contemporary Research Methods in Pharmacy and Health Services (pp. 583–599). Elsevier.
Anchunda, H. Y., & Kaewurai, W. (2025). An instructional model development based on inquiry-based and problem-based approaches to enhance prospective teachers’ teamwork and collaborative problem-solving competence. Social Sciences & Humanities Open, 11, 101480.
Andrews-Todd, J., Jiang, Y., Steinberg, J., Pugh, S. L., & D’Mello, S. K. (2023). Investigating collaborative problem solving skills and outcomes across computer-based tasks. Computers & Education, 207, 104928.
Baucal, A., Jošić, S., Stepanović Ilić, I., Videnović, M., Ivanović, J., & Krstić, K. (2023). What makes peer collaborative problem solving productive or unproductive: A qualitative systematic review. Educational Research Review, 41, 100567.
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101.
Brod, M., Pohlman, B., & Waldman, L. T. (2023). Qualitative Research and Content Validity. In Encyclopedia of Quality of Life and Well-Being Research (pp. 5658–5666). Springer International Publishing.
Brown, T. A. (2015). Confirmatory factor analysis for applied research (2nd ed.). Guilford publications.
Brown, T. A., & Moore, M. T. (2012). Confirmatory factor analysis. In R. H. Hoyle (Ed.), Handbook of structural equation modeling (pp. 361–379). Guilford Press.
Care, E., Scoular, C., & Griffin, P. (2016). Assessment of Collaborative Problem Solving in Education Environments. Applied Measurement in Education, 29(4), 250–264.
Carrillo-Avalos, B. A., Leenen, I., Trejo-Mejía, J. A., & Sánchez-Mendiola, M. (2025). Bridging Validity Frameworks in Assessment: Beyond Traditional Approaches in Health Professions Education. Teaching and Learning in Medicine, 37(2), 229–238.
Chai, H., Hu, T., & Wu, L. (2024). Computer-based assessment of collaborative problem solving skills: A systematic review of empirical research. Educational Research Review, 43, 100591.
Chen, L., Inoue, K., Goda, Y., Okubo, F., Taniguchi, Y., Oi, M., Konomi, S., Ogata, H., & Yamada, M. (2020). Exploring Factors that Influence Collaborative Problem Solving Awareness in Science Education. Technology, Knowledge and Learning, 25(2), 337–366.
Collier, J. E. (2026). Applied Structural Equation Modeling Using AMOS: Basic to Advanced Techniques (2nd Ed). Routledge.
Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). SAGE Publications.
Fauziah, A. N. M., Wasis, Hendratmoko, A. F., Mahdiannur, M. A., Ermawan, M. Z. F., Suwandi, E., & Ratri, S. Y. (2024). Relationship Between Critical Thinking and Scientific Argumentation in Science Learning. Jurnal Pendidikan IPA Indonesia, 13(2), 239–254.
Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013). Achieving Integration in Mixed Methods Designs—Principles and Practices. Health Services Research, 48(6pt2), 2134–2156.
Fetters, M. D., & Tajima, C. (2022). Joint Displays of Integrated Data Collection in Mixed Methods Research. International Journal of Qualitative Methods, 21.
Field, A. (2017). Discovering statistics using IBM SPSS statistics (5th ed.). SAGE.
Gao, Q., Zhang, S., Cai, Z., Liu, K., Hui, N., & Tong, M. (2022). Understanding student teachers’ collaborative problem solving competency: Insights from process data and multidimensional item response theory. Thinking Skills and Creativity, 45, 101097.
García-Carmona, A. (2020). From Inquiry-Based Science Education to the Approach Based on Scientific Practices. Science & Education, 29(2), 443–463.
Gillies, R. M. (2023). Teaching Science That Is Inquiry-Based: Practices and Principles. In Challenges in Science Education (pp. 39–58). Springer International Publishing.
Goretzko, D., Siemund, K., & Sterner, P. (2024). Evaluating Model Fit of Measurement Models in Confirmatory Factor Analysis. Educational and Psychological Measurement, 84(1), 123–144.
Hao, J., Liu, L., Kyllonen, P., Flor, M., & von Davier, A. A. (2019). Psychometric Considerations and a General Scoring Strategy for Assessments of Collaborative Problem Solving. ETS Research Report Series, 2019(1), 1–17.
Hariyono, E., Rosdiana, L., Budiyanto, M., Satriawan, M., Nabilah, H., & Rizki, I. A. (2026). Factors Determining Socio-Scientific Issues in STEAM Education to Enhance Problem-Solving Skills for Pre-Service Teachers: Development and Validation of a Measurement Model. Jurnal Pendidikan IPA Indonesia, 15(1), 14–28.
Haryani, E., Coben, W. W., Pleasants, B. A.-S., & Fetters, M. K. (2021). Analysis of Teachers’ Resources for Integrating the Skills of Creativity and Innovation, Critical Thinking and Problem Solving, Collaboration, and Communication in Science Classrooms. Jurnal Pendidikan IPA Indonesia, 10(1), 92–102.
Hendratmoko, A. F., Fauziah, A. N. M., Susiyawati, E., Astriani, D., Nurita, T., Martini, Erman, Septaria, K., Fi’aunillah, T. K., & Boey, A. (2026). Integrating ChatGPT-driven self-assessment and Debate Based on Inquiry Learning (DBOIL) to enhance students’ argumentation skills: Leveraging AI for quality science learning aligned with SDG 4. E3S Web of Conferences, 696, 02005.
Hendratmoko, A. F., Madlazim, M., Widodo, W., & Astutik, S. (2024). Debate Based on Inquiry Learning (DBOIL): An Innovative Learning Model to Improve Students’ Skills in Scientific Argumentation. TEM Journal, 13(4), 3295–3303.
Hidayat, R., Habibi, A., Mohd Saad, M. R., Mukminin, A., & Wan Idris, W. I. bin. (2019). Exploratory and confirmatory factor analysis of PERMA for Indonesian students in mathematics education programmes. Pedagogika, 132(4), 147–165.
Kaspi, S., & Venkatraman, S. (2023). Data-Driven Decision-Making (DDDM) for Higher Education Assessments: A Case Study. Systems, 11(6), 306.
Kline, R. B. (2023). Principles and practice of structural equation modeling. Guilford publications.
Koo, T. K., & Li, M. Y. (2016). A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. Journal of Chiropractic Medicine, 15(2), 155–163.
Lin, L., Zhou, D., Wang, J., & Wang, Y. (2024). A Systematic Review of Big Data Driven Education Evaluation. Sage Open, 14(2).
MacCallum, R. C., & Austin, J. T. (2000). Applications of Structural Equation Modeling in Psychological Research. Annual Review of Psychology, 51(1), 201–226. U
Magaji, A. (2021). Promoting Problem-Solving Skills among Secondary Science Students through Problem Based Learning. International Journal of Instruction, 14(4), 549–566.
Maier, U., & Klotz, C. (2022). Personalized feedback in digital learning environments: Classification framework and literature review. Computers and Education: Artificial Intelligence, 3, 100080.
Marsh, H. W., Morin, A. J. S., Parker, P. D., & Kaur, G. (2014). Exploratory Structural Equation Modeling: An Integration of the Best Features of Exploratory and Confirmatory Factor Analysis. Annual Review of Clinical Psychology, 10(1), 85–110.
McNeish, D., & Wolf, M. G. (2022). Dynamic fit index cutoffs for one-factor models. Behavior Research Methods, 55(3), 1157–1174.
Meijer, H., Hoekstra, R., Brouwer, J., & Strijbos, J.-W. (2020). Unfolding collaborative learning assessment literacy: a reflection on current assessment methods in higher education. Assessment & Evaluation in Higher Education, 45(8), 1222–1240.
Mutmainna, M., Hasyim, M., Syamsuriwal, S., Setiaji, B., & Setiawati, F. A. (2024). Modeling of Colorado Learning Attitude Science Survey in Indonesian Version: A Study with Applying Item Response Theory. JP3I (Jurnal Pengukuran Psikologi Dan Pendidikan Indonesia), 13(1), 80–104.
Negrón, J., López‐Cruz, M., Felmer, P., & Sepúlveda, C. L. (2026). Collaborative problem solving in science classrooms: Tools for developing 21st century skills. School Science and Mathematics, 126(3), 235–250.
Nugraha, M. F. (2022). The Development and Validation of Jihad Academic Scale (JAS). JP3I (Jurnal Pengukuran Psikologi Dan Pendidikan Indonesia), 11(2), 182–195.
Nurita, T., Fauziah, A. N. M., Mahdiannur, M. A., Susilowati, R., & Hendratmoko, A. F. (2025). Fostering Creative Thinking Skills in Creating Domain of Anderson’s Taxonomy for Prospective Science Teachers Using E-Modul. TEM Journal, 14(3), 2339–2348.
OECD. (2017). PISA 2015 Results (Volume V): Collaborative Problem Solving. OECD Publishing.
Ouyang, F., Dinh, T. A., & Xu, W. (2023). A Systematic Review of AI-Driven Educational Assessment in STEM Education. Journal for STEM Education Research, 6(3), 408–426.
Pergo, A. T. C., França, A. B., Teixeira, P. H. M., Siqueira, L. B., Cortez, P. J. O., & Vitorino, L. M. (2023). Development and validation of a vaccine hesitancy scale for COVID-19 to the Portuguese language. Research, Society and Development, 12(3), e4012340357.
Retnawati, H. (2016). Proving content validity of self-regulated learning scale (The comparison of Aiken index and expanded Gregory index). REID (Research and Evaluation in Education), 2(2), 155–164.
Richey, R. C., & Klein, J. D. (2007). Design and development research: Methods, strategies, and issues. Lawrence Erlbaum Associates.
Rosen, Y., Wolf, I., & Stoeffler, K. (2020). Fostering collaborative problem solving skills in science: The Animalia project. Computers in Human Behavior, 104, 105922.
Sajidan, S., Suranto, S., Atmojo, I. R. W., Saputri, D. Y., & Etviana, R. (2022). Problem-Based Learning-Collaboration (PBL-C) Model in Elementary School Science Learning in the Industrial Revolution Era 4.0 and Indonesia Society 5.0. Jurnal Pendidikan IPA Indonesia, 11(3), 477–488.
Sari, C. M., Ridwan, A. R., & Sastrawijaya, Y. S. (2024). Learning Environment of Numeracy: Development and Validation With Confirmatory Factor Analysis and Rasch Model. Pedagogika, 153(1), 185–198.
Scoular, C., & Care, E. (2019). A Generalized Scoring Process to Measure Collaborative Problem Solving in Online Environments. Educational Assessment, 24(3), 213–234.
Sincovich, A., Gregory, T., Zanon, C., Santos, D. D., Lynch, J., & Brinkman, S. A. (2019). Measuring early childhood development in multiple contexts: the internal factor structure and reliability of the early Human Capability Index in seven low and middle income countries. BMC Pediatrics, 19(1), 471.
Siswanto, J., Khoiri, N., & Saphira, H. V. (2025). The Effectiveness of the Algodoo-Assisted IBMRO Model in Improving Physics Problem-Solving Skills. Jurnal Pendidikan IPA Indonesia, 14(3), 496–508.
Sun, C., Shute, V. J., Stewart, A. E. B., Beck-White, Q., Reinhardt, C. R., Zhou, G., Duran, N., & D’Mello, S. K. (2022). The relationship between collaborative problem solving behaviors and solution outcomes in a game-based learning environment. Computers in Human Behavior, 128, 107120.
Sun, C., Shute, V. J., Stewart, A., Yonehiro, J., Duran, N., & D’Mello, S. (2020). Towards a generalized competency model of collaborative problem solving. Computers & Education, 143, 103672.
Tang, T., Vezzani, V., & Eriksson, V. (2020). Developing critical thinking, collective creativity skills and problem solving through playful design jams. Thinking Skills and Creativity, 37, 100696.
Thornhill-Miller, B., Camarda, A., Mercier, M., Burkhardt, J.-M., Morisseau, T., Bourgeois-Bougrine, S., Vinchon, F., El Hayek, S., Augereau-Landais, M., Mourey, F., Feybesse, C., Sundquist, D., & Lubart, T. (2023). Creativity, Critical Thinking, Communication, and Collaboration: Assessment, Certification, and Promotion of 21st Century Skills for the Future of Work and Education. Journal of Intelligence, 11(3), 54.
Tomar, S., Arundhathi, A., Gupta, S., & Sharma, M. (2024). Digital assessment: impact on student motivation, peer learning, group dynamics. Journal of Education and Learning (EduLearn), 18(1), 9–17.
von Davier, A. A., Hao, J., Liu, L., & Kyllonen, P. (2017). Interdisciplinary research agenda in support of assessment of collaborative problem solving: lessons learned from developing a Collaborative Science Assessment Prototype. Computers in Human Behavior, 76, 631–640.
Widodo, W., Rosdiana, L., Sari, D. A. P., Sari, D. P., Purnomo, A. R., Roqobih, F. D., & Hendratmoko, A. F. (2026). Activity-Based-Lesson Learn-Reflection (ALLR) Assisted by HTML5 Package (H5P) to Develop Critical Thinking Skills of Future Science Teachers. TEM Journal, 15(2), 1826–1841.
Wilson, M. (2023). Constructing Measures: An Item Response Modeling Approach (2nd ed.). Routledge.
Zhang, D., & Hwang, G.-J. (2023). Effects of Interaction between Peer Assessment and Problem-Solving Tendencies on Students’ Learning Achievements and Collaboration in Mobile Technology-Supported Project-Based Learning. Journal of Educational Computing Research, 61(1), 208–234.
