VALIDATING THE S-STEM AMONG MALAYSIAN PRE-UNIVERSITY STUDENTS ABSTRACT The 2purpose of this present study is to validate the2measure of student attitudes toward science, technology, engineering, and mathematics (S -STEM). Using the cross-sectional design, this study employed translation and cultural adaptation as well as providing evidence of reliability and validity of the S-STEM. The instrument was administered to 748 pre-university students in Penang, Malaysia. Data was analysed using 13confirmatory factor analysis (CFA) with AMOS 19.0. Results support S-STEM as there-factor multidimensional construct, namely attitude towards science, attitude towards technology/engineering, and attitude towards mathematics. S-STEM also has the potential to facilitate the choice of courses to enrol at university. All statistics such as factor loadings, average variance explained, construct reliability, evidence of discriminant validity and goodness-of-fit indices were found to be at acceptable values. These positive results are important because although the instrument has undergone numerous modifications, such as translation, etc., the generalizability of the instrument is still preserved in pre-university Malaysian students. Counsellors may administer the instrument to provide additional information for the students. As for researchers, the instrument may be utilized to gather data in providing measures to improve students’ participation in learning STEM. The practical implications as well as the methodological limitations of the present study are discussed. Keywords: attitude towards STEM; confirmatory factor analysis; pre-university students; S-STEM INTRODUCTION It is widely known and generally accepted that mathematics has been considered as a valuable subject that is useful in daily life that helps us to perform effectively in society. To illustrate, mathematics allows us to know that we are not being overcharged in day-to-day transactions and that the cake we want to bake has the right proportions of flour, sugar, and water. More than that, mathematics is also closely related to other subjects, especially science. Logical thinking and deductive reasoning are the two skills best nurtured in mathematics that are useful in science. Mathematics and science are irrefutably the most recognized fields of study and have been the driving force in education for a country to thrive. Nevertheless, with the change of the world economy, knowledge and competencies in mathematics, as well as science, are no longer adequate to meet the challenges of the 21st century. Today’s technology and engineering encompass all aspects of life. For example, when doctors need advanced equipment to diagnose or treat patients, computer experts are needed to produce software that can help to accomplish the task. This is how interdisciplinary approaches help to address 21st-century problems. This is also the main reason that the concept of Science, Technology, Engineering, and Mathematics (STEM) education was introduced, in which engineering and technology were included together with mathematics and science as essential subjects at schools as well as tertiary institutions. STEM was firstly used by the National Science Foundation (NSF) in the United States in 2001 that includes various subjects such as science, mathematics, engineering, and computer science, as well as areas in social sciences like psychology, economics, sociology and political science. Subsequently, the definition continues to change. For example, Moore et al. (2014) states that STEM elements exist when there is a combination of any two fields in STEM, while according to Breiner et al. (2012), STEM is used to refer to science, technology, engineering, and mathematics interchangeably. However, at present, there seems to be an agreement to define STEM as the field that deals with science, technology, engineering, and mathematics directly. On the other hand, STEM literacy is a necessary construct resulted from researchers’ interest in STEM. STEM literacy focuses on knowledge and understanding regarding science, technology, engineering, and mathematics (Reeve, 2015). Other researchers, such as Bybee (2010), quotes that STEM literacy also includes higher-order skills such as the application of STEM knowledge and skills. Like other countries, Malaysia also strives to strengthen STEM literacy among its students. The national education plan includes (1) increase students' interest in STEM through formal, informal, and non-formal learning approaches, (2) improve teachers' knowledge, skills, and abilities in implementing STEM education through teacher competency enhancement programs, and (3) raising students’ and the public’s awareness through STEM education campaigns at various levels. Still, the initiatives are not able to gauge student engagement in STEM literacy just as yet. This is shown by the decline in enrolment among secondary school students in STEM-related stream. For example, the number of students from higher secondary schools graduated in STEM programs is only at 45%. The number certainly does not meet STEM-related labor demands, which is estimated to be at about one million starting in 2020. It should be noted that the trend is not unique for Malaysia. According to Thomas & Watters (2015), western countries, as well as several high-income countries in Asia, also recorded a significant drop in students’ interest and motivation toward STEM learning. There is a growing number of studies that focus on attitude towards STEM globally (Kelley & Knowles, 2016; Wang, 2013; Capobianco et al., 2011). One of the prime reasons is that attitude can contribute 8to students’ learning and success in retaining STEM content (Bell et al., 2009). Unfortunately, in Malaysia, despite the importance of this area of study, research on attitude towards STEM is still scarce. One of the possible reasons is that there is no specific instrument developed for the purpose in the past. However, recently, Unfried et al. (2015) have successfully developed a comprehensive instrument to measure attitude towards STEM. The instrument, known as the 2measure of student attitudes toward science, technology, engineering, and mathematics (S -STEM) was developed through rigorous theoretical and methodological procedures as well as using a substantial sample of respondents. Nevertheless, this promising instrument is still subjected to cross-cultural validation since attitudes are considered as context-specific. Since the S-STEM was developed in the United States, its meaning and measure might be different when tested in other cultures. As such, the purpose of this study is to validate S-STEM as a new instrument of measuring attitude towards STEM. In addition to the psychometric aspects of the S-STEM, we also investigate the difference in attitude towards STEM between male and female pre-university students. The issue is crucial and has been widely studied (Ceci et al., 2014; Cheryan et al., 2017) without conclusive results. As such, it is important to (1) ascertain whether the phenomenon exists within the Malaysian pre-university students, and (2) offer explanations based on existing theories why the disparity is still evident after so many initiatives to reduce the gap have been implemented. Measuring Attitude towards STEM S-STEM is an instrument developed 2to measure students’ attitudes toward science, technology, engineering, and mathematics (Unfried et al., 2015). The instrument was developed rigorously by applying multiple theoretical as well as methodological approaches. The development process involves gauging a large number of samples of 4th to 5th and 6th to 12th-grade students. The developers also employed various methodologies such as content validity ratio, the 10confirmatory factor analysis, and the structural equation modeling to provide evidence of reliability and validity of the measure. However, being a new instrument, the S-STEM requires more studies in different cultures to provide stronger evidence on its stability and generalizability in measuring the attitude towards STEM. Before the S-STEM, several established instruments measuring single STEM subjects were available for use. For example, the Test of Science-Related Attitudes (TOSRA) was widely used in Malaysia by researchers such as Lay & Khoo (2012) who used the scale among 15pre-service science teachers. The test was also employed in studies related to upper secondary school students in Malaysia, aged 16 to 18 years old as well as for lower secondary school students, aged 13 to 15 years old (Chua & Karpudewan, 2017; Karpudewan & Meng, 2017). However, these studies did not provide adequate evidence of the validity of TOSRA. This is important since the reliability and validity of TOSRA across culture have yet to be established. For example, while Navarro et al. (2016) showed acceptable psychometric properties in the sample of Spanish students, the internal structure of TOSRA was found to be influenced by race/ethnicity (Villafañe & Lewis, 2016). Concerning the attitude towards mathematics, there is an abundance of instruments used by Malaysian researchers such as the Fennema-Sherman Mathematics Attitude Scale (Mohamed & Razak, 2018; Zakaria & Nordin, 2018) and Mathematics Interest Inventory (Wong & Wong, 2019). The 40-item Attitude towards Mathematics Inventory was also widely adapted (Singh et al., 2019; Long & Jiar, 2014). Yet, these studies also did not present the psychometric properties of the instruments. One might speculate that since the attitude towards mathematics is a context-specific construct, the instrument might demonstrate some deficiencies that might be detrimental to the measurement. While studies on attitudes towards science and attitudes towards mathematics are abundant, the same cannot be said for the other two components of STEM, namely, technology and engineering. When it comes to technology, documented research is significantly related to information communication technology (ICT) rather than STEM-related technology. Perhaps it is not too fallacious to suggest that the dearth of studies may be attributed to the lack of quality instruments to measure the constructs. Therefore, the development of S-STEM may provide useful insights in understanding students' attitudes towards engineering and technology. METHODS The present study employs the cross-sectional research design. The sample of this study consists of 748 pre-university students from a matriculation college in Penang, Malaysia with 166 males (22.2%) and 582 females (77.8%). The sample represents 61.01% of the total number of students. We used random sampling to collect the data. In this study, we employed the 26-item of S-STEM (Unfried et al., 2015). Several modifications to the measure have been made in two phases. Firstly, the instrument 1was translated into the Malay language by engaging a panel of experts consisting of a psychometric lecturer and a psychology lecturer through back-to-back translation procedure. In this procedure, both experts translated the original version, and 1their translations were later compared. Subsequently, the consensus was obtained on the final translation draft. Further, a language teacher with more than 20 years of experience examined the draft and provided the researchers with the final translated version of the scale. Both the English and the Malay equivalent are presented in the questionnaire. During the second phase, several items have been slightly rephrased to suit the Malaysian educational context based on discussions with the language expert. For example, we changed the item “knowing science will help me earn a living” to “I know that my knowledge in science will help me to find a job." Similarly, we changed the item 11“I am sure I could do advanced work in math” to “I can solve questions related to high order thinking in mathematics." The S-STEM defines the measurement of the attitude towards STEM as a multidimensional three-factor model consisting of (1) attitude towards science, (2) attitude towards engineering/technology, and (3) attitude towards mathematics as depicted in Table 1. The attitude towards the technology scale did not exhibit sufficient evidence of content validity based on the content validity ratio statistics. Therefore, the construct was combined with the attitude towards engineering scale to become a new attitude towards engineering/technology scale (Unfried et al., 2015). In this study, we invited the participants to complete the online questionnaire on the Google platform. The process was done on a voluntary basis. Once started, it takes about ten minutes for the participants to complete the questionnaire. Altogether, it takes about three weeks to gather the responses. The responses were collected and transferred into IBM SPSS 24. Prior to filling in the questionnaire, a statement that ensures the confidentiality of the data was provided as well. The validation of the S-STEM was conducted by providing evidence on two critical facets of validity, namely, the 4convergent and discriminant validity. According to Hair et al. (2010), convergent validity deals with an investigation of whether indicators (items) of a particular construct shared an adequate proportion of variance between them. The authors quote that evidence of convergent validity may be examined in terms of 5(1) standardized factor loadings (?), (2) average variance extracted (AVE), and (3) construct reliability (CR). Standardized factor loading indicators correlation between the indicators and the construct. High values of standardized factor loadings indicated that there is a strong correlation, which in turn shows that the indicator is appropriate in measuring the construct. AVE is a measure of whether the group of indicators representing a latent construct. AVE of a set of indicators n is calculated using the following formula (Hair et al., 2010): 𝐴𝑉𝐸 = ∑𝑖𝑛 ?2 (∑𝑛𝑖 ?()∑2+𝑛𝑖?∑)𝑖𝑛21−?2. It provides evidence on the 𝑛 . Meanwhile, the calculation of CR is given by the following formula: 𝐶𝑅 = 16internal consistency of the indicators measuring the construct. 4According to Hair et al. (2010), appropriate evidence of convergent validity is presented when the value of the statistics is above the following guidelines: standardized factor loadings >.50, AVE > .50, and CR > .60. Another crucial facet of validity, the 9discriminant validity, involves the extent to which a particular construct (and its indicators) differs from other constructs.3In this study, we assessed the evidence of discriminant validity using the criterion set by Fornell & Larcker (1981). This criterion asserts that evidence of discriminant validity is presented when the values of the AVEs are 14higher than the correlation between the two constructs. Apart from evidence for the convergent and discriminant validity, another essential analysis associated with CFA is the model-fit evaluation. In brief, when the fit indices are within the acceptable values, it indicates that the model is acceptable. The indices provide evidence on how well the theorized three-factor S-STEM model fits the empirical data. If not, then there is a need to modify the model. The present study opts for four 6goodness-of-fit indices, namely, the normed chi-square (χ2/df), Tucker-Lewis (TLI) index, 7comparative fit index (CFI) and the root square mean error of approximation, RMSEA. RESULTS AND DISCUSSION Table 2 shows that the standardized factor loading estimates ranged from .43 to .93. Three items, namely, items 21, 22, and 23, showed values lower than the cutoff value of .50. Nonetheless, upon further investigation, we believe that the meaning of items is appropriate in measuring attitude towards mathematics; therefore, we retain the items in the measurement model. The empirical data exhibited that the AVE for all dimensions is above .50. Also, we found that the construct reliability of attitude towards science, attitude towards engineering/technology, and attitude towards mathematics are reasonably high at .91, .94, and .86, respectively. Hence, we conclude that the empirical data supports S-STEM in demonstrating excellent reliability and significant evidence of convergent validity to measure attitude towards STEM. Meanwhile, Table 3 shows evidence of discriminant validity based on a criterion by Fornell & Larcker (1981). It shows that all the square root values of AVE were more significant than the correlation between the dimensions. Thus, there is enough evidence to show that the dimensions of S-STEM were truly distinct between each other. Meanwhile, Table 4 shows goodness-of-fit indices for the three-factor correlated in the S-STEM model. All the indices showed an acceptable fit based on the mentioned criteria. Based on this evaluation of model fit, the present study has provided strong evidence that the S-STEM was statistically valid in measuring attitude towards STEM among pre-university students. Finally, Table 5 shows descriptive statistics of the mean scores between the male and female pre- university students. In general, the male students demonstrated higher mean scores in all constructs compared to their female counterpart. Based on the independent sample t-test analysis, the difference is significant for the constructs of attitude towards engineering/technology [t (746) = 7.103, 0 = .000] and attitude towards mathematics [t (746) = 2.919, p = .004]. We report no significant difference in the mean score of attitudes towards science [t (746) = 1.405, p = .161]. The present findings support S-STEM as a multidimensional model that consists of four underlying dimensions, namely, attitude towards science; attitude towards engineering/technology; and attitude towards mathematics. CFA analysis omitted five items that showed poor loadings. However, we feel the need to maintain three items (items 21, 22, and 23) even though they presented loading lower than the intended value because the meaning of the items is considered appropriate to measure attitude towards mathematics. One of the possible reasons for this is that these items are relatively short, which in turn may contribute to high measurement error and a small percentage of common variance (Ximénez, 2015). Therefore, increasing wordings while maintaining their meaning for these items might be appropriate to increase the loadings. Apart from the issue at the item level, the S-STEM demonstrated acceptable evidence of reliability with high values of CR even though most of the items are relatively short. The instrument also revealed acceptable convergent validity based on the acceptable values of AVE. To be precise, the groups of indicators in S-STEM were deemed appropriate in measuring the particular constructs. The excellent structure of S-STEM was further strengthened by the evidence of good discriminant validity between the constructs in S-STEM. Moreover, based on the indices, the empirical data collected also fit the theoretical three-factor model. In general, along with other studies such as Luo et al. (2019) as well as Khor & Zakaria (2019), results from this study indicate that S-STEM demonstrated strong factor structure, which is useful in measuring attitude towards S-STEM in different cultures. Comparison between the male and female pre-university students showed a significant difference in the mean scores on attitude towards engineering/technology and attitude towards mathematics. Reiking & Martin (2018) offer to explain this phenomenon using the gendered socialization perspective. According to them, gender roles cause boys and girls to socialize differently, which directly influences gender stereotypes in STEM professions. Stereotypes, especially with regards to girls’ subpar ability in mathematics (Gunderson et al., 2011), occur among girls at a young age through both teachers and parents. As a result, girls leaving the STEM pipeline before entering the STEM-related workforce (Dasgupta & Stout, 2014). In relation to practical implications, we believe that the validated S-STEM will encourage more STEM- related studies in Malaysia. We are also certain that the S-STEM has the potential to be used across grades in Malaysia because of its short and easy-to-read wordings. As rightly observed by Wang (2013), motivation to pursue career in STEM is a longitudinal process starting even at secondary level. As such, S-STEM is very suitable to be used to design intervention programs to increase their motivation in STEM. In being more specific to the pre- university students, S-STEM also has the potential to facilitate the choice of courses to enroll at university. This is because there's so many courses to choose from. Literature shows that the factors are varied, ranging from background, environmental factors and intrinsic factors of the individual (Nugent et al., 2015). However, perceptions (Kier et al., 2013) and interest (Langdon, McKittrick & Beede (2013); Faber et al. (2013) in STEM careers are known as important predictors. As such, it would be beneficial to take the S-STEM to provide initial details about their major university. Counselors may administer the instrument to provide additional information for the students. As for researchers, the instrument may be utilized to gather data in providing measures to improve students’ participation in learning STEM. 6The findings of this study are limited by the lack of similar studies in other countries; hence we are not able to compare the finding with studies from different cultures. It should be noted that even though there is some methodological limitation, especially with regards to sampling, comparison with findings from other studies might shed some light, particularly concerning the factor structure of S-STEM. This information is vital to support the generalizability of S-STEM in measuring attitude towards STEM. The present study was also limited in terms of generalization since the samples were only from pre-university students in a matriculation college. We would suggest future studies to look into validating S-STEM among school students to further develop norms within the Malaysian sample. Norms help researchers to have a better understanding of the attitude towards S-STEM in Malaysia. Future works should also focus on the effort to establish the predictive validity of S-STEM in investigating its adequacy to measure attitude towards S-STEM with related constructs such as achievement. Despite not being able to compare with other studies, the uniqueness of the study still preserves the nature of this study and the rigorous process of validation. 12Assessment of validity and reliability of the instrument was carried out on a national sample, in tandem with several analyses, including demographic features. Furthermore, the CFA findings were compared with well-established guidelines. In addition, we have also provided comprehensive literature review and have been able to understand what has already been published in the area of attitude towards science, technology, and engineering. CONCLUSION Based on the government's aspiration to increase students’ engagement in STEM, non-cognitive information such as attitude towards STEM is vital to support initiatives at school, pre-university and university levels. For this purpose, it is essential to have a valid and reliable instrument such as S-STEM. The instrument is reliable and also has good convergent and discriminant validity. Moreover, the empirical data also support the factor structure of the instrument. Adding to this, S-STEM is also practical for various levels of users. Thus, we strongly believe that the instrument has great potential to be used as a tool not only to provide a better understanding regarding attitude towards STEM among Malaysian samples of students but also to steer more research involving attitude towards STEM among researchers.