The Evaluation of Higher Order Thinking Skills in English School Nationally Standardized Examination at State Senior High School 6

____________________ Abstract ___________________________________________________________________ The English school nationally standardized examination as a means of the assessment of the learning outcomes conduct by educational unit is designed to measure the student`s achievement competencies, to determine student`s graduation from the school, it is very significant to conduct test items evaluation since it gives a clear portrait of the quality of the items and the test as a whole. The purpose of this study was to explain the realization of higher order thinking skills reflected in English school nationally standardized examination at state senior high school 6 Semarang academic year of 2018/2019. The researcher adopted the qualitative descriptive approach using an analysis card to analyze the realization of higher order thinking skills in English school nationally standardized examination test items. The results indicated that HOTS realized in listening, reading, and writing questions. The listening in the form of spoken written, because it is to test listening simultaneously with speaking. There is one listening question categorized into HOTS of level analyzing, in reading the questions belonging to HOTS reach 8 questions that consist of 5 questions of analyzing level and


INTRODUCTION
Curriculum 2013 has been launched by the government since the academic year 2013/2014 and implemented gradually at all levels of school throughout Indonesia. The curriculum 2013 is prepared for generations to face the globalization. The government develops curriculum 2013 by enhancing student potential for competitiveness in the 21st century. Thoyyibah, Hartono and Bharati (2019) argue that the implementation of the curriculum 2013 will be successful if all the components of education give their support and contribution to it. The aspect that changes in curriculum 2013 is the assessment. Assessment is an important component in the education system to determine the progress and level of achievement of learning outcomes. It is done to make sure whether the learning process has been running well or not.
Assessment of learning outcomes by education units is conducted to measure student competencies as recognition of learning achievement of the education unit. An example of learning outcomes assessment in Indonesia is the school nationally standardized examination or in Indonesian term ujian sekolah berstandar nasional (abbreviation into USBN) is an activity to measure student competency achievement carried out by the education unit refer to graduates' competency standards to gain recognition for learning achievement (The regulation of the Minister of Education and Culture number 24, 2018). It is an internal assessment to measure what have students learned. Because the school nationally standardized examination results determine graduation from education units, the test items are expected to meet the requirements good instrument so as to provide valid and objective information. Test items that are not good provide information that is not in accordance with the achievements students so that they can harm students and provide incorrect information or misleading decision makers.
Generally, school nationally standardized examination test items are constructed by teachers only measure the lower level of thinking skills not inserted higher order thinking skills that cause learners not accustomed to working on higher order thinking skills test item. On the other hand, in nationally examination learners are required the ability to work on higher order thinking skills questions. Every year the percentage of higher order thinking skills test items which are inserted in the national examination is improved. Wahyudi, Rukmini, and Bharati (2019) stated that high order thinking skills become the requirements of the most curriculum, including the 2013 curriculum in Indonesia. It is in line with BSNP (2018) stated that in the national examination year 2018/2019 there is approximately 10-15% of higher order thinking skills test items. Therefore, higher order thinking skills should be prepared and inserted in school nationally standardized examination test items.
It is important to insert higher-order thinking skills in English school nationally standardized examination. Thomas and Thorne (2009) explain that higher order thinking skills as a way of thinking at a higher level than memorizing or retelling something that is told by other people. Furthermore, King, Goodson, and Rohani (2013) argue that higher-order thinking skills are the ability to think that not only requires the ability to remember, but also higher capabilities.
Meanwhile, Anderson and Krathwohl (2001, p.21) define higher-order thinking skills is specified in three-dimensional thinking consist of analyzing, evaluating and creating.
English school nationally standardized examination is very important to determine the students` mastery of higher order thinking skills. Besides, the outcomes of school nationally standardized examination become one of the elements to determine students` graduation from the educational unit. Higher order thinking skills of item test on school nationally standardized examination is an important key to know whether or not each item has met the process stages of thinking and competencies in the 21st century.
This study is primarily anchored on Bloom`s revised taxonomy by Anderson et al (2001, p.28), namely: remembering, understanding, applying, analyzing, evaluating and creating. The theory assumes that learners move from remembering to another until the learner fully understands and putting the elements together to form a coherent or functional whole of the material. The first level is remembering in order to make this stage more functional. Students are not only expected to have facts but also to remember them accurately for use. There is an increase in cognitive processes for example learners are not required to know a concept only but also remember the concept they studied. Then understanding student able to construct meaning form different types of function is they written or graphics. Applying refers to the ability to carry out or using a procedure through executing or implementing. Analyzing deals with breaking the concept into the part, determining how the parts relate to one another or to an overall structure or purpose. Furthermore, evaluating refers to making judgments based on criteria and standards through checking and critiquing. Finally, creating deals with putting the elements together to form a coherent or functional whole. The higher order thinking skills, thinking level is seen from Bloom's revised cognitive domain they are analyzing, evaluating, and creating.
It is worth noting that higher order thinking skills happen when learners find out the answer from the test items. Applying, analyzing, evaluating and creating are instances of the level of items developed based on the taxonomy affects the performance of learners in answering test items. The effects of using (HOTS) strategies do not only improve the learner's listening and reading comprehension but also their thinking, brainstorming and writing abilities. However, despite the significance of evoking students' higher order thinking skills, many test items are still designed to test students' LOTS.
The object of the study of this thesis is the higher order thinking skills. Thus we see it is essential to review some studies previously conducted by Pilten (2010) study the evaluation of the skills of 5 th -grade primary school students' high-order thinking levels in reading. The result shows that the evaluation skills are high, the synthesis skills are mean, the analysis skills are mean, and the interpretation skills are low. Assaly and Smadi (2015) evaluating the cognitive levels of the questions following the reading texts of Master Class textbook. The result shows indicated that about 40% of the textbook's questions emphasized higher-order thinking skills, which goes with the requirements of the revised curriculum. Yuwono and Pasani (2018) study the evaluation of higher order thinking skills assessment of special needs education students with guided inquiry method. The result shows that in arranging an assessment, the lecturers have not fully used the case study model to reach HOTS level. Pratama and Retnawati (2018) study the urgency of higher order thinking skills (HOTS) content analysis in Mathematics textbook. The result shows that HOTS is one of the main goals in education and become one of the top five variables that can improve student achievement. HOTS can be developed but cannot be automated and requires practice.
Research on higher order thinking skills through Bloom`s taxonomy and Marzano was conducted by Mei Heong et al (2011); Dubas et al (2016) and Stayanchi, (2017). These researches investigate the level of Bloom and Marzano Higher Order Thinking Skills among the students. The use of Bloom's Taxonomy as a framework for teaching and assessment is an effective way to structure a lesson, unit, or curriculum. The taxonomy includes both ways to assess knowledge (factual, conceptual, procedural, and metacognitive) as well as develop cognitive skills (remembering, understanding, applying, analyzing, evaluating and creating). While the use of Marzano`s taxonomy indicated that students perceived they have a moderate level for investigation, experimental inquiry, comparing, deducing, constructing support, inducing and invention. However, decision making, problem solving, error analyzing, abstracting, analyzing perspectives and classifying are at a low level. The conclusion of these researches points out that the development of higher order thinking skills is crucial to academic success.
There are also some other studies on higher order thinking skills test items conducted by Indonesian researchers. Laily and Wisudawati (2015) study analyzed higher order thinking skills in national examination of Chemistry Rayon B academic year 2012/2013. The result shows that the type of HOTS characteristics that appear is stimulus while the creative and critical thinking conclusion can not be generated. Winarti et al, (2015) study investigated higher order thinking skills content in Physics item tests using dimensions of thinking by Bloom's Taxonomy given to Madrasah Aliyah students in Yogyakarta. The result shows that the examinations test items dominated the memorizing and the formula applications level, it also reveals that the level of evaluating and creating is rarely and even never used in the examinations. Iffa et al (2016)  Those previous studies recommend that other researchers need to conduct studies in analysis of HOTS items to see to the realization of the higher order thinking and students thinking process to solve the HOTS items. Thus, in this study, the researcher is interested to conduct the research about the realization of the higher order thinking skills in English school nationally standardized examination.

METHOD
This qualitative research purpose to explain the realization of higher order thinking skills in the English school nationally standardized examination The subject of this research is the English school nationally standardized examination of the academic year 2018-2019. The object of this study is the higher order thinking skills test items on English school nationally standardized examination. The major instrument in conducting this study is documentation. The researcher also developed a tool called categorical higher order thinking skills, describe and analyze data based on the characteristics of higher order thinking skills in English school nationally standardized examination in the form of analysis card.
To gather all the information needed, the researcher collected all suitable documents that are available. The documents collected in this research were, the test paper and answer sheets of English school nationally standardized examination for senior high school in academic year 2018/2019 obtained from schools, and the English school nationally standardized examination blueprint from Board of Education National Standard.
To analyze higher order thinking skills on English School nationally standardized examination, several steps are done in analyzing data. First of all, select the school nationally standardized examination paper test items. After selecting the document, the writer identifying and put into types of the question-based on Bloom`s Taxonomy. Then the data of higher order thinking skills were analyzed through analysis card. Finally, the writer analyzes and interprets the finding based on the data.

RESULTS AND DISCUSSION
Data of the questions asking students' LOTS and HOTS were obtained from the paper test of English school nationally standardized examination. In order to show how the data were analyzed, some part of the total data was chosen as an illustration. For this reason, some items of the English school nationally standardized examination are presented as an example.

HOTS in Listening Questions
Based on the paper test of English school nationally standardized examination, it was discovered that listening and speaking questions had similar questions and options The listening comprehension was required in 11 items of questions, the listening assesses through the spoken written form. Here students are required to give the response to the questions, the students answer the question not orally but choosing the correct answer in written form.
Among the 11 listening questions, there are 10 test items of lower order thinking skills (LOTS). The various levels of the nine lower order thinking skills test items consist of understanding level 3 items, applying level 6 items. Furthermore, there is 1 listening question of higher order thinking skill. The variation of higher order thinking skills only covers analyzing level. It means that higher order thinking skills in listening questions is realized even though the number and variation level of higher order thinking skills need to be improved so that the higher order thinking skills not focused in one level but more varies and lead the students to improve their process of thinking.
In order to show how the data were categorized and analyzed, some part of the total data was chosen as an illustration. For this reason, some items of the English school nationally standardized examination of the 2018/2019 academic year are presented as an example.

Figure 1. HOTS question in Listening English school nationally standardized examination
The level of higher order thinking skills in the question above is analyzing level, it means that to find out the answer the students should breaking concepts into the part, determining how the parts relate to one another or to an overall structure or purpose. It is in line with the operational word that uses in the question "conclude", to find out the answer/make conclusion the students should read or listen all the text then related the text to determine an overall structure or purpose of the text, in this case, the text is an announcement in spoken written form. In the term of the depth of the knowledge level, the question categorized as strategic thinking/reasoning because here students analyze and draw conclusions from more complex data and/or multiple sources. Furthermore, the complexity in the listening question above is factual because the knowledge that needed to answer this question is detailed knowledge and specific elements refer to the knowledge of events, place, people, date, source of information, etc. The last analysis of the role of the HOTS in that questions as the critical and creative thinking, the steps of critical thinking here focus-reason-inference. During focus step students identify the problem, then in reason step, the reasons given are logical or not to be concluded. Then in the inference step, if the reasons' being developed is correct, the reasons should be sufficient for the actual conclusion.

HOTS in Speaking Questions
Actually, testing speaking has a specific procedure usually through monolog, interview, and free conversation. In English school nationally standardized examination, speaking and listening testing simultaneously in the form of spoken written. There are seven questions that test listening and speaking simultaneously, which the testees are required giving the response in the written form.
The speaking performance was covered in seven items. The distribution of the questions that testing listening and speaking simultaneously in question number 1, 2, 3, 5, 6, 7, 8. From the seven questions of listening and speaking simultaneously, the higher order thinking skills are not realized because all of them categorized as the lower order thinking skills questions. The background that the speaking and listening tested simultaneously because first, speaking cannot be tested massively but should be tested individually. It means that speaking test need much time, cost and labors to conduct that test. Second, speaking is productive skills that relate with the performance, so speaking test more appropriate conduct separately with the other skills. Third, the ministry of education and culture handover the speaking tests in practice examination. This practice exam is not part of the English school nationally standardized examination because this blueprint of the speaking test is made by the school so the speaking questions are not nationally standardized.
The speaking skill assesses through the spoken written form simultaneously with the listening questions because in senior high school the degree of literacy at the informational level, it means students are able to access knowledge with language skills. It is in line with Widdowson (1996:59) says that the skill of speaking involves both receptive and productive participation.
Higher order thinking skills in speaking questions are not realized because from the total question that tested listening and speaking simultaneously in spoken written form, the higher order thinking skills is not appear. All of those questions categorized in lower order thinking skills with the level of understanding 4 questions and 3 questions in categorized of applying level.

HOTS in Reading Questions
The findings show that reading questions is the biggest part of the paper test in English school nationally standardized examination. The distribution of reading questions is 28 questions from a total of 45 questions. The other finding, based on the blueprint and numbering guideline of school nationally standardized examination in reading question there are ten questions provided by the ministry of education and culture. Even though there is the realization of the eight questions categorized as HOTS questions have met the proportion, actually, the variation level of the HOTS question only covers analyzing and evaluating.
The example of the HOTS in reading questions is presented below:

Figure 2. HOTS question in Reading English school nationally standardized examination
From the example question above, the level of higher order thinking skills in the analyzing level, it means that to find the answer the students should breaking concepts into the part, determining how the parts relate to one another or to an overall structure or purpose. It is in line with the operational word that uses in the question "correlate", to find out the answer/correlate the relationship the students should read the whole text then breakdown the information and then find out the correlation of the complex material, in this case, the text is a personal letter. In the term of the depth of the knowledge level, the question categorized as strategic thinking/reasoning because here students compare information within the text and also how it affects the interpretation of the reading selection.
Furthermore, the complexity in the reading question above is conceptual because the knowledge that needed to answer this question is knowledge of principle or generalizes the relation to present the systematic point of views about the subject matter. The role of the HOTS in that questions is the critical and creative thinking, the steps of critical thinking here focus-reasoninference-situation. The focus step students identify the problem, then in reason step the reasons given is logical or not to be concluded. Then in the inference step, if the reasons' being developed is correct, the reasons should be sufficient for the actual conclusion. And the last step is the situation, here compare or correlate with the subject matter.
Concerning these findings, it can be said that the LOTS questions are dominated in English school nationally standardized examination, the questions needed to be enriched with more HOTS such as analyzing, evaluating and creating. In other words, more evaluating and creating level questions should be provided so that students would have the opportunity to express their opinions, feelings, and attitudes to be creative and innovative thinkers. Students of senior high school are demanded to be able to cope with the development of technology as well as the creative industry. It is in line with Trilling and Fadel (2009, p.8) who point out that there will be a rising demand of workers who can fill in the jobs that involve higher levels of knowledge and applied skills like expert thinking and complex communicating. Consequently, students need to sharpen their knowledge and insight, exercise their minds to think critically and learn to communicate effectively so that they can survive to deal with the needs 21st century.

HOTS in Writing Questions
In writing, there are two types of questions multiple choice and essays. In multiple-choice, the students are required to find the answer to the questions by completing the sentence with the answers provided. While in essays form the students should answer the question in written form. In English nationally standardized examination at state senior high school 6 Semarang there are 6 questions of writing. One question is multiple choice and 5 questions in the form of essays.
The example of the HOTS question in writing is illustrated such as follow: The level of higher order thinking skills of the example question above is creating, it means that to find the answer the students should be able to pull together parts of knowledge to form a new whole and build relationships for a new situation. It is suitable with "create" as the operational word that uses in the question. To answer in this case create the dialogue based on the situation given students should mastery the expression about the situation in that case expression that used is expression giving suggestion. The students also relate the expression of suggestion with the situation to make the dialogue cohesive and coherence. In the term of the depth of the knowledge level, the question categorized as extended thinking because here students synthesize information across multiple sources or text then design a model to inform and solve a real-world, complex situation. Furthermore, the complexity of the question above is metacognitive because the students develop the knowledge about the strategy and thinking process to solve problems by finding solutions, planning, create a procedure to achieve a particular purpose, or produce something new. And the last analysis of the role of the HOTS in that questions as the problem solving, the steps of the problem solving here the ability to solve problems by finding solutions, planning, procedure to achieve a particular purpose, or produce something new to define the problem so that it becomes more detailed, and prepare criteria to determine the outcome of the discussion of the problem faced. Then students exploring problems, determining objects that are related to problems, examining problems related to assumptions and stating related to the problems. After that student plans a solution where students develop a plan to solve a problem, map the sub-material related to the problem, choose the principle theory and approach that suitable with the problem, and determine the information to find a solution. Students carry out the plan, in this stage students apply the plan that has been set then check the solutions, evaluate solutions used to solve the problems. Finally evaluating, in this step, the solution is examined, assumptions related to the solution are made, estimating the results obtained when implementing solutions that have been made.
The writing questions in the form of multiple choice. The students are not required to write down answers which enable teachers to assess their correct spelling or the students' ability to organize and develop ideas logically. But it can be cover in essay form. Second, the indicators of students writing skills mentioned in the blueprint merely cover the students' competence to arrange jumbled sentences into a paragraph and to fill in the blanks of the cloze test. According to Brown (2004, pp. 201-210) these types of assessment tasks are classified into assessing interactive reading; cloze test and sentence-ordering task. Furthermore, in assessing student`s writing, Valette (1967) states that a test must consequently be so structured that they measure the various aspects of students' progress toward the acquisition of the skills. It means that the students can explore their higher order thinking skills through writing because writing can be done by oneself.

CONCLUSION AND SUGGESTION
This study focused on the evaluation of higher order thinking skills in English school nationally standardized examination at senior high school 6 Semarang. In this phase, the researcher shows the conclusion of the seven questions.
The realization of higher order thinking skills in listening is realized in one question it means that the variation level of the higher order thinking needed to be enriched. So, the student`s lost their opportunity to use higher order thinking to find out the answers. The listening questions are in the form of spoken written not in the audio form. When the researcher asked about the listening in the form of spoken written form she said that there is no obligation to conduct listening questions in audio but the spoken written form is adjusted with the material of listening blueprint about interaction transactional. The spoken written questions format would test listening concepts without having the students actually listen to the audio. She thought that it is difficult to prepare the audio in native-like style. She knew that the listening question in the form of spoken written is acceptable in senior high School because the level of the literacy is informational literacy level.
The questions of speaking in the English school nationally standardized examination is assessing simultaneously with the listening in the form of spoken written. In speaking questions the realization of higher order thinking skills has not appeared, the test of speaking question is completed in practice exam but the blueprint and questions are not nationally standardized. It is free for the teacher to construct the questions of speaking practice exam.
The realization of higher order thinking skills in reading questions indicate the presence of almost all levels of cognitive domains in reading questions, except "creating" which is the highest level of thinking in Bloom`s revised taxonomy. It is obvious that lower order thinking skills (LOTS) are the main concern in reading questions. This finding reveals that there is still much room for reading questions be the driving force in the effort to make learners critical thinkers.
The writing questions consist of two types namely multiple choice and essay form. In the form of a multiple-choice test, writing question is more likely to cross over into the domain of assessing reading because the words which serve as the options. In essay form, the questions of higher order thinking skills realized in the highest level of Bloom`s revised taxonomy level of creating. Actually, essay form in writing questions is the best way to realize higher order thinking skills because students could express the ability of critical and creative thinking, to find out and analyzing information in the problem-solving problem faced.