Usability Evaluation of Lecturer Information Systems Using Sirius Framework and Moscow Technique

Purpose: This study aims to evaluation of the SID ITB STIKOM Bali usability was carried out using used framework and the Moscow technique, where until now there has never been an evaluation of the usability of SID, causing the relevant stakeholders not to know the usefulness of the system and not knowing the improvements that need to be made to the SID system. Methods: The method chosen in this study is literature study and observation, SID system analysis, distribution and processing questionnaire data, data analysis, and concluding. Result: The result of this study is the global usability value for SID based on ten aspects of the Sirius framework usability and recommendations for improving the SID STIKOM Bali system based on the Moscow Technique ranking. Novelty: This research's novelty can be a reference material for stakeholders in determining policies related to improving the SID system at ITB STIKOM Bali.


INTRODUCTION
The development of technology is increasingly rapid, and many have shown extraordinary progress. This is increasingly affecting various sectors of human life. One of the influences is in the field of education, especially at the university [1]. Constructively IT (Information Technology), business process or job activity process in university can be put across and efficient [2]. An example of the use of technology in university is developing an information system for the higher education institution to support performance and facilitate the flow of information. An information system is an organized way to collect, enter, and process data and store, manage, control and report it to support a company or organization to achieve its goals [3]. One of the universities that develops information systems is the Institute of Technology and Business (ITB) STIKOM Bali is the Institute of Technology and Business (ITB) STIKOM Bali is an information technology-based university that has various supporting information systems for its institutions. One of the information systems developed is the Lecturer Information System (SID), considering the role of Lecturers as the main actor in the world of higher education.
Lecturer Information System (SID) was developed by the Institute of Technology and Business (ITB) STIKOM Bali starting in 2013 to facilitate and streamline Lecturers' work. SID provides information related to one of the Tri Dharma Perguruan Tinggi activities, namely education. The Dharma of education is about teaching, but there are other activities such as guiding final assignments, practical work, guiding as guardian lecturers, assigning duties as course coordinator lecturers, and many others. The implementation of SID should benefit tertiary institutions so that the resources spent on building this system are not wasted. However, until now there has never been an evaluation of the usability of SID, causing related stakeholders such as SID system developers, heads of study programs, deans and academics to not know the usefulness of this system for users who in this case are lecturers and do not know the improvements that need to be made in SID system.
The usability of technology is important because it deals with user performance, effectiveness, and efficiency. Effectiveness refers to accuracy and completeness in achieving certain goals. Efficiency refers to resources spent by the user to increase the accuracy and completeness in achieving their goals [4]. Usability testing is a category of usability evaluation methods used to evaluate a product by testing it directly on the user. The aim is to identify usability test problems such as collecting qualitative and quantitative data, measuring convenience, measuring efficiency and determining user satisfaction with the product [5]. Usability testing of applications or software has been carried out in many previous studies by [6] - [7]. Evaluation of the level of usability in a software shows whether the software is feasible and meets the usability criteria, namely effectiveness, efficiency and user satisfaction [8]. Systems with high usability will be used for a long time because many people feel the system's benefits. Meanwhile, systems that have low usability will eventually be ignored by users [9].
Based on the problems described above, this research will evaluate the usability of SID ITB STIKOM Bali using the Sirius framework and the Moscow technique. The Sirius framework is a website usability evaluation tool based on heuristic assessments. The advantages of this framework include being able to detect usability errors from an expert's point of view, using comprehensive aspects and criteria in measuring usability and being able to produce a quantitative global usability value for a website according to its type and generate feedback for the website developer to improve its user interface. The Sirius framework uses ten aspects of usability, namely general aspects, identity and information, structure and navigation, labeling, layout of the page, comprehensibility, and ease of interaction, control and feedback, multimedia elements, search, and help [10]. In addition to using the Sirius framework, this research also uses the MoSCoW technique to determine priority needs for improvements that should be made to the system. The advantages of MoSCoW are that it is easy and fast to sort the list of requirements and can produce an accurate order. This technique classifies all needs into 4 (four) classes based on their priority level [11].
The previous research by [11] applies a combination of the Sirius framework and the Moscow Technique to evaluate several websites, where Sirius can generate global usability values and a list of system interface elements that need to be improved and evaluated results are ranked using the Moscow Engineering priority scale. Based on previous research, this research examines the usability of an information system related to lecturers at an information technology-based university using the Sirius Framework and the Moscow Technique. The differences of this research from research by [11] where the expert is a lecturer with knowledge related to usability and system development directly interact or have used SID. By using the evaluator, it is hoped that the evaluation results will improve the accuracy of the evaluation, namely to produce the most urgent criteria order and a priority list of improvement criteria according to real conditions. The results of the research are the global usability value for SID based on ten aspects of the Sirius framework usability and recommendations for improving the SID STIKOM Bali system based on the Moscow Technique ranking. This research is expected to be a reference material for stakeholders in determining policies related to the improvement of the SID system at ITB STIKOM Bali.

Related Research
Several previous studies have evaluated usability using different methods. Research by [12] evaluating the usability of UNDIKSHA Lecturer SIAK by using ISO / IEC 9126-4, which has three attributes, namely Effectiveness, Efficiency, and User Satisfaction. The results of this study, namely the effectiveness attribute, the results obtained were 86.92% for novice respondents and 86.92% advanced respondents, which means that SIAK UNDIKSHA was effective. In the efficiency attribute, the results obtained for novice respondents were 87.29776% and for the advanced respondent group of 84.6087% which means that the Undiksha SIAK is efficient. In the attribute of user satisfaction, the result is 77.1%, which means that the respondent is satisfied in doing the task at SIAK UNDIKSHA. Research by [13] conducted an E-Raport evaluation at schools in East Lombok. Based on the results of the evaluation, the e-report card has not been effective (completion rate <90%) and has not met user satisfaction (SUS score <68), even though the e-report card has been said to be efficient both from experts and lay users (not significant with ρ <0.05). There are several recommendations for improvements to the e-report card, namely the need to improve the database design, the need for additional help modules from all submenus so that users are helped if they forget or are confused in the operation of e-report cards and need to improve the final import and export class interface. Research by [14] evaluating usability on Bank Mandiri Syariah's (BSM) KNOWN system. Based on the evaluation results, the usability value yields a total value of 3.21 while the content value produces a total value of 2.99. Some recommendations for improving KNOWN are developing the KNOWN site so that it can be accessed anywhere, adding or updating content on the KNOWN site and integrating with other employee systems. Research by [15] evaluate the usability of Google applications, namely Google Drive, Google Classroom and Google Pluas based on student perspectives. Based on the results of the evaluation, Google Classroom has the highest utility value of 86.45 and also shows a statistically significant difference in scholastic achievement in the application of Google Classroom in the educational process. Research by [16] evaluate the usability of the online debate website. Based on the results of the evaluation, it shows that the average score of ease of learning and ease of use is high at 3.97. As a result of evaluating user experience, perspective and efficiency were found to be the highest among students and instructors at 3.9. This shows that the online debate website is useful for students and instructors in identifying debate status.
Research related to the Sirius framework was carried out by [10] conducting a heuristic evaluation of the SIMKESMAS website using the Sirius framework. Based on the evaluation results, the global website usability value is 57.65% which indicates that most of the usability aspects have not reached the ideal value. However, from the results of the analysis, it was concluded that only about 5% of the usability criteria were mutually agreed upon by all evaluators as an issue that must be fixed by the website building team. Research related to the Sirius framework and the Moscow technique was carried out by [11] conducting usability evaluations for three websites, namely Lective, Seminar Indonesia and SIMKESMAS. Based on the results of the evaluation of the usability value for Lective, it was 63.56%, for the Indonesian Seminar it was 69.61% and for SIMKESMAS it was 57.65%, so it can be concluded that the three websites were in a proper condition. In addition, based on the ranking using the Moscow Technique, the conclusion is that for the Lective website, the most important usability criteria group is related to the labeling aspect, namely content labeling, the Indonesian Seminar website, the most important aspect is group comprehensibility and ease of interaction and the Simkesmas website emphasizes the control aspect. and feedback.

Theoritical Framework Usability Testing
Usability is actually a measurement of the usefulness a system from the user's perspective [17]. The study of usability is part of the multi-disciplinary field of Human Computer Interaction (HCI). Usability comes from the word Usable which generally means to be used well. Something can be said to be useful well, especially failure in its use can be eliminated or minimized as well as providing benefits and satisfaction to the user [18]. Usability testing is one of the methods used to find out all functions that can work in a system interface, namely by paying attention directly when a system user is using it. The purpose of doing usability testing is to be able to identify existing usability problems, get data for research in the form of qualitative data and quantitative data, and measure the satisfaction that users have with the system [19].

Information System
Information system is a system within an organization that meets the needs of daily transaction processing, supports operations, is managerial and strategic activities of an organization and provides certain external parties with the necessary reports [20]. An information system is a collection of interconnected components in collecting, processing, storing, providing and distributing information to support decision making and control within the organization [21].

Academic Service Information System
The STIKOM Bali academic service information system is an information system that supports all academic processes in STIKOM Bali. This information system includes. 1) The online information system (SION) is an information system used by students to find out all information related to academic activities. 2) Academic information system (SINAK) is an information system used by the academic department to schedule lectures, plotting lecturers, recaps lecturers' teaching and others.
3) The online attendance information system is an information system used by lecturers to conduct student attendance at each meeting. 4) E-learning is an information system that can be used by students and lecturers to support the learning process, both uploading lecture materials, assignments, discussions and others. 5) The guardianship information system is an information system used by students to select courses or guardianship at the beginning of each semester.

Lecturer Information System
The Lecturer Information System is one of the information systems available at ITB STIKOM Bali which is used by lecturers to make it easier to find information related to education such as teaching, guiding students, student guardian counseling, final assignment examination activities and downloading files such as decision documents related to lecturer activities.

Framework Sirius
Sirius is a usability measurement framework that can be applied to websites. Sirius is a usability evaluation framework that is heuristic and measures tailored to the type of website. Evaluation using the Sirius framework is carried out by experts, including a set of usability criteria and metrics to produce a value that indicates the level of usability of the website. The strengths of this framework include being able to detect usability errors from an expert's point of view, using comprehensive aspects and criteria in measuring usability, and being able to produce a quantitative global usability value for a website according to its type. The output from Sirius is the ideal website criteria, and the level of usability of each of these criteria is sorted based on the potential improvement given if the ideal conditions of each of these criteria are achieved which is called increased value. The framework model is shown in Figure 1 [22].  Table 1 and each aspect has sub-criteria so that the total criteria are 83 items. Features that should be there but the benefits are less than the features in the "S" category Would Have Features that are not necessary but can be implemented if there is excess budget

Research Methodology
This section will explain the steps in the research. The research methodology carried out can be seen in Figure 2.  Figure 2 shows the research methodology carried out in which the first step is literature study and observation is carried out by looking for references related to usability, the Sirius framework, Moscow Technique and ITB STIKOM Bali lecturer information systems. At this step, observations were also made at ITB STIKOM Bali regarding the management of SID. The second step is the analysis of the SID system where the SID ITB STIKOM Bali feature analysis is carried out and the analysis of the relationship between aspects of the Sirius framework and the features available in SID. The third step is the distribution and collection of questionnaire data where the questionnaires are distributed to the selected evaluators who are experts in the field of user interface and website, then the results of the questionnaires that have been filled in are collected for processing. The fourth step is the analysis of the result data where at this step the questionnaires that have been collected are evaluated using the Sirius framework and also the Moscow technique. The fifth step is the conclusion where this step is carried out by drawing conclusions on the global usability value of the SID system and in the form of a priority list of elements of SID ITB STIKOM Bali.

RESULT AND DISCUSSION SID Feature Analysis
This research was conducted to evaluate the usability of SID. Therefore it is necessary to analyze the SID feature used by lecturers. The features of the SID are as follows. This feature for downloading documents related to lecturer assignments and functions such as teaching decree, supervisor lecturer decree, functional position calculation, credit score assessment, final assignment guidelines, and others.

Evaluation Result Using Sirius Framework
The questionnaire was distributed online using a google form distributed to 8 experts or evaluators, where this expert was a lecturer who taught human and computer interaction and web programming. The questionnaire distributed consists of ten aspects of usability with the Sirius framework with the criteria broken down into 83 questions. Each evaluator can assess with a format using the numbers 0 to 10 where the value 0 represents the criterion wholly unsuitable and the value 10 indicates a very suitable category. The second format is by determining labels/categories where there are 5 categories, namely NWS, NML, NHP, NPI, YES. The results of the SID system evaluation for each evaluator can be seen in Table 3.  Table 3 shows the results of the usability value for the SID system for each evaluator. Different levels of evaluator expertise caused the various evaluators' assessments of the tested websites. From the evaluation results, the lowest assessment is 68% where this value shows the extent to which the usability aspect is implemented in SID. The highest score of the evaluator is 100% where this value states that most of the usability aspects have been applied. The percentage of global usability value measured using the Sirius framework is 89%, which is included in the "very feasible" category. The usability problems found were the result of a reduction in the value of the scale (100% -89%) so that the usability problems found were as large as 11%.

Ranking Result Using Moscow Technique
Based on the results of the priority setting by the representatives of the SID developer team that were analyzed, it was found that the priority setting was Moscow. There are considerations in determining priorities based on several factors, namely cost, risk, and business value. The weight and definition of each consideration factor is left to the priority determinants, in this case the representatives of the SID development team. In knowing which of the 83 Sirius Framework criteria are considered the most important (must have) on each website, a calculation of the number of criteria that includes the "must have" criteria is calculated. The SID mapping graph is shown in Figure 3. Figure 3. SID mapping chart Figure 3 shows the SID system mapping graph, based on the figure, it can be concluded that for the SID system, the most important group of usability criteria is related to aspects of Comprehensibility and Ease of Interaction, namely related to the quality of text content, icons, and interface settings.

CONCLUSION
According to eight evaluators, the conclusion that can be drawn based on the evaluation of the usability of the SID system with the Sirius Framework is that the lowest assessment is 68% and the highest score from the evaluator is 100%. Based on the evaluation results by all experts, the global usability value is 89%, which is included in the "very feasible" category, meaning that most of the ideal usability criteria are achieved on the SID website. Furthermore, for the SID system, the most important group of usability criteria using the Moscow technique is related to aspects of Comprehensibility and Ease of Interaction, namely related to the quality of text content, icons, and interface settings. Suggestions for the implementation of the next research stage are the use of other frameworks to evaluate the usability of the SID system and recommendations for improvements that should be made.