Developing Tasks Designed Test to Assess Professional Competencies at the Stage of Study at the University

Abstract

We aim to develop and test tasks designed to assess professional competence at the university's formation stage. The system of internal competency assessment utilizing the requirements of professional standards in the IT industry enables bachelor's degree graduates to assess their level of preparation for professional activities. Our study employed the R&D design proposed by Borg & Gall, M.D. The essence of R&D design in our case is that it is necessary to implement the following five steps: 1) conducting research, 2) developing a prototype, 3) a small-scale experiment to improve the prototype into a model, 4) a large-scale experiment to improve the model into a final product, 5) dissemination through publication. We defined a competency model to develop tasks: Competence = [Knowledge + (Skills and Abilities)] in labor functions. Labor functions are described in professional industry standards. Our formula is based on the definition of the competency model in Computing Curricula 2020, specifying the context concerning professional activity. A version of 20 tasks assessing the prerequisite knowledge, skills, and abilities to perform the labor functions of the Database Administration standard was prepared for testing. Our experience has demonstrated that preliminary work must meet professional standards. Some knowledge and skill requirements are formulated without considering the property of measurability. Cases of requirement redundancy also exist for graduates who do not have practical experience. The results were processed by classical test theory. The main indicators of task quality were the correlation of each task with the total score, the difficulty level, and the discriminativeness index of each task. Cronbach's alpha, an indicator of test reliability, was also calculated. The obtained values of quality indicators enable us to posit that the tasks can be used for assessment. However, some tasks require content correction and revision by distractors. Our findings can be employed to develop a competency assessment system based on the requirements of professional standards in industries other than IT.