Integrating questions about learners’ code in an automated assessment system

Abstract

Questions about Learners' Code (QLCs) assess programming students' program comprehension skills by providing personalised questions targeting the students' own program code. We conducted a preliminary, experimental implementation of integrating QLCs in the Automated Assessment System (AAS) used in an introductory programming course using Java. QLCs targeted some of the code assignments which students had to complete during the course. We collected 889 answers to QLCs, answered by 13 students over five course modules. We found that as the complexity of exercises increases, the success rate of the same type of QLC may not improve, and even exhibit a decline over time. We further analysed incorrect answers individually to relate them to possible misconceptions.