A probabilistic skill map model for assessing learning processes

Unlike the summative assessment, that points to grade the learning outcome of a student, the formative assessment is ongoing and aims to improve knowledge, skills and abilities of that student by guiding the teaching and learning process at the individual level (DiBello & Stout, 2007). In addition, it helps the teacher to ascertain whether the educational intervention is effective in promoting specific learning or needs to be modified. In such a formative approach, we propose a model for assessing the knowledge of a student in the different steps of the learning process and the effectiveness of educational interventions in the acquisition of specific skills. THE MODEL: The theoretical framework is the Knowledge Space Theory (Doignon & Falmagne, 1999). In such a context, the knowledge state of a student is represented by the set of problems in a specific knowledge domain that he is able to solve. In our approach, the learning process of the student is modelled as a function of the interaction between his knowledge state and the effect of an educational intervention (learning object). A Hidden Markov Model has been developed in order to assess the effect of learning objects on the attainment of skills required to solve problems in a given field of knowledge. A Skill Map has been used to associate with each problem the skills that are necessary and sufficient to solve it. Given the learning objects, the skills are assumed to be locally independent. The model parameters are the initial probability of the skills, the difficulty of skills attainment, the effect of the learning objects on the skills, the careless error and lucky guess probabilities of the problems. The maximum likelihood estimates of the parameters have been computed by means of a generalized expectation-maximization algorithm for Hidden Markov Models, called Baum-Welch algorithm. EMPIRICAL APPLICATION: A collection of 14 open response problems in elementary probability theory were presented to two groups of university students (N = 67) using a Computer-Based Testing (CBT) procedure. A 2 x 4 experimental design with two kinds of learning objects (effective vs. ineffective) and four assessment steps was planned. Effective learning objects were supposed to be useful to learn the skills required to solve the problems, while ineffective learning objects were supposed to be not. In our application each student was presented with 3 learning objects (effective for the students of the first group, ineffective for the students of the second group) and responded to the same 14 problems four times. Model parameters have been estimated on the data of the 67 students. RESULTS: The model enables to assess the initial knowledge state of a student and the change in this state that occurs in the different steps of the learning process. In particular, it permits to observe the effect of a learning object in facilitating or disturbing the acquisition of specific skills, and to test the significance of this effect. Estimated probability of skills acquisition is significantly greater in the group with effective learning objects than in the other group. Moreover, the careless error and lucky guess probabilities permit to check the validity of each problem in detecting the underlying skills. Teaching and learning process can be better guided by assessing the knowledge of the student, the effectiveness of the educational interventions in specific domains, and their interrelationships.