| Abstract: |
The gold-standard evaluation of an educational technology product is a randomized study comparing students randomized to use a computer-based learning platform (CBLP) to students assigned to a "business as usual" condition, such as pencil-and-paper work, and estimating average treatment effects. However, not everyone uses the same CBLP in the same way--indeed, an individual may engage with CBLP in multiple different ways over the course of a study--and the platform's effectiveness may depend on how students are using it. This paper introduces a model that serves two aims: classifying different modes of problem-solving or engagement among CBLP users, and estimating varying program effectiveness with varying usage patterns. The model uses mixed-type problem-level variables, such as time spent, the number of errors committed, and the number of hints requested, to cluster each problem attempt by each student into one of a number of categories, using a model-based, probabilistic, latent profile model. Students differ from each other based on their probabilities of working on problems in each of the identified modes. Finally, the model uses a fully latent principal stratification approach to estimate varying treatment effects as a function of those probabilities. In this paper, we describe the model and estimation in detail and illustrate application using data from two large randomized field trials, one evaluating Cognitive Tutor Algebra I, and the other evaluating ASSISTments. [For the complete proceedings, see ED675485.] |