On the instructional sensitivity of CAD logs

• Autores: Charles Xie, Zhihui Zhang, Saeid Nourian, Amy Pallant, Siobhan Bailey
• Localización: The International journal of engineering education, ISSN-e 0949-149X, Vol. 30, no. 4, 2014, págs. 760-778
• Idioma: inglés
• Texto completo no disponible (Saber más ...)
• Resumen

  • Computer-aided design (CAD) logs provide fine-grained empirical data of student activities for assessing learning in engineeringdesign projects. However, the instructional sensitivity of CAD logs, which describes how students respond to interventions withCAD actions, has rarely been examined. For the logs to be used as reliable data sources for assessments, they must beinstructionally sensitive. This paper reports the results of our systematic research on this important topic. To guide the research, wefirst propose a theoretical framework for computer-based assessments based on signal processing. This framework viewsassessments as detecting signals from the noisy background often present in large temporal learner datasets due to manyuncontrollable factors and events in learning processes. To measure instructional sensitivity, we analyzed nearly 900 megabytes ofprocess data logged by our Energy3D CAD software as collections of time series. These time-varying data were gathered from 65high school students who solved a solar urban design challenge using Energy3D in seven class periods, with an interventionoccurring in the middle of their design projects. Our analyses of these data show that the occurrence of the design actions unrelatedto the intervention were not affected by it, whereas the occurrence of the design actions that the intervention targeted reveals acontinuum of reactions ranging from no response to strong response. From the temporal patterns of these student responses,persistent effects and temporary effects (with different decay rates) were identified. Students’ electronic notes taken during thedesign processes were used to validate their learning trajectories. These results show that an intervention occurring outside a CADtool can leave a detectable trace in the CAD logs, suggesting that the logs can be used to quantitatively determine how effective anintervention has been for each individual student during an engineering design project.