MMWCON

Guidelines for the Design, Implementation, and Analysis of Game Telemetry

Title

Guidelines for the Design, Implementation, and Analysis of Game Telemetry

Presenter

Abstract

This presentation will address the question: What are appropriate methods of conceptualizing in-game behavior to support (a) data capture in the game itself, (b) the development of measures from the data, and (c) the mining, analyses, and interpretation of student performance in the game?

Background and Context

The need for a systematic approach to the design of “learning telemetry” exists because of three ongoing trends: (a) increasing acceptance of games in education and training and other forms of instrumentable systems; (b) the desire to harvest data to make inferences about student learning in situ [vs. through external assessments]; and (c) the lack of a coherent framework that conceptualizes the process of designing telemetry in online systems—what should be captured, how it should be captured, and how it should be formatted. Currently, ad hoc approaches often result in inadequate capture of the relevant information, resulting in unusable data.

Design and Analysis Guidelines

Drawing on principles from the long tradition of behavioral observation and assessment, a set of design guidelines will be presented based on over 15 years of R&D (at CRESST) on learning games, instructional simulations, and online learning. The key elements are:
1. Conceive game telemetry as behavioral observations, with careful attention paid to the validity of observations—a clear definition of what to measure, why to measure, and how to measure.
2. Conceive game interactions as the unit of observation
3. Adopt a learning-first perspective that emphasizes the logging of game interactions that directly relate to cognitive processes believed to reflect engagement with the to-be-learned content.

A case study will illustrate design guidelines using examples drawing from our work on math games on the topics of (a) specifying what to measure, (b) formatting the telemetry data package, and (c) analyses enabled by the data package.

Details

Type

Presentation

Technical Level

Basic

Area
  • Gamification
Topics
  • CSS Compilers
  • Game Design & Mechanics
  • Analytics and Data Analysis