Evaluating Experiential Leader Development

Leader development programs often employ experiential learning exercises. The impact of such exercises is not clear. This research investigated experiential leader development using a quasi-experimental design to analyze the differences in two consecutive US Air Force Squadron Officer School (SOS) in-residence classes. The curriculum was altered between classes by the addition of the Combat Leadership Exercise (CLX), an experiential war-gaming activity. Experiential programs regularly use mean differences between pretest and posttest measurements to represent program impact. However, research shows that participants may change the way they evaluate themselves between test administrations due to their experiences in the programs, a phenomenon known as response shift. Response shift renders results of mean differences evaluation invalid. The common means differences showed SOS had weak impact on leader development and showed no difference between the treatment class (CLX) and the comparison class (no CLX). However, structural equation modeling identified the presence of response shift within each SOS class, indicating that students had reconceptualized or recalibrated certain aspects of leadership measured before and after SOS. The implications of response shift and its measurement are discussed. An argument for changing the leader development evaluation paradigm to legitimize response shift as a program outcome is presented.