Development of the Toolbox

All questionnaires of the FIRE Toolbox were systematically developed in the context of several master theses. The development process was divided into several sections that are illustrated by the image.

 

The image shows the development steps of all FIRE questionnaires.
Development of the FIRE questionnaires (each color corresponds to a master thesis)
© OWMs

 

Interviews

First, interviews were conducted with instructors and participants of training events. These focused on the question of what aspects make up a good training event or a well-set examination.

Item Generation

Based on the interviews and relevant literature, items (= questionnaire questions) were derived that covered all the aspects mentioned.

Content Check

The derived items were presented to instructors and participants of training events. They evaluated their comprehensibility as well as the relevance and completeness of the mentioned aspects.

Statistical Check

In the statistical check, the items were presented to a large number of course participants (on average 316 persons). They completed the items in relation to their current course. Exploratory factor analyses could be calculated on the basis of the data. These provided information on how many and which superordinate dimensions the items were based on in each case (e.g. "instructor behavior", "structure", ...). For the final version of each questionnaire, items were selected that represented their respective dimension well and also had positive statistical measurement properties (e.g., sufficient variability in order to differentiate well).

Adaptation of FIRE items

The FIRE questionnaire serves as the basis for the FIRE-ST and the FIRE-CU. The already well-tested FIRE items were largely adopted, adapted in wording and supplemented in content.

Validation

Validation studies with an average sample size of 335 participants showed the following:

  • The assumed dimensionality of the FIRE questionnaires was confirmed (using confirmatory factor analysis).
  • The FIRE questionnaires correlate with established evaluation questionnaires with similar content, e.g., from the university context.
  • The results of the FIRE questionnaires are largely independent of factors such as the current mood of the participants or their age.
  • The results of the FIRE questionnaires are significantly related to important criterion variables, e.g., the satisfaction of the participants or their assessment of the evaluated event or examination.