One important measure of reliability for psychometric instruments is that of temporal stability, often known as 'test-retest'. For an instrument to be useful, it is important for it to have a reasonable level of temporal stability that can be related to the defining measures of the constructs.

The Team Management Profile Questionnaire

As an example, consider the Margerison-McCann Team Management Profile Questionnaire which determines a person's team role preference and reports it as a location on the Team Management Wheel. The instrument measures people's responses across four work preference scales as follows:

If a respondent scored strongly as an extrovert on a particular day and then, say, two weeks later scored strongly as an introvert we may begin to question whether this instrument is measuring anything useful. That is why all instrument designers should publish their test-retest data so that users know the level of stability they can rely on.

'Test-retest' involves administering the same test to people over a time span. Intervals can vary from a short-term interval of, say, two to four weeks to longer-term ones of several months or years.

Test-retest reliability can tap sources of error external to the actual instrument. Issues such as mood swings, health, motivational factors and recording errors are just some of the aspects that can cause response variations over time. The length of time between testings is crucial here as well. Short periods will be subject to bias from 'remembering' effects and longer periods will be subject to the fact that people will change if the time duration is long enough.

It is possible that a change in scores does represent a legitimate change in the true score rather than random error. For example in the Team Management Profile Questionnaire constructs there is ample evidence that many people can move from a more introverted relationship preference at work to a more extroverted one, as they develop confidence in their ability to communicate.

Test-Retest of the Team Management Profile Questionnaire

An example of a test-retest study on the Team Management Profile Questionnaire is given in the Team Management Systems Research Manual. A sample of 100 people were asked to complete the questionnaire over a time period of between one and six years. The break down of time intervals between administrations was as follows:

The data were analysed and reported in two ways. Table 1 shows the correlation matrix between administration at time #1 and time #2.

TABLE 1. Correlation table of Administration 1 and Administration 2: Individual Scales

As can be seen all the correlations are above the acceptable level of 0.70- 0.75

Table 2 shows the stability of role preferences over one to six years reported as the percentage of respondents who changed their score across the poles of the four work preference measures. For example someone who scores E 6 at time 1 and I 5 at time two would be judged as having changes their preferences from extroversion to introversion.

TABLE 2. Stability of preferences over one to six years

As can be seen, 64% of respondents reported no change in their overall team role preference and a further 27% changed their scores across one of the work preference measures only. In other words 91% of respondents had no change or only one change in the poles of the four work preference measures.

From this, it is possible to conclude that the Team Management Profile Questionnaire gives excellent stability on an individual's work preferences, certainly more than enough for it to be used reliably in personal and team development interventions.

Table 1 also gives useful information on which of the work preferences are more likely to change over time. The data suggests that for the sample surveyed, respondents recorded a greater change on their structure-flexible scores than any of the other three. From interviews with many participants in our worldwide programs it seems that this change is likely to be encouraged by many organizations. We have many examples of people starting off their careers with a 'flexible' score on the organization work preference and then changing to be more structured as management development programs encourage them to work in a way that is planned, time sensitive and converges on desired outputs.

The QO₂^TM

Another instrument worth using as an example is the QO₂^TM - the opportunity-obstacles quotient. This questionnaire is based on a fifth scale which measures different aspects from those defined by the Team Management Profile Questionnaire. This fifth scale is probably the most difficult to use in a work context, although it is widely used in the realm of clinical psychology where it helps distinguishes between normal and abnormal behaviour. The scale is often used to identify such characteristics as depression, anxiety and neuroticism.

The QO₂^TMwas developed to provide people with useful feedback on whether they are likely to see the world through a 'half-full glass' or a 'half-empty' one. It is a workplace instrument which adapts the fifth scale so that it has good levels of face validity - so important when giving feedback to managers!

The instrument measures people's responses on an Opportunities - Obstacles scale.

At one extreme, people can focus most of their energy on Seeing Opportunities. They are always positive when new ideas are presented and will look optimistically at most situations. The downside is the energy they assign to Seeing Obstacles. They don't always see potential difficulties and this can cause them to misjudge situations.

At the other end of the scale, people may focus most of their energy on Seeing Obstacles. When working on a project, they are more likely to put a lot of effort into looking at all the things that might go wrong. When faced with potential opportunities they may well ignore them, presenting cogent arguments to support their view that the risks are too great.

The Opportunities-Obstacles scale is synthesised from five sub-scales: Optimism, Moving Towards Goals (MTG) Energy, Multi-Pathways, Fault-finding, and Psychological Time. Questions in these sub-scales load onto either the Opportunities or the Obstacles end of the QO₂^TM scale, resulting in a unidimensional scale where the two poles are negatively correlated (-0.6).

Test-Retest of the QO₂^TM

The fifth dimension is an interesting scale as it contains measures (optimism for example) which are known to be influenced by mood swings. The four scales of the Team Management Profile Questionnaire are, by comparison, much more stable. In fact one of the learning concepts used in the QO₂^TM workshops is that of MIPs – mood induction procedures - which can be used to shift a person's position up or down the QO₂^TM scale.

The interesting question is how much variability might there be on an instrument of this type? For the QO₂^TM to be of use we need to be confident that it has sufficient stability in personal and team development applications, for which it was designed.

A sample of 95 people was analysed with a time interval of re-administration ranging between one and nine months. The age range was 19-60 years with 45% men and 55% women.

The test-retest correlation coefficients were 0.73 for the Opportunities items and 0.70 for the Obstacles items. These are at the low end of the range normally accepted but perfectly adequate for a scale known to have elements subject to short-term effects. Over short periods of time there will be sufficient stability to give people useful feedback to help them understand their approach to work and what aspects may be currently impacting their work behaviour.

Copyright © Team Management Systems. All rights reserved.

Visit our E-Shop to view information on: Team Management Systems Research Manual

With a background in science, engineering, finance and organizational behavior, Dick McCann has consulted widely for organizations such as BP, Hewlett Packard and Hong Kong & Shanghai Bank. He is coauthor of Team Management: Practical New Approaches with Charles Margerison; author of How to Influence Others at Work and The Workplace Wizard: The Definitive Guide to Working with Others; and coauthor with Jan Stewart of Aesop's Management Fables and The Half-Empty Chalice. Involved in TMS operations worldwide for over 15 years, Dick is now Managing Director of TMS Australia, a Director of TMS Development International and President of Team Management Systems Inc.