MEASURING THE CLINICAL UTILITY OF AN ASSESSMENT:

Measuring the Clinical Utility of an Assessment:

The Example of the Canadian Occupational Performance Measure

Jan Miller Polgar, PhD, O.T. Reg. (Ont.) and Ingrid Barlow, M.Sc. O.T.

INTRODUCTION

Rehabilitation clinicians and researchers use assessments for three main purposes: to identify and describe key issues to be addressed in rehabilitation, to evaluate the outcome of intervention and to predict future function (Miller Polgar, 1998). Further, in the current health care environment demands for evidence-based practice come from employers and funding agencies. Identification and appraisal of appropriate assessments for use in clinical practice is therefore, an important part of meeting these purposes and demonstrating the evidence base of our practice. An abundance of literature is available to aid critique of instrument construction and psychometric properties of reliability and validity (e.g. APA, AERE, NCME, 1999; Nunnally & Bernstein, 1995). However, even when an assessment has solid psychometric properties, if it is not feasible to use in a clinical setting or does not provide useful clinical information, then it is of little value to a clinician. These latter aspects refer to the clinical utility of an assessment, an aspect of appraisal that has received little attention in the literature.

This paper will relate clinical utility to other psychometric properties and test construction. It will define the concept and provide a framework with which a clinician can appraise the clinical utility of an assessment. Clinical utility of the Canadian Occupational Performance Measure (Law, Baptiste, Carswell, McColl, Polatajko, & Pollock, 1998) has been examined as part of the authors’ research and will be presented as part of the workshop.

THE RELATIONSHIP OF CLINICAL UTILITY TO OTHER PSYCHOMETRIC PROPERTIES

Psychometric properties are those aspects of test development and evaluation that are essential to ensure that an assessment is appropriate for a particular client group, provides reliable and valid information, and is administered and interpreted in a consistent and ethical manner (APA, AERA, NCME, 1999; Nunnally & Bernstein, 1995). Two psychometric properties that are best known are reliability and validity. Reliability is a measure of the stability and reproducibility of a test score (ref). It provides an estimate of the true score and sources of error that contribute to that score. Validity is an accumulation of evidence that supports the "appropriateness, meaningfulness, and usefulness of inferences and actions that are based on test scores" (Messick, 1988). Primary considerations here are that the assessment adequately represents the relevant domain of concern and that either the items or the method of response do not require information or actions that are irrelevant to the domain of concern. There must be evidence for validity of the use of the test for the intended population and purpose. An outcome measure must be sensitive to change where that has occurred.

Test construction is another important aspect to critique. Here consideration must be made of item selection, response format, normative data derivation, and population for which the assessment is intended. Once the clinician is satisfied that an assessment is sound from a psychometric perspective, and then an analysis of clinical utility should be undertaken.

CLINICAL UTILITY: DEFINED

Simply stated, clinical utility refers to ease and efficiency of use of an assessment, and the relevance and meaningfulness, clinically, of the information that it provides (Law, King, & Russell, 2001; Letts et al., 1999). Realistically, if an assessment does not meet these criteria, it will not be incorporated into practice. We have subcategorized this concept into the following:

Availability and ease of use
Administration time
"Learnability" and clinician’s qualifications
Format
Scoring and information derived
Meaningful and relevant information obtained

A FRAMEWORK FOR EVALUATING CLINICAL UTILITY

For each of the areas above, we present a series of questions that the clinician can apply to an assessment when critiquing an assessment.

Availability and Ease of Use:

Is the assessment readily available and reasonably priced?
Are the instructions provided to the clinician on how to administer the evaluation complete and clear?
Are the instructions given to the client to elicit a response to assessment items complete and clear?
How are client responses recorded?
What materials are required to administer the evaluation? Are they provided with the evaluation?
Are there particular space or location requirements? Are these requirements located within the clinical facility? Can the evaluation be administered in the client’s home?

Administration Time:

How much time does it take to administer the evaluation?
Can the evaluation be conducted over more than one session without jeopardizing reliability or validity?

"Learnability" and Clinician’s Qualifications:

Can administration of the assessment be learned in a reasonable amount of time?

Is a training manual readily available?

Is special training required to administer the assessment? I.e. is a certification course required in order to administer the assessment or are particular credentials e.g. trained as a psychologist with a PhD, required?

Is particular knowledge required to understand the theoretical concepts underlying the design of the assessment?

Does the clinician require knowledge of testing theory to administer or interpret the results of the assessment?

Format:

Is the format of the assessment acceptable to the client? E.g. does it ask questions that might be uncomfortable to the client? Do the procedures cause the client discomfort?
Does the format require active participation of the client?
Is the format of the assessment acceptable to the clinician? E.g. does it use invasive procedures?
Is the client capable of forming a response in the manner required by the assessment? E.g. is the client able to give a motor response when one is required?

Scoring and Information Derived:

Is the assessment easy to score?
Is computer-based scoring or interpretation available?
Can the clinician interpret the scores or must they be sent to another source for interpretation?
What knowledge or background is needed to interpret the results?

Meaningful and Relevant information obtained:

Is the assessment norm-referenced, criterion referenced or neither?

What information results from interpretation of the results of this assessment?

How is that information useful within the clinical setting?

Is the assessment responsive to clinically meaningful change?

Do floor or ceiling effects compromise the utility of the assessment?

CLINICAL UTILITY OF THE CANADIAN OCCUPATIONAL PERFORMANCE MEASURE:

Information from the Test Manual:

The test manual indicates that the COPM takes between 30 and 40 minutes to administer (Law et al., 1998). It is easily accessible and economic. Information is recorded directly on the form. Both a detailed manual and training video are available from the publishers. A majority of clinicians report that it provides meaningful and useful information and that it helps frame practice within a client-centred model. The semi-structured interview format is acceptable to both clients and clinicians (Toomey, Nicholson & Carswell, 1995). The measure is flexible since allowances and modifications can be made to fit a specific clinical situation. Some concerns with the measure include difficulty of some clients, particularly those early in the rehabilitation process or those with limited insight, to identify occupational performance issues. The rating scale has also been problematic with some clients and there can be a tendency not to use its full range (Law et al.). The information gained provides specific information regarding occupational performance issues and indication of the client’s perception of importance and satisfaction with occupational performance as well as any change that has occurred (Toomey, Nicholson & Carswell, 1995).

References

American Educational Research Association, American Psychological Association, National Council on Measurement in Education. Standards for Educational and Psychological Testing. Washington: AERA, 1999.
Gowland C, King G, King S, Law M, Letts L, MacKinnon E, Rosenbaum P, Russell D. Review of Selected Measures in Neurodevelopmental Rehabilitation. (Neurodevelopmental Clinical Research Unit Rep. 91-2). Hamilton, ON: Neurodevelopmental Clinical Research Unit, 1991.
Law M, Baptiste S, Carswell A, McColl M, Polatajko H, Pollock N. Canadian Occupational Performance Measure, 3^rd Ed. Toronto: Canadian Association of Occupational Therapists, 1998.
Law M, King G, Russell D. Guiding decisions about measuring outcomes in occupational therapy. In: Measuring Occupational Performance. Thorofare, NJ: Slack Inc, 2001; 31-40.
Letts L, Law M, Pollock N, Stewart D, Westmorland M, Philpot A, Bosch J. A Programme Evaluation Workbook for Occupational Therapists: An Evidence-Based Practice Tool. Ottawa, ON: Canadian Association of Occupational Therapists, 1999.
Messick S. The once and future issues of validity: Assessing the meaning and consequences of measurement. In: Test Validity. Hillsdale, NJ: Lawrence Erlbaum Associates, 1988; 33-45.
Miller Polgar J. Critiquing assessments. In Willard and Spackman’s Occupational Therapy 9^th Ed. Philadelphia: J.B. Lippincott, 1998; 169-184.
Nunnally J, Bernstein I. Psychometric Theory 3^rd Ed. Toronto: McGraw-Hill, 1995.
Stein F, Cutler S. Clinical Research in Occupational Therapy. San Diego: Singular, 2000.
Toomey M, Nicholson D. Carswell A. The clinical utility of the Canadian Occupational Performance Measure. Canadian Journal of Occupational Therapy 1995, 62:242-249.

Notes: