Laura J. Cohen PT, ATP; Shirley Fitzgerald, PhD; Elaine Trefler, MEd,
OTR/L, FAOTA, ATP; Michael Boninger, MD, Michael McCue, PhD, CRC
VA Human Engineering and
Research Laboratory, University of Pittsburgh
Department of Rehabilitation Science and Technology, School of Health and Rehabilitation Science, University of Pittsburgh
Abstract:
The purpose of this project was to develop a measure to quantify clinical rationale of wheelchair and seating prescription. This paper will present the development process of the clinical rationale measure and report concurrent validity test results.
A review of the literature reveals
a dearth of research related to effective means of increasing the competence
and proficiency of professionals working in the field of seating and wheeled
mobility. Prescribing wheelchairs and seating systems is typically completed by
a team consisting of the client, rehabilitation technology supplier, therapist
and physician.
The therapist usually performs the physical, functional and environmental evaluation. The scope and depth of evaluation skills of the therapist can vary widely. Varying levels of competence result in varying quality of advice and equipment prescription for consumers [6]. Many feel that targeted professional training will maximize the consumer/technology match [5]. Unfortunately, experienced and/or specially educated physical therapists (PT’s) and occupational therapists (OT’s) trained to provide seating and wheeled mobility prescription can be hard to find [5, 6]. In order to determine the most effective way to improve competency and proficiency of professionals, a tool measuring clinical competency and proficiency is required. Currently, no tool exists.
The purpose of this paper is
to describe the development of both the tool and the scoring method used to
evaluate clinical rationale fundamental to seating and wheeled mobility
prescription. The rationale clinicians use to prescribe wheelchairs and seating
systems were studied by examining the documented thought process used to create
a prescription for a case example. Concurrent validity testing of this measure
was completed.
DEVELOPMENT & EVALUATION
The development of the
Clinical Rationale Assessment of Prescription, a quantitative testing and
scoring method, was completed over two separate trials. The first trial
included a group of 20 PT and OT students about to enter the field; the second
trial involved 89 PT and OT professionals with varying amounts of experience in
seating and wheeled mobility prescription. A videotape of an assistive
technology practitioner performing a seating and wheeled mobility evaluation of
a client was shown in order to standardize information presented to subjects.
Subjects were provided with a generic wheelchair prescription form consisting
of three columns (problems, goals and recommendations). They were asked to
identify client problems, translate them into goals and write a generic
prescription/recommendation for a wheelchair and seating system. Test duration
was approximately 1 ˝ hour (45 minute video, 45 minute documentation).
A panel of 6
“expert” clinicians, with a mean of >12 years of experience, were polled to
create a list of common seating and mobility problems, goals and equipment
features. This list was used to generate a checklist-grading sheet to transfer
data from subjects’ test sheets.
Two separate “expert”
clinicians viewed the videotape independently and completed the
checklist-grading sheet. They reviewed the checklist-grading sheet, and
discussed what constituted a “correct” response. The agreed-upon response
became the “gold standard” key. The two independent scorers then completed a
practice scoring session using two tests from the first trial. They graded
these and then compared, discussed and reached consensus about the use of the
checklist-grading sheet. The subjects’ checklist-grading sheet was then
compared to the “gold standard” key of potential “correct” answers. A score for
each subject was tallied based on four areas: correct problems (CP), correct
goals (CG), correct recommendations (CR), and correct grand total (CGT)
responses. The scorers then graded a set of 20 tests from the first trial of
the study. These results were tested and found to have poor intrarater and
interrater reliability [4].
Due to the poor reliability
findings from the first trial, “expert” scorers met to discuss modifications of
the scoring sheet. The entire process of revising the checklist-grading sheet
and “gold standard” answer key was repeated for the development of version 2.
Items on the grading sheet were reorganized, clarified or omitted when
redundant. Then a second practice scoring session was completed using 5 tests
from the second trial. Following the practice scoring session, the grading
sheets were compared, discussed and consensus reached. A new “gold standard”
answer key was completed using, version 2, the revised checklist-grading sheet.
The scorers then graded a separate set of 20 tests from the second trial. These
results from trial 2 were tested and also found to have poor interrater
reliability[4].
Concurrent
Validity Testing
Evidence for concurrent validity was gathered
using a classification method. Nonparametric tests were chosen due to the small
sample size (n=20). The Spearman’s rho correlation revealed a significant
relationship (p = .000) between correct grand total scores for the two
independent scorers. Because of this significant relationship the scores were
averaged from the two scorers resulting in grand total scores used for further
analyses.
Additional Spearman’s rho analyses were performed comparing
average grand total (AVGT) to hours per
week of seating and mobility service provision, years of clinical experience,
years of seating and mobility experience and age. No significant relationships
were found. A Kruskal-Wallis test was performed to investigate the relationship
between AVGT and profession and AVGT and gender. No significant results were
found for AVGT and profession. AVGT and gender was significant (p=0.01),
however after reviewing gender with other characteristics (i.e. years of
experience, years of seating and mobility experience and profession) we
determined that the significance was most likely due to the small sample size
(N= 5 males, 15 females).
It is difficult to develop a measurement tool that will provide evidence of competency and rationale since abilities are not tangible; and must be inferred. Because ability is dependent on contextual or situational factors, it is difficult to develop a reliable and valid measurement tool that will include the factual knowledge, clinical skills and professional judgments a clinician must demonstrate to provide evidence of competency [1, 7]. In test situations, researchers are challenged to standardize patient examples and protocols in order to increase reliability of clinical grading. This practice often compromises external validity. Typically, competency tests are performed because we intend to draw conclusions about a clinician’s ability in nonstandardized contexts [1].
To determine criterion validity, studies of group differences and studies of correlates of the test under study are most frequently performed [8]. Two different types of criterion validity include concurrent and predictive validity and differ in the criteria used as the standard of judgment. The classification method of concurrent validity testing can be used when sample characteristics are known (i.e. years of experience, years of seating and mobility experience, profession, gender, etc.) Studies of group differences involve administering a new instrument to groups who are expected, in theory, to score at different levels on the instrument and to compare the obtained results with these expectations [8]. If the results concur with expectations, the evidence accumulated demonstrates the criterion validity of the new instrument [8]. Unfortunately, results did not show significant evidence, suggesting that there is poor concurrent validity for the Clinical Rationale Assessment of Prescription. Predictive validity is the evaluation of a test to determine how accurately it can be used to make a prediction using as evidence the relationship between a target test and a criterion test. Currently there is no criterion test available to measure clinical rationale for this purpose. In summary, our results did not show evidence of criterion validity.
Other limitations of this measurement system include:
decreased generalizability of this tool to other nonstandardized contexts due
to the specificity of the one case example. The negative findings of this study
indicate that this test instrument lacks concurrent validity and causes us to
question findings in our previous research [2, 3]. Future work will include the development and testing of a
different clinical rationale measure based on what we have learned and applying
different strategies towards its development.
1. Chambers CW (1999). Faculty ratings as part of a competency-based evaluation clinic grading system. Evaluation & The Health Professions, 22, 86-106.
2. Cohen, L. J., Fitzgerald, S., Trefler, E., & Boninger, M. (2001). Teaching clinical rationale for seating and wheeled mobility prescription: A randomized controlled trial of four instructional methods. In E. Trefler (Ed.), Orlando, FL: University of Pittsburgh.
3. Cohen, L. J., Fitzgerald, S., Trefler, E., & Boninger, M. (2001). Teaching seating and wheeled mobility prescription: A randomized controlled trial of four instructional methods. In R. Simpson (Ed.), Reno, NV: RESNA Press.
4. Cohen, L. J., Fitzgerald, S., Trefler, E., Boninger, M., & McCue, M. (2001). Development and Reliability Testing of a Clinical Rationale Measure of Seating and Wheeled Mobility Prescription. Submitted in RESNA Press.
5. Fifield, M. G. & Fifield, M. B. (1997). Education and training individuals involved in delivery of assistive technology devices. Technology and Disability, 6, 77-88.
6. Herman, J. H. & Lange, M. L. (1999). Seating and positioning to manage spasticity after brain injury. NeuroRehabilitation, 12, 105-117.
7. Prislin M.D., Giglio M., Lewis, E. M., Ahearn S., & Radecki S. (2001). Assessing the Acquisition of Core Clinical Skills through the Use of Serial Standardized Patient Assessments. Academic Medicine, 75, 480-483.
8. Richard M.Wolf (2001). Determining the validity of a measure. WWW [On-line]. Available: http://www-iea.fmi.uni-sofia.bg/Module5/measure.htm
ACKNOWLEDGMENTS
The VA
Center of Excellence in Wheelchair and Related Technology, F2181C. Special thanks
to Jessica Pederson, OTR/L, and the staff of the Center for Assistive
Technology at the University of Pittsburgh for their assistance on this
project.