Development and pilot testing of a novel tool for evaluating practical skills in hematopathology residents in Pakistan

Document Type

Report

Department

Pathology and Laboratory Medicine

Abstract

Background: The mini-Clinical Evaluation Exercise (mini-CEX) and direct observation of procedural skills (DOPS) are reliable tools for work-based assessment of medical trainees. Tools of this type do not yet exist for evaluation of practical laboratory skills of pathology residents.
Objective: We developed and piloted a 9-item instrument for direct observation of laboratory skills (DOLS).
Methods: We used the DOLS tool with 10 hematopathology residents (PGY-1 to PGY-5) from Aga Khan University. Each resident was evaluated by 3 faculty members in the laboratory during 4 separate encounters using the DOLS instrument. We assessed construct validity, interrater reliability and G coefficient, feasibility of using DOLS, and learner satisfaction.
Results: A total of 120 encounters were observed with a mean score (±1 SD) of 56.7% (±12.44). Assessment scores moderately correlated with the number of laboratory procedures previously performed by participants (r = 0.658 and 0.641; P = .0001) and with PGY level. Interrater reliability ranged between 0.47 and 0.96. Cohen's d was 1.64. Residents accounted for a large component of estimated variance (73%), suggesting DOLS can differentiate residents' laboratory skills; variance associated with assessors was small (0.01%). Residents reported being satisfied with the tool. Mean time (±1 SD) taken for observing and feedback was 17.89 ± 5.89 minutes.
Conclusions: The new DOLS instrument could provide reliable scores for observing laboratory skills. Residents were satisfied with the tool, and rating times make the tool feasible for formative assessments.

Publication (Name of Journal)

Journal of Graduate Medical Education

Link to Full Text

Share

COinS