Title

Development and pilot testing of a novel tool for evaluating practical skills in hematopathology residents in Pakistan

Document Type

Report

Department

Pathology and Laboratory Medicine

Abstract

Background: The mini-Clinical Evaluation Exercise (mini-CEX) and direct observation of procedural skills (DOPS) are reliable tools for work-based assessment of medical trainees. Tools of this type do not yet exist for evaluation of practical laboratory skills of pathology residents.
Objective: We developed and piloted a 9-item instrument for direct observation of laboratory skills (DOLS).
Methods: We used the DOLS tool with 10 hematopathology residents (PGY-1 to PGY-5) from Aga Khan University. Each resident was evaluated by 3 faculty members in the laboratory during 4 separate encounters using the DOLS instrument. We assessed construct validity, interrater reliability and G coefficient, feasibility of using DOLS, and learner satisfaction.
Results: A total of 120 encounters were observed with a mean score (±1 SD) of 56.7% (±12.44). Assessment scores moderately correlated with the number of laboratory procedures previously performed by participants (r = 0.658 and 0.641; P = .0001) and with PGY level. Interrater reliability ranged between 0.47 and 0.96. Cohen's d was 1.64. Residents accounted for a large component of estimated variance (73%), suggesting DOLS can differentiate residents' laboratory skills; variance associated with assessors was small (0.01%). Residents reported being satisfied with the tool. Mean time (±1 SD) taken for observing and feedback was 17.89 ± 5.89 minutes.
Conclusions: The new DOLS instrument could provide reliable scores for observing laboratory skills. Residents were satisfied with the tool, and rating times make the tool feasible for formative assessments.

Publication ( Name of Journal)

Journal of Graduate Medical Education

Link to Full Text

Share

COinS