Development & validation of an assessment instrument for direct observation of laboratory skills (DOLS) for hamatology residency programme
Date of Award
2016
Document Type
Thesis
Degree Name
Master of Health Professions Education (MPHE)
Department
Educational Development
Abstract
In 2007, the College of Physicians and Surgeons Pakistan (CPSP) emphasized on a competency-based residency curriculum for Pakistan. This resulted in focusing on appropriate training and assessment of skills of the residents. As a result, workplace based assessments have been employed across many residency programs. Mini CEX and DOPS have been used and reported in both international and national literature as valid and reliable for workplace based assessment. However, no instrument was found for workplace based assessment for residency programme in pathology. This study was conducted to develop and validate an instrument for assessing the construct 'technical ability' of the pathology resident by direct observation of laboratory skills (DOLS) . The hypothesis was that as experienced residents (performing 10 or more procedures) differ from novice residents (performing 1-4 procedures) in their laboratory skills, therefore the test scores on DOLS should be different for the two groups of residents. Method After approval from Aga Khan's University's Ethics Committee, this study was conducted at the Aga Khan University (AKU), which is a tertiary care academic institute. Ten haematology residents (PGY 1-5) were enrolled in the study. Four lab tests were selected for observation. All residents had performed these laboratory tests at least once before DOLS was conducted. Each resident was individually evaluated by three faculty members in haematology laboratory simultaneously, during each encounter. Performance was rated on a six-point DOLS ranging from 0-6. Total score was computed as percentage of score (x) obtained from maximum possible score (i.e. 48). SPSS version 22 and Stata version '12 were used for data analysis. Construct validity and inter-rater reliability was computed using Spearman correlation and intra class correlation (ICC) respectively. Effect size was calculated using Cohen's d while Generalizability coefficient G was computed using ANOVA. Feedback from residents regarding the process of formative assessment was also analysed, and the results tabulated as learner satisfaction. Feasibility was determined by time spent in observation and feedback and cost implication. Cost was determined on the basis of consumables, stationery and faculty/resident and administrative staff time. Results A total of 120 encounters were observed. Overall mean score ±1SD on the 6-point DOLS was 56.74% ±12.44 (minimum was 27.08 % ; maximum was 66.67 %). On an individual item level, the score was lowest at 2.98%±1.22 in technical skills while highest in professionalism (3.93%±0.25). `Total score' and rating on 'overall performance' moderately correlated with the 'number of laboratory procedures previously performed' by the residents (r= 0.658** and 0.641"; p-value=0.01). Inter-rater reliability, measured by ICC of individual eight-items ranged between 0.47 and 0.96. Technical approach demonstrated the highest ICC of 0.96 (95% CI 0.94-0.98). Interrater reliability for total score (% of possible marks) was 0.95 (95%CI: 0.92- 0.98). Cronbach's alpha was high at 0.989. Cohen's d was 1.64 which was a large difference in practical sense. Estimated variance component was highest for residents (73.32%) and was negligible for assessor*lab test interaction (0.01). G Coefficient was 0.931 with one assessor examining five procedures. On a 7-point scale, the process was rated 5 by 53.3% residents and 6 by 30% residents showing their 'above average' satisfaction in DOLS. Mean time (±1SD) taken by three assessors for observing was 13.17±5.76 minutes and for providing feedback was 4.72±0.66 minutes. Cost of conducting DOLS was estimated at PKR 2675 or 25$ per session. Conclusion The study found that newly developed DOLS provided evidence for construct validity and reliability of scores for observing laboratory skills of residents. Both assessors and the residents reported high level of satisfaction with DOLS usage. It is anticipated that DOLS would provide a high reliability even if
Recommended Citation
Moiz, B. (2016). Development & validation of an assessment instrument for direct observation of laboratory skills (DOLS) for hamatology residency programme (Unpublished doctoral thesis). Aga Khan University, Karachi, Pakistan.