Evaluation of the quality of clinical supervision in Postgraduate Medical Education at Khyber Pakhtunkhwa province using modified "EFFECT" questionnaire /
Date of Award
2014
Document Type
Thesis
Degree Name
Master of Health Professions Education (MPHE)
Department
Educational Development
Abstract
Clinical supervision of postgraduate residents is of immense importance in medical education. Effective clinical supervision results in improvement in quality of clinical training of new generation of specialists which in turn translates into improvement in patient care. In order to ensure optimal supervision, institutions need to establish valid and reliable mechanisms for evaluation of supervisors and for providing feedback to the supervisors regarding their supervisory abilities from various sources, most importantly from the residents. Any deficiencies pointed out by residents in supervisory practices need to be rectified while strengths of supervisors should be re-. enforced. There is lack of mechanisms ensuring structured, valid and reliable feedback from residents to their supervisors in most public sector postgraduate teaching hospitals in Pakistan, especially in Khyber Palchtunkhwa (ICP) province. One method of evaluation and providing feedback to the clinical supervisors is the use of a valid and reliable instrument for this purpose. Such instruments have been reported in literature; however these must be validated in the local context. Information gained from these validated instruments should be used to provide feedback to clinical supervisors about the perceptions of residents regarding their teaching and training perforinance with the aim to improve their supervisory skills. Methods: The EFFECT questionnaire based on supervisory roles laid down in CanMEDS was selected for evaluating the quality of clinical supervision at teaching hospitals affiliated with Postgraduate Medical Institute (PGMI), Peshawar. This questionnaire has already been validated in the Netherlands. We attempted to confirm its reliability and validity in our own educational environment after adapting it to the local context. Group discussion and piloting were carried out in order to modify the original questionnaire to suit our own local educational environment. The "modified" EFFECT questionnaire was distributed among 360 residents representing all four public sector teaching hospitals across KP that are affiliated with PGMI. This instrument uses a 5-point Likert scale for rating the quality of each of the supervisory abilities with ratings ranging from 1= very poor to 5 = excellent. These questionnaires were filled voluntarily and anonymously after signing an informed consent. Content validity of the modified EFFECT questionnaire was confirmed during a group discussion with residents and supervisors followed by piloting of the questionnaire. Reliability of the modified questionnaire was confirmed using Cronbach's alpha, inter-item correlation and item-total correlation. Construct validity was tested using exploratory factor analysis with varimax rotation to identify the main domains and items loading onto each domain. For descriptive data, analysis of variance was used to compare means between hospital, specialty and year of training while independent t-test was used to compare means between genders. Results: Demographic data of participating residents and their perceptions on supervisory skills: Data was then collected from the study group of residents using the modified EFFECT questionnaire. Out of 360 questionnaires that were distributed, completed questionnaires were collected from 296 residents giving a response rate of 82%. Residents from year 2 to year 5 were eligible for completing the evaluation questionnaire. Participating residents were evenly distributed among the four public sector teaching hospitals of the province. Of the 296 respondents, 72% were men with nearly half of the participants (47%) in the second year of residency, 27% were third year residents, 23% were fourth year residents while 3% were in the fifth year. Specialties represented were classified into general medicine (22%), medicine and allied specialties (19%), general surgery (14%), surgery and allied specialties (28%), gynecology and obstetrics (14%) while the rest were classified as "others" (3%). Overall, 18 medical and surgical specialties were represented. Mean ratings for the test items regarding quality of clinical supervision ranged between 3.11 and 4.25. Strengths in clinical supervision in our study were identified as: role modeling for general roles, role modeling for professionalism and personal support. Weaknesses in clinical supervision in our study were identified as: inappropriate teaching methods, lack of internal assessment and lack of feedback. This indicates that the questionnaire was able to identify areas of strengths and weaknesses in supervisors. We found no differences between mean ratings according to gender, hospital of training, year of training or specialty of training. Validation of the modified EFFECT instrument: The original EFFECT questionnaire was reviewed by 41 residents, 4 medical educationists and 7 clinical supervisors for content validity. Some of the items were identified to be irrelevant or not applicable in our set-up and hence they were either deleted or modified. A modified EFFECT questionnaire was developed after review of the original questionnaire as described above. Piloting of the modified EFFECT questionnaire was done on 52 residents to further confirm its comprehensibility and reliability. Cronbach's alpha for the seven domains of EFFECT questionnaire (role modeling, task allocation, planning, feedback, teaching methods, assessment and personal support) ranged between 0.92 to 0.95. Iter-item correlation ranged from 0.56 to 0.57 while itemtotal correlation was above 0.4 (0.42-0.86). Construct validity of our modified instrument was confirmed using exploratory factor analysis. EFA identified seven domains with eigenvalues above 1. Since the seventh domain had only one item loading onto it therefore it was merged with domain 6 which seemed more appropriate for that item. All items loaded onto the respective domains with factor load of 0.4 or above except one ("provides feedback that is not insulting" which had a factor loading of 0.365). The modified EFFECT questionnaire retained much of its similarities with the original EFFECT questionnaire Conclusion: The modified EFFECT questionnaire has good reliability and was found valid in the local context of Pakistani postgraduate medical education environment. Our residents rated their supervisors high as clinicians but their teaching and assessment skills were rated low by the residents. Steps need to be taken to improve supervisory practices in medical institutions including introduction of system for evaluation of supervisors and faculty development programs for clinical supervisors.
Recommended Citation
Jameel, A. (2014). Evaluation of the quality of clinical supervision in Postgraduate Medical Education at Khyber Pakhtunkhwa province using modified "EFFECT" questionnaire / (Unpublished doctoral thesis). Aga Khan University, Karachi, Pakistan.