Day 2 : Oral Presentations (Theme: Curriculum-Assessment)
Evaluation of an objective and structured method of assessment of professional skills
Location
AKU Auditorium
Start Date
27-1-2013 9:10 AM
Abstract
Background: College of Physicians and Surgeons Pakistan (CPSP) has made very pleasing progress in innovating Task Oriented Assessment of Clinical Skills (TOACS) method for the assessment of clinical skills. On the other hand the Laboratory Skills are assessed by the traditional ‘Practical Examination’ with inherent problems of subjectivity and low predictive value. Similar traditional assessment tools are still used by various Universities for degrees courses in laboratory disciplines i.e. BSc, MSc and M.Phil
Objective: To evaluate an indigenously designed Objective and Structured Method of Assessment of Laboratory Skills and Attitudes in Chemical Pathology.
Methods: Method: Examination was designed as 10 stations comprising instrument handling, quality control experiments, method validation, laboratory problem solving, data interpretation, data analysis and communication skills etc. This examination set was given, as a part of workshop, to groups of students of Chemical Pathology attending various course e.g. FCPS (Chem Path), MPhil and MSc. ‘Skip candidate technique’ was used for resetting instruments. The mock examination was followed by facilitator’s feedback. Students` perspective was recorded as feedback, individual interviews and focal group discussions (FCD). These workshops (with minor modifications in content) were conducted at PNS SHIFA Hospital Karachi (Dec 2011, n=22 Students) and Armed Forces Institute of Pathology (Feb 2012; n=18 students), Rawalpindi and is planned for Quaid-e-Azam Medical College Bahawalpur (Dec 2012). Inter-rater reliability was calculated for the open-ended questions using Cohn`s Kappa Coefficient and internal consistency was determined by Cronbach`s alpha using SPSS ver 17.0.
Results: Results: The percentage of students who strongly agreed (69%) or agreed (46%) with all seven questions was significantly higher than students who gave neutral answers (10%), disagreed (3%) or strongly disagreed (0%) with the propositions. Other students gave mixed answers. Maximum students strongly agreed to the questions of usefulness and content validity of the exam (77% and 73% respectively). Content analysis of interviews and FCD themes related to high students` satisfaction but less number of stations and shorter duration of time was their main concern. Kappa coefficient and Cronbach`s alpha for various OSCE stations were 0.72 and 0.81, respectively.
Conclusions: This objective and structured examination format designed for the assessment of psychomotor and effective domains of Chemical Pathology showed high content validity as indicated by students’ feedback, in addition to adequate inter-rater agreement and internal consistency. As members of Faculty of Pathology CPSP, the authors intend to recommend this format of examination for implementation in 2015.
Key words: Objective structured Assessment, Inter-rater reliability, Internal consistency
Evaluation of an objective and structured method of assessment of professional skills
AKU Auditorium
Background: College of Physicians and Surgeons Pakistan (CPSP) has made very pleasing progress in innovating Task Oriented Assessment of Clinical Skills (TOACS) method for the assessment of clinical skills. On the other hand the Laboratory Skills are assessed by the traditional ‘Practical Examination’ with inherent problems of subjectivity and low predictive value. Similar traditional assessment tools are still used by various Universities for degrees courses in laboratory disciplines i.e. BSc, MSc and M.Phil
Objective: To evaluate an indigenously designed Objective and Structured Method of Assessment of Laboratory Skills and Attitudes in Chemical Pathology.
Methods: Method: Examination was designed as 10 stations comprising instrument handling, quality control experiments, method validation, laboratory problem solving, data interpretation, data analysis and communication skills etc. This examination set was given, as a part of workshop, to groups of students of Chemical Pathology attending various course e.g. FCPS (Chem Path), MPhil and MSc. ‘Skip candidate technique’ was used for resetting instruments. The mock examination was followed by facilitator’s feedback. Students` perspective was recorded as feedback, individual interviews and focal group discussions (FCD). These workshops (with minor modifications in content) were conducted at PNS SHIFA Hospital Karachi (Dec 2011, n=22 Students) and Armed Forces Institute of Pathology (Feb 2012; n=18 students), Rawalpindi and is planned for Quaid-e-Azam Medical College Bahawalpur (Dec 2012). Inter-rater reliability was calculated for the open-ended questions using Cohn`s Kappa Coefficient and internal consistency was determined by Cronbach`s alpha using SPSS ver 17.0.
Results: Results: The percentage of students who strongly agreed (69%) or agreed (46%) with all seven questions was significantly higher than students who gave neutral answers (10%), disagreed (3%) or strongly disagreed (0%) with the propositions. Other students gave mixed answers. Maximum students strongly agreed to the questions of usefulness and content validity of the exam (77% and 73% respectively). Content analysis of interviews and FCD themes related to high students` satisfaction but less number of stations and shorter duration of time was their main concern. Kappa coefficient and Cronbach`s alpha for various OSCE stations were 0.72 and 0.81, respectively.
Conclusions: This objective and structured examination format designed for the assessment of psychomotor and effective domains of Chemical Pathology showed high content validity as indicated by students’ feedback, in addition to adequate inter-rater agreement and internal consistency. As members of Faculty of Pathology CPSP, the authors intend to recommend this format of examination for implementation in 2015.
Key words: Objective structured Assessment, Inter-rater reliability, Internal consistency