A step by step methodological approach to appraise the quality of written feedback using comment analysis

Several studies have been conducted to analyze the quality of written feedback given by teachers on students’ written assignments. However, it has been observed that there is no clear existing method or tool to facilitate the analysis of teachers’ written feedback. This paper aims to document a step by step methodological approach to analyze teachers’ comments and appraise the quality of feedback on students’ written assignments. A self-initiated tool was developed from the comment analysis process which revealed the use of various formats and modes employed to provide written feedback to the students. In addition, this tool helped to determine the focus and tone of the feedback. Using a step by step approach, the tool also helped to identify clarity in the given feedback. This paper is an addition to the existing literature in the qualitative research method for in-depth analysis of teachers’ comments. This effort will not only appraise the quality of Method Article Ghazal et al.; BJESBS, 10(4): 1-10, 2015; Article no.BJESBS.17713 2 given feedback but also help in further development of a comprehensive qualitative tool to assess its quality.


INTRODUCTION
Teachers' feedback on students' academic written assignments "serves as a road map" and helps them in enhancing their subsequent learning [1]. Yet, empirical evidence shows that at times students are unable to understand and follow the teacher's written feedback [2]. To develop a better understanding of teachers' written feedback, several researchers have analyzed teachers' comments on students' scripts/ papers [3,4,5,6]. However, none of them has clearly described their methodology on comment analysis in their studies. This paper documents a step by step approach to carry out a comment analysis on the written feedback. This paper focuses only on the methodology extracted from an existing research study carried out by the researcher [7] that was used to appraise the quality of written feedback given by the teachers.

LITERATURE REVIEW
An extensive literature review was undertaken to explore the current research studies pertinent to the quality of written feedback and comment analysis. Electronic data bases (Ovid, Science Direct, and Google Scholar) were searched using key terms, "assignment feedback", "written feedback and students' perceptions at graduate level", "quality of written feedback", and "utilization of feedback and graduate students" etc. Pertinent publications from 1980 to 2014 were included in the study. Most of the literature found was from the developed countries such as the UK, Canada, and Australia etc. However, some of the research studies were conducted in south Asian countries such as China, Singapore and Hong Kong. Yet, no published study was found on comment analysis in the context of Pakistan's educational institutions.
Glover and Brown [4] using a qualitative approach explored practices of written feed-back in two British universities. The researchers analyzed the comments of teachers' feedback on students' marked papers (n=147). The comments on the papers were analyzed for the types and depth of feedback. The subcategories of types include: focusing on content; feedforward remarks for future improvement; motivating comments -praise or encouraging remarks; and demotivating comments -criticism or personal comments. They also analyzed the depth of comments into three categories. The first category consists of comments identifying students' errors with no remedial action or advice. The second category includes comments that pointed out a mistake with rectifying advice to fix the problem. Category three contains comments highlighting a mistake with a detailed explanation of what went wrong and why; followed by suggestions for improvement. The researchers calculated that most of the teachers' comments were to justify a grade and not to aid learning. The analysis revealed that the feedback lacked in clarity and focused more on correction, but without explaining the basis of its correction. The corrections were mainly focused on spellings and grammatical errors. The team of researchers thus suggested a need to restructure the assessment tool in order to achieve a real change for the improvements in students' learning.
Stern and Solomon [1] analyzed teachers' comments on 598 graded papers in the undergraduate programmes at the Illinois University, USA. The comments analysis was performed at the micro and macro levels. The analysis suggested that the teachers' comments mainly addressed corrections at the micro levelspelling, grammar, word choice, and missing words. Comments at the macro and mid-levellogical flow, conceptual clarity, and quality of ideas, were missing. Consequently, the researchers identified that the absence of feedback at the macro level may hinder students' interest to improve the quality of their written work.
Lizzio and Wilson [8] used a mixed method approach to investigate the students' perceptions about written feedback at the Griffith University, Australia. The researchers asked the students (n=57) to reflect on, and describe, the features of comments that they found helpful or unhelpful. In addition, a script analysis on students' graded papers were also carried out to assess the quality of teachers' written feedback from students' perspectives. After qualitative analysis, the team of researchers also developed a questionnaire to identify students' preference for written feedback in a larger group (n=277). The findings from qualitative and quantitative analysis revealed that students preferred unbiased, developmental, and encouraging remarks. The students perceived developmental feedback as what praised them, engaged them and provided them future directions.
Magno and Amarles [9], using a quantitative descriptive design, determined students' views (n=380) on teachers' comments for their academic writing compositions. A 30-items questionnaire was used to explore form (15 items), content (8 items), and writing style (7 items). Feedback items on form consisted of grammatical features, capitalizations, punctuations, tenses etc. Feedback items on content included logical flow, quality of thoughts or supporting ideas, enough details, and length of the paper. The feedback items on style comprised of "the use of language, persuasion, originality, and creativity" [9]. Using a confirmatory factor analysis the inter-correlations between form, content and writing style were assessed.
Significant inter-correlations were found among these three categories and based on these categories; a written evaluation framework was developed. This framework suggested teachers to formulate their feedback on the form, content and writing genre.
Hyland and Hyland [10] conducted an in-depth analysis of two teachers' comments on students' papers at the University of Hong Kong, China. The purpose of the comments analysis was to explore the teachers' expressions on praise, criticisms, and suggestions. The analysis revealed that 44% of the comments were on praise as compared to criticism (31 %), and suggestions (25%). It was observed that the praise was used not to encourage the students or appreciate the work but rather to soften the criticisms and suggestions. In addition, it was revealed that the use of hedging devices, such as; some of, a little bit, could, might and personal attributions, were also found to lessen the criticisms. However, these researchers also found that this mitigation in teachers' comments created indirectness and resulted in the loss of the essence of constructive criticisms; thus failed to communicate the areas of improvement to the students.
The literature cited above reveals that researchers have used both the qualitative and quantitative approach to analyze teachers' written feedback from various aspects. While some researchers have explored the quality of teachers' written feedback from the students' perspectives, some have analyzed the teachers' comments from the students' marked scripts/papers, and others have combined various sources of data. However, when it comes to their method of comment analysis, none of them clearly describe the process which they had utilized to conduct the comment analysis in their research papers. Therefore, this paper documents a step by step approach to develop a method of conducting comment analysis on written feedback to appraise the quality of written feedback on students' assignments.

METHODS
The comment analysis process used in this paper is based on the qualitative study, that is published somewhere else [7]. A total of 15 postgraduate students from different programmes including Masters in Nursing, Masters in Epidemiology and Biostatics, Masters in Health Policy and management and PhD in Basic Sciences at a private University in Pakistan participated in this study. The students were requested to share two of their papers; one which they found as the most effective feedback; the other with the least effective feedback (based on students' perception). Moreover, they were interviewed to comment on the characteristics of the most effective and the least effective written feedback in their marked assignments. Altogether 20 papers were taken as samples to conduct the comment analysis of teachers' feedback. The comments from scripts of assignments and students' interviews were reviewed, categorized and recorded on the selfdeveloped tool. This paper describes the step by step process of tool development that helped to analyze the teachers' comments in the above mentioned study through an inductive approach. The refined tool can be used to evaluate teachers' comments on written feedback for future studies.

Process of Teachers' Comment Analysis
The following grid highlights the five steps involved in comment analysis process.
Steps of Comment Analysis Process.
Step I Assessing the presentation of feedback Step II Creating a self-developed tool Step III Analyzing focus in the feedback Step IV Analyzing tone in the feedback Step V Examining clarity in the feedback

Step I: Assessing the presentation of feedback
First of all, the presentation of teachers' written feedback format and mode of delivery (from different selected postgraduate programmes) was broadly assessed. For this purpose, a table categorizing programmes, format of comments and mode of delivery was created. Then, each paper was reassessed and as per identified broad categories asterisks were plotted on the table to analyze the presentation of the feedback. Finally, the frequencies of these patterns were calculated and recorded (Table 1).
For the format of comments, common patterns of styles were identified and subcategorized into combination of annotation and summary, annotations only, summary notes only and standard format (rubric) categories and their frequencies were noted and recorded on a raw sheet. This initial analysis revealed that majority of the teachers used a combination of annotation and summary notes in the paper, and some teachers used the standard format. However, few teachers only used annotation or summary notes exclusively for providing their feedback.
For the mode of feedback delivery, it was noticed that teachers used a different modes to provide their feedback. Most of them used handwritten mode while some used typed or electronic. This analysis revealed that teachers were more comfortable in writing their comments on paper.

Step II: Creating a self-developed tool
To understand the nature of given feedback, an in-depth analysis of teachers' comments along with students' perspective regarding the feedback was required. Thus, the data needed to be recorded and collated on a format. For this purpose, a self-developed template was created keeping in mind two data sets obtained from teachers' written comments on students' written assignments and students' perception related to each comment recorded during their interviews.
The analysis of available data on feedback revealed that the written feedback included complete sentences, phrases, words and symbols. So these forms of comments were defined and labeled as "expressions" in the feedback. The comment that conveyed a single message to the student was defined as one "unit of expression" for example, complete sentences (You are confused about predisposing and enabling factors, before you write about these factors you should be clear) phrase (well done!), word (good), or symbol (√ Sign marks, ) [7]. The definitions of each unit of expression helped the researchers to measure its frequencies easily. To record data about expressions, a three column grid was designed on a word document. The first column referred to the serial number of the comments. The next column was reserved for teachers' comments on the students' paper. In this column, the data in the form of teachers' comments was recorded using an inductive approach. Moreover, in this column the comments were typed by the researchers as they appeared on the student paper. In order to avoid any double data entry of teachers' feedback the frequencies were counted manually and mentioned in a bracket beside the comment for example "very good (2)". The third column was reserved for students' perception related to each comment, indicating their perceptions on the teachers' comments. (Table. 2) This initial step in creating tool helped the researchers to assemble the data from each student assignment. After data entry in the self-developed tool, it was coded with a number. Same technique was applied to all 20 papers, collected for comment analysis. Table 2 shows a few data entries example from one student assignment (Table. 2).

Step III: Analyzing focus in the feedback
According to Magno and Amarles [9], a significant aspect of teachers' feedback is that it has to be focused on form, content and the writing style. At this stage, the researchers decided to use Magno and Amarles [9] framework to analyze the focus of the feedback. A deductive approach was applied to further categorize and reorganize the collated data, and to appraise the quality of written feedback in terms of its focus. Following Magno and Amarles' framework [9] the focus of teachers' comments was analyzed and categorized for form, content, and writing style. Teachers' comments highlighting grammar, punctuation, tenses, and syntax related errors were coded as form. Remarks about conceptual clarity and coherence in thoughts were categorized as a focus on content whereas teachers' feedback on genre like literature synthesis, argument building, and reflective writing were coded as writing style.
Taking Magno and Armless's framework [9] as model, the collated data in the initial tool were also needed to be categorized for the focus of feedback. For this purpose, a new column was inserted (before the serial number column in the initial tool) and was divided into three subsections namely; form, content and style (Table. 3). Here, each comment was critically analyzed by referencing and matching the definitions (form, content and style) given in the framework. At this step, all the comments were reorganized and classified into their respective sub-sections. (Table. 3).
In order to find out where the focus of the feedback is skewed in the available data, it was important to calculate their frequencies of the expressions in the feedback in each sub-section including form, content, and style (Table. 3). Hence, it required refinement in the initial tool, for this purpose, the column of serial number was changed into the frequency of the comments.
During this analysis, it was identified that majority of the teachers were highlighting or correcting the in text referencing within the papers and the bibliography i.e. use of American Psychology Association style (APA) and Vancouver style. Therefore, it was decided to include referencing style in the writing style of the (focus) feedback which could be an addition to Magno and Amarles [9] definition of writing style. "v. good-well articulated account of the need assessment" Appraisal 2 √ sign marks (12)* "It indicated that content is ok but not something outstanding." 3 'Very good' (2)* "The teacher likes my points." 4 Underlined content (36)* "This showed that something important has been highlighted by the teacher. "Having no comments makes it difficult to understand the feedback." 5 "Well said-end of conclusion" "Faculty liked my expression in conclusion" 6 "Why do you think so? Is there any reference?" "She liked my observation but helped me to reflect why this happen or I have any reference to support my observation."

*Number of times a comment appeared on the paper
The findings revealed that focus of the feedback was more on content than form and writing style.

Step IV: Analyzing the tone of feedback
The tone is another imperative aspect in the given feedback that could have positive as well as negative impact on feedback utilization by the students. Hyland and Hyland's framework [10] focused tone of the feedback which includes praise, criticism and suggestions. In order to appraise the tone of the feedback Hyland and Hyland's model [10] was chosen. Teachers' comments that appreciated student work were categorized as praise, while comments indicating "expression of dissatisfaction or negative comment" [10] were marked as criticism.
Comments that assisted students' corrective action, for example "you need to improve the paper's logical flow" were defined as suggestions. In line with the Hyland and Hyland's framework [10] the praise, criticism and suggestions were found in this analysis also.
In addition to Hyland and Hyland [10], the current analysis revealed that symbols in the feedback were also convening praise or criticism to the students. Thus, the analysis of symbols was important in terms of the tone of the feedback. According to Ghazal, Gul, Hanzala, Jessop and Tharani [7], 40% of the feedback was expressed through symbols while 60% of the feedback was stated in words, phrases, or complete sentences. The symbols were expressed in the form of a question mark (?), tick mark (√), happy () or sad () face, a cross (×) or lines crossing text (-, ⁄, =), used to delete certain content in the paper.
In order to appraise the tone of the feedback, the designed tool needed further refinement, therefore, a new section to note comments related to tone was added which was further divided into three sub-sections on the left side of the grid (Table. 4). Each comment was read and matched with its definition of praise, criticisms and suggestions and then marked with an asterisk* in the designated column to calculate the frequencies. In addition, symbols referring to praise or criticism were marked in the respective sub sections. This analysis could be an addition to the definitions of praise and criticisms provided by Hyland and Hyland [10].
Moreover, while analyzing the tone of feedback, some of the teachers' comments were found open-ended which could be an attempt to stimulate students' critical thinking and to require conceptual clarity. It was observed that these comments did not match with the definition of the tone suggested by Hyland and Hyland [10].This category differed in tone in two ways compared to the definition of suggestion as the comments were found in the form of a question (Why & How) which could not be matched with the definition of suggestions given in Hyland and Hyland [10]. Moreover, the nature and purpose of the comments required conceptual clarity in students' writing. Therefore, this new category was identified and labeled as reflection [7] and to analyse this, a separate column was added to the grid as reflection.

Step V: Analyzing the clarity of the feedback
It was observed that several types of symbols such as; sad and happy faces, brackets, circles and line crossings/ deleting texts (40% of the total data) were also used to provide feedback. Though these symbols were conveying feedback messages to the students, they were without any explanations and thus lacked clarity. As these symbols were identified as one of the ways of expressions and most of the time students perceived these symbols as a mark of criticism; therefore, they were labeled, analyzed and categorized separately. For this reason, another column "others" was added next to the tone of feedback column [7]. This step facilitated in eliminating the overlaps within the unit of expressions. However, in terms of clarity in the feedback it was noted that feedback in the form of sentences and phrases was clearer to the students in contrast to single words and symbols. Hence, it was decided to merge these sub-categories into two: sentences & phrases; words & symbols, based on the clarity in their expressions. (Table 5 above shows a combined form of data analysis in terms of numbers of unit of expressions in each sub-category).

DISCUSSION
As mentioned earlier, the aim of this paper was to document a process of step by step approach to comment analysis; thus, the discussion will encompass the key outcomes of the analysis process. The key outcomes which resulted from this analysis included a systematic approach to analyze teachers' comments and appraised the quality of teachers' comments in terms of its focus and tone. This approach can be helpful to the teachers and future researchers in assessing the quality of written feedback in terms of focus tone and clarity in the feedback.
In previous literature there are only two studies that have provided framework to conduct comment analysis [9,10]. However, Magno and Armless [9] only concentrated on the focus of teachers' feedback whereas Hyland and Hyland [10] only focused on the tone in teachers' feedback. This paper combined both the frameworks [9,10] to appraise the quality of written feedback. In addition, this analysis contributed in identifying two other important aspects of the written feedback that are use of reflective question and symbols by the teachers that were not addressed in earlier studies. Although not addressed by Hyland and Hyland [10], the step by step comment analysis process showed that reflective questions in the feedback created two way communications among students and teachers which was found to be engaging students more for their subsequent learning. However, the current analysis revealed that reflective questioning was marginally used in giving feedback. It was also identified that students' appreciated reflective questioning feedback and found it more effective. They also valued it because it added to both their conceptual clarity and enhanced comprehensiveness of the written content [7].
This finding suggests that teachers should be trained to use reflective questions more liberally in giving feedback. For example one of the comments, critiquing student's lack of understanding of different types of factors responsible for a health care behavior, a teacher directly wrote, "You are confused about predisposing and enabling factors, before you write about these factors you should be clear". This comment can be an example of a direct critique on student's work, because if student knew this difference s/he would have not written it in that way. This direct critique was also disliked by the students [7]. However, if it is changed into a reflective question then one of the alternative comments could be, "There is a fine difference between predisposing and enabling factors. Can you read about the difference and then reflect and review this piece of writing?" This shows that the teachers would require a conscious effort to modify their comments within the suggestion or reflection frameworks rather than critique so that they have a positive impact on students. Thus, this comment analysis added reflective questions as a part of tone of the feedback, contributing to Hyland and Hyland [10].
Another none-existing written feedback component in literature was the teachers' use of symbols in their feedback. It was found to be an important aspect of teachers' feedback and indicated that if teachers use it excessively, it could be perceived as criticism or an ineffective and ambiguous feedback by the students, hindering their learning. Thus, the teachers need to be thoughtful about using symbols. Since, symbols are used in written feedback by the teachers; therefore, this needs to be added in the framework to assess the quality of feedback [10].

Strengths and Limitations
The current study contributes to the existing body of knowledge of comment analysis process by broadening the horizon for viewing the process (mentioned in the previous studies) in terms of content and tone of the feedback. However, as this study was carried out on 20 students' assignments only, it has its constraint because of small sample size. A larger sample for comment analysis process would have definitely strengthened it.

RECOMMENDATIONS, FUTURE IMPLICATIONS AND CONCLUSION
The recommendations below have implications for the entire learning community -teachers, students, and the institutions as they are based on the data analysis, suggestions from the study participants, researcher's view point, and suggestions in the existing literature. Considering the value of reflective questioning, teachers should make use of it to enhance students' conceptual clarity while softening the criticality. This could be a useful strategy to shift the responsibility of learning and action from teachers to students, making students accountable for their own learning. Moreover, symbols should be used cautiously at the graduate level, as they tend to have a distressing effect on the receptors and could also be difficult to interpret at times. The current analysis combined the focus [9], tone [10] and clarity of the feedback in one tool, to assess the quality of written feedback. This initial effort will not only help the teachers to devise their written feedback comprehensively but also help them to further develop the existing qualitative tool to appraise the quality of written feedback.