Working Inside the Black Box


Embed or link this publication

Popular Pages

p. 1

phi delta kappan vol 86 #1 pp 8-21 2004 working inside the black box assessment for learning in the classroom in their widely read article inside the black box mr black and mr wiliam demonstrated that improving formative assessment raises student achievement now they and their colleagues report on a follow-up project that has helped teachers change their practice and students change their behavior so that everyone shares responsibility for the students learning by paul black christine harrison clare lee bethan marshall and dylan wiliam i n 1998 inside the black box the predecessor of this article appeared in this journal.1 since then we have learned a great deal about the practical steps needed to meet the purpose expressed in the article s subtitle raising standards through classroom assessment in the first part of inside the black box we set out to answer three questions the first was is there evidence that improving formative assessment raises standards the answer was an unequivocal yes a conclusion based on a review of evidence published in over 250 articles by researchers from several countries.2 few initiatives in education have had such a strong body of evidence to support a claim to raise standards this positive answer led naturally to the second ques tion is there evidence that there is room for improvement here again the available evidence gave a clear and positive answer presenting a detailed picture that identified three main problems 1 the assessment methods that teachers use are not effective in promoting good learning 2 grading practices tend to emphasize competition rather than personal improvement and 3 assessment feedback often has a negative impact particularly on low-achieving students who are led to believe that they lack ability and so are not able to learn paul black is a professor emeritus department of education and professional studies king s college london where christine harrison is a lecturer in science education clare lee is a teacher advisor for the warwickshire county council bethan marshall is a lecturer in english education department of education and professional studies school of education king s college london dylan wiliam is director of the learning and teaching research center educational testing service princeton n.j they wish to thank the medway and oxfordshire local education authorities and the schools directly involved in the project described here especially the teachers who shouldered the greatest burdens the work on which this article is based was supported by grants from the nuffield foundation and from the u.s national science foundation as part of the stanford university classroom assessment project to improve teaching and learning capital but the opinions expressed are those of the authors © 2004 dylan wiliam september 2004 9


p. 2

however for the third question is there evidence about how to improve formative assessment the answer was less clear while the evidence provided many ideas for improvement it lacked the detail that would enable teachers to implement those ideas in their classrooms we argued that teachers needed a variety of living examples of implementation been extended more recently to involve teachers of english in the same schools and teachers of other subjects in other schools spreading the word throughout the development of the project we have responded to numerous invitations to talk to other groups of teachers and advisers indeed over five years we have made more than 400 such contributions these have ranged across all subjects and across both primary and secondary phases in addition there has been sustained work with some primary schools all of this gives us confidence that our general findings will be of value to all although some important details may differ for different age groups and subjects furthermore a group at stanford university obtained funding from the national science foundation to set up a similar development project in collaboration with king s in schools in california extension of our own work has been made possible by this funding and we also acknowledge support from individuals in several government agencies who sat on the project s steering group offered advice and guidance and helped ensure that assessment for learning see assessment for learning below is a central theme in education policy in england and scotland the learning gains from our review of the international research literature the journey learning with teachers since 1998 we have planned and implemented several programs in which groups of teachers in england have been supported in developing innovative practices in their classrooms drawing on the ideas in the original article while this effort has amply confirmed the original proposals it has also added a wealth of new findings that are both practical and authentic thus we are now confident that we can set out sound advice for the improvement of classroom assessment the kmofap project to carry out the exploratory work that was called for we needed to collaborate with a group of teachers willing to take on the risks and extra work involved and we needed to secure support from their schools and districts funding for the project was provided through the generosity of the nuffield foundation and we were fortunate to find two school districts oxfordshire and medway both in southern england whose supervisory staff members understood the issues and were willing to work with us each district selected three secondary schools oxfordshire chose three coeducational schools and medway chose one coeducational school one boys school and one girls school each school selected two science teachers and two mathematics teachers we discussed the plans with the principal of each school and then we called the first meeting of the 24 teachers so in january 1999 the king s-medway-oxfordshire formative assessment project kmofap was born full details of the project can be found in our book assessment for learning putting it into practice.3 for the present purpose it is the outcomes that are important the findings presented here are based on the observations and records of visits to classrooms by the king s college team records of meetings of the whole group of teachers interviews with and writing by the teachers themselves and a few discussions with student groups initially we worked with science and mathematics teachers but the work has 10 phi delta kappan assessment for learning s assessment for learning is any assessment for which the first priority in its design and practice is to serve the purpose of promoting students learning it thus differs from assessment designed primarily to serve the purposes of accountability or of ranking or of certifying competence an assessment activity can help learning if it provides information that teachers and their students can use as feedback in assessing themselves and one another and in modifying the teaching and learning activities in which they are engaged such assessment becomes formative assessment when the evidence is actually used to adapt the teaching work to meet learning needs.


p. 3

we were convinced that enhanced formative assessment would produce gains in student achievement even when measured in such narrow terms as scores on state-mandated tests at the outset we were clear that it was important to have some indication of the kinds of gains that could be achieved in real classrooms and over an extended period of time since each teacher in the project was free to choose the class that would work on these ideas we discussed with each teacher what data were available within the school and we set up a mini-experiment for each teacher the average effect size was around 0.3 standard deviations such improvements produced across a school would raise a school in the lower quartile of the national performance tables to well above average thus it is clear that far from having to choose between teaching well and getting good test scores teachers can actually improve their students results by working with the ideas we present here how change can happen we set out our main findings about classroom work under four headings questioning feedback through grading peer and self-assessment and the formative use of summative tests most of the quotations in the following pages are taken directly from pieces written by the teachers the names of the teachers and of the schools are pseudonyms in keeping with our policy of guaranteeing anonymity questioning many teachers do not plan and conduct classroom dialogue in ways that might help students to learn research has shown that after asking a question many teachers wait less than one second and then if no answer is forthcoming ask another question or answer the question themselves.4 a consequence of such short wait time is that the only questions that work are those that can be answered quickly without thought that is questions calling for memorized facts consequently the dialogue is at a superficial level as one teacher put it i d become dissatisfied with the closed q a style that my unthinking teaching had fallen into and i would frequently be lazy in my acceptance of right answers and sometimes even tacit complicity with a class to make sure none of us had to work too hard they and i knew that if the q a wasn t going smoothly i d change the question answer it myself or only seek answers from the brighter students there must have been times still are where an outside observer would see my lessons as a small discussion group surrounded by many sleepy onlookers james two bishops school many teachers do not plan and conduct classroom dialogue in ways that might help students to learn each teacher decided what was to be the output measure for his or her class for grade-10 classes this was generally the grade achieved on the national school-leaving examination taken when students are 16 the general certificate of secondary education or gcse for grade-8 classes it was generally the score or level achieved on the national tests administered to all 14-year-olds for other classes a variety of measures were used including end-of-moduletest scores and marks on the school s end-of-year examinations for each project class the teacher identified a comparison class in some cases this was a parallel class taught by the same teacher in previous years and in one case in the same year in other cases we used a parallel class taught by a different teacher or failing that a nonparallel class taught by the same or a different teacher when the project and the control classes were not strictly parallel we controlled for possible differences in prior achievement by the use of input measures such as school test scores from the previous year or other measures of aptitude this approach meant that the size of the improvement was measured differently for each teacher for example a grade-10 project class might outperform the comparison class by half a gcse grade but another teacher s grade8 project class might outscore its control class by 7 on an end-of-year exam to enable us to aggregate the results we adopted the common measuring stick of the standardized effect size calculated by taking the difference between the scores of the experimental and control groups and then dividing this number by the standard deviation a measure of the spread in the scores of the groups for the 19 teachers on whom we had complete data the key to changing such a situation is to allow longer wait time but many teachers find it hard to do this for it requires them to break their established habits once they change the expectations of their students are challenged increasing waiting time after asking questions proved difficult to start with due to my habitual desire to add something almost immediately after asking the september 2004 11


p. 4

original question the pause after asking the question was sometimes painful it felt unnatural to have such a seemingly dead period but i persevered given more thinking time students seemed to realize that a more thoughtful answer was required now after many months of changing my style of questioning i have noticed that most students will give an answer and an explanation where necessary without additional prompting derek century island school exchange ideas and experiences about good questions is very valuable the questions themselves then become a more significant part of teaching with attention focused on how they can be constructed and used to explore and then develop students learning here s one teacher s thinking on the matter i chose a year-8 middle-band group and really started to think about the type of questions i was asking were they just instant one-word answers what were they testing knowledge or understanding was i giving the class enough time to answer the question was i quickly accepting the correct answer was i asking the girl to explain her answer how was i dealing with a wrong answer when i really stopped to think i realized that i could make a very large difference to the girls learning by using all their answers to govern the pace and content of the lesson gwen waterford school one teacher summarized the overall effects of her efforts to improve the use of question-and-answer dialogue in the classroom as follows questioning · my whole teaching style has become more interactive instead of showing how to find solutions a question is asked and pupils are given time to explore answers together my year 8 [grade 7 target class is now well-used to this way of working i find myself using this method more and more with other groups no hands · unless specifically asked pupils know not to put their hands up if they know the answer to a question all pupils are expected to be able to answer at any time even if it is an i don t know supportive climate · pupils are comfortable with giving a wrong answer they know that these can be as useful as correct ones they are happy for other pupils to help explore their wrong answers further nancy riverside school effective questioning is also an important aspect of the impromptu interventions teachers conduct once the students are engaged in an activity asking simple questions such as why do you think that or how might you express that can become part of the interactive dynamic of the classroom and can provide an invaluable opportunity to extend students thinking through immediate feedback on their work overall the main suggestions for action that have emerged from the teachers experience are · more effort has to be spent in framing questions that increasing the wait time can help more students become involved in discussions and increase the length of their replies another way to broaden participation is to ask students to brainstorm ideas perhaps in pairs for two to three minutes before the teacher asks for contributions overall a consequence of such changes is that teachers learn more about the students prior knowledge and about any gaps and misconceptions in that knowledge so that teachers next moves can better address the learners real needs to exploit such changes means moving away from the routine of limited factual questions and refocusing attention on the quality and the different functions of classroom questions consider for example the use of a big question an open question or a problem-solving task that can set the scene for a lesson and evoke broad discussion or prompt focused small-group discussions however if this strategy is to be productive both the responses that the task might generate and the ways of following up on these responses have to be anticipated collaboration between teachers to 12 phi delta kappan


p. 5

are worth asking that is questions that explore issues that are critical to the development of students understanding · wait time has to be increased to several seconds in order to give students time to think and everyone should be expected to have an answer and to contribute to the discussion then all answers right or wrong can be used to develop understanding the aim is thoughtful improvement rather than getting it right the first time · follow-up activities have to be rich in that they create opportunities to extend students understanding put simply the only point of asking questions is to raise issues about which a teacher needs information or about which the students need to think when such changes have been made experience demonstrates that students become more active participants and come to realize that learning may depend less on their capacity to spot the right answer and more on their readiness to express and discuss their own understanding the teachers also shift in their role from presenters of content to leaders of an exploration and development of ideas in which all students are involved feedback through grading when giving students feedback on both oral and written work it is the nature rather than the amount of commentary that is critical research experiments have established that while student learning can be advanced by feedback through comments the giving of numerical scores or grades has a negative effect in that students ignore comments when marks are also given.5 these results often surprise teachers but those who have abandoned the giving of marks discover that their experience confirms the findings students do engage more productively in improving their work many teachers will be concerned about the effect of returning students work with comments but no scores or grades there may be conflicts with school policy my marking has developed from comments with targets and grades which is the school policy to comments and targets only pupils do work on targets and corrections more productively if no grades are given clare [lee observed on several occasions how little time pupils spent reading my comments if there were grades given as well my routine is now in my target class i to not give grades only comments ii to give comments that highlight what has been done well and what needs further work and iii to give the minimum follow-up work expected to be completed next time i mark the books nancy riverside school to be unjustified and neither parents nor school inspectors have reacted adversely indeed the provision of comments to students helps parents to focus on the learning issues rather than on trying to interpret a score or grade we now believe that the effort that many teachers devote to grading homework may be misdirected a numerical score or a grade does not tell students how to improve their work so an opportunity to enhance their learning is lost a commitment to improve comments requires more work initially as teachers have to attend to the quality of the comments that they write on students work collaboration between teachers in sharing examples of effective comments can be very helpful and experience will lead to more fluency there is however more involved because comments become useful feedback only if students use them to guide further work so new procedures are needed after the first inset [inservice training meeting i was keen to try out a different way of marking books to give pupils more constructive feedback i was keen to try and have a more easy method of monitoring pupils response to my comments without having to trawl through their books each time to find out if they d addressed my comments i implemented a comment sheet at the back of my year-8 class books it is a4 [letter in size and the left-hand side is for my comments and the right-hand side is for the pupils to demonstrate by a reference to the page in their books where i can find the evidence to say whether they have done the work the comments have become more meaningful as the time has gone on and the books still take me only one hour to mark sian cornbury estate school initial fears about how students might react turned out we have encountered a variety of ways of accommodating the new emphasis on comments some teachers have ceased assigning scores or grades at all some teachers enter scores in their own record books but do not write them in the students books others give a score or grade only after a student has responded to the teacher s comments some teachers spend more time on certain pieces of work to ensure that they obtain good feedback and to make time for this either do not mark some pieces or look at only a third of their students books each week or involve the students in checking the straightforward tasks a particularly valuable method is to devote some lesson time to rewriting selected pieces of work so that emphasis can be put on feedback for improvement within a supportive environment this practice can change students expectations about the purposes of class work and homework as they tried to create useful feedback comments many of the project teachers realized that they needed to reassess september 2004 13


p. 6

the work that they had asked students to undertake they found that some tasks were useful in revealing students understandings and misunderstandings while others focused mainly on conveying information so some activities were eliminated others modified and new and better tasks actively sought overall the main ideas for improvement of feedback can be summarized as follows · written tasks alongside oral questioning should encourage students to develop and show understanding of the key features of what they have learned · comments should identify what has been done well and what still needs improvement and give guidance on how to make that improvement · opportunities for students to respond to comments should be planned as part of the overall learning process the central point here is that to be effective feedback should cause thinking to take place the implementation of such reforms can change both teachers and students attitudes toward written work the assessment of students work will be seen less as a competitive and summative judgment and more as a distinctive step in the process of learning peer assessment and self-assessment students can achieve a learning goal only if they understand that goal and can assess what they need to do to reach it so self-assessment is essential to learning.6 many teachers who have tried to develop their students self-assessment skills have found that the first and most difficult task is to get students to think of their work in terms of a set of goals insofar as they do so they begin to develop an overview of that work that allows them to manage and control it for themselves in other words students are developing the capacity to work at a metacognitive level in practice peer assessment turns out to be an important complement to self-assessment peer assessment is uniquely valuable because students may accept criticisms of their work from one another that they would not take seriously if the remarks were offered by a teacher peer work is also valuable because the interchange will be in language that students themselves naturally use and because students learn by taking the roles of teachers and examiners of others.7 one teacher shared her positive views of peer assessment as well as assessing and marking through discussion and clear guidance their own work they also assess and mark the work of others this they do in a very mature and sensible way and this has proved to be a very worthwhile experiment the students 14 phi delta kappan know that homework will be checked by themselves or another girl in the class at the start of the next lesson this has led to a well-established routine and only on extremely rare occasions have students failed to complete the work set they take pride in clear and well-presented work that one of their peers may be asked to mark any disagreement about the answer is thoroughly and openly discussed until agreement is reached alice waterford school the last sentence of this teacher s comments brings out an important point when students do not understand an explanation they are likely to interrupt a fellow student when they would not interrupt a teacher in addition to this advantage peer assessment is also valuable in placing the work in the hands of the students the teacher can be free to observe and reflect on what is happening and to frame helpful interventions we regularly do peer marking i find this very helpful indeed a lot of misconceptions come to the fore and we then discuss these as we are going over the homework i then go over the peer marking and talk to pupils individually as i go round the room rose brownfields school however self-assessment will happen only if teachers help their students particularly the low achievers to develop the skill this can take time and practice the kids are not skilled in what i am trying to get them to do i think the process is more effective long term if you invest time in it it will pay off big dividends this process of getting the students to be more independent in the way that they learn and to take the responsibility themselves tom riverside school one simple and effective idea is for students to use traffic light icons labeling their work green yellow or red according to whether they think they have good partial or little understanding these labels serve as a simple means of communicating students self-assessments students may then be asked to justify their judgments in a peer group thus linking peer assessment and self-assessment this linkage can help them develop the skills and the detachment needed for effective self-assessment another approach is to ask students first to use their traffic light icons on a piece of work and then to indicate by hands-up whether they put a green yellow or red icon on it the teacher can then pair up the greens and the yellows to help one another deal with their problems while the red students meet with the teacher as a group to deal with their deeper problems for such peer-group work to


p. 7

succeed many students will need guidance about how to behave in groups including such skills as listening to one another and taking turns in some subjects taking time to help students understand scoring rubrics is also very helpful students can be given simplified versions of the rubrics teachers use or they can be encouraged to rewrite them or even to create their own again peer assessment and self-assessment are intimately linked observers in several language arts classrooms saw children apply to their own work lessons they had learned in peer assessment a frequently heard comment was i didn t do that either or i need to do that too students reflection about their understanding can also be used to inform future teaching and their feedback can indicate in which areas a teacher needs to spend more time a useful guide is to ask students to traffic light an endof-unit test at the beginning of the unit the yellow and red items can be used to adjust priorities within the teaching plan our experience leads us to offer the following recommendations for improving classroom practice · the criteria for evaluating any learning achievements must be made transparent to students to enable them to have a clear overview both of the aims of their work and of what it means to complete it successfully such criteria may well be abstract but concrete examples should be used in modeling exercises to develop understanding · students should be taught the habits and skills of collaboration in peer assessment both because these are of intrinsic value and because peer assessment can help develop the objectivity required for effective self-assessment · students should be encouraged to keep in mind the aims of their work and to assess their own progress toward meeting these aims as they proceed then they will be able to guide their own work and so become independent learners the main point here is that peer assessment and self-assessment make distinct contributions to the development of students learning indeed they secure aims that cannot be achieved in any other way the formative use of summative tests the practices of self-assessment and peer assessment can be applied to help students prepare for tests as in tackling the following problem [the students did not mention any of the reviewing strategies we had discussed in class when questioned more closely it was clear that many spent their time using very passive revision [reviewing tech niques they would read over their work doing very little in the way of active revision or reviewing of their work they were not transferring the active learning strategies we were using in class to work they did at home tom riverside school to remedy this situation students can be asked to traffic light a list of key words or the topics on which the test will be set the point of this exercise is to stimulate the students to reflect on where they feel their learning is secure which they mark green and where they need to concentrate their efforts in yellow and red these traffic lights then form the basis of a review plan students can be asked to identify questions on past tests that probe their red areas then they can work with textbooks and in peer groups to ensure that they can successfully answer those questions the aftermath of tests can also be an occasion for formative work peer marking of test papers can be helpful as with normal written work and it is particularly useful if students are required first to formulate a scoring rubric an exercise that focuses attention on the criteria of quality relevant to their productions after peer marking teachers can reserve their time for discussion of the questions that give widespread difficulty while peer tutoring can tackle those problems encountered by only a minority of students one other finding that has emerged from research studies is that students trained to prepare for examinations by generating and then answering their own questions outperformed comparable groups who prepared in conventional ways.8 preparing test questions helps students develop an overview of the topic pupils have had to think about what makes a good question for a test and in doing so need to have a clear understanding of the subject material as a development of this the best questions have been used for class tests in this way the pupils can see that their work is valued and i can make an assessment of the progress made in these areas when going over the test good use can be made of group work and discussions between students concentrating on specific areas of concern angela cornbury estate school developments such as these challenge common expectations some have argued that formative and summative assessments are so different in their purpose that they have to be kept apart and such arguments are strengthened when one experiences the harmful influence that narrow highstakes summative tests can have on teaching however it is unrealistic to expect teachers and students to practice such separation so the challenge is to achieve a more positive relationship between the two all of the ways we have deseptember 2004 15


p. 8

scribed for doing so can be used for tests in which teachers have control over the setting and the marking but their application may be more limited for tests in which the teacher has little or no control overall the main possibilities for improving classroom practice by using summative tests for formative purposes are as follows · students can be engaged in a reflective review of the work they have done to enable them to plan their revision effectively · students can be encouraged to set questions and mark answers so as to gain an understanding of the assessment process and further refine their efforts for improvement · students should be encouraged through peer assessment and self-assessment to apply criteria to help them understand how their work might be improved this may include providing opportunities for students to rework examination answers in class the overall message is that summative tests should become a positive part of the learning process through active involvement in the testing process students can see that they can be the beneficiaries rather than the victims of testing because tests can help them improve their learning find ways to help students restructure their knowledge to build in new and more powerful ideas in the kmofap classrooms as the teachers came to listen more attentively to the students responses they began to appreciate more fully that learning was not a process of passive reception of knowledge but one in which the learners were active in creating their own understandings put simply it became clear that no matter what the pressure to achieve good test scores learning must be done by the student students came to understand what counted as good work through exemplification sometimes this was done through focused whole-class discussion around a particular example at other times it was achieved through the use of sets of criteria to assess the work of peers engaging in peer assessment and self-assessment is much more than just checking for errors or weaknesses it involves making explicit what is normally implicit and thus it requires students to be active in their learning as one student wrote after a pupil marking my investigation i can now acknowledge my mistakes easier i hope that it is not just me who learned from the investigation but the pupil who marked it did also next time i will have to make my explanations clearer as they said it is hard to understand i will now explain my equation again so it is clear reflections some underlying issues the changes that are entailed by improved assessment for learning have provoked us and the teachers involved to reflect on deeper issues about learning and teaching learning theory one of the most surprising things that happened during the early inset sessions was that the participating teachers asked us to run a session on the psychology of learning in retrospect perhaps we should not have been so surprised at this request after all we had stressed that feedback functioned formatively only if the information fed back to the learner was used by the learner in improving performance but while one can work out after the fact whether or not any feedback has had the desired effect what the teachers needed was a way to give their students feedback that they knew in advance was going to be useful to do that they needed to build up models of how students learn so the teachers came to take greater care in selecting tasks questions and other prompts to ensure that students responses actually helped the teaching process such responses can put on the table the ideas that students bring to a learning task the key to effective learning is then to 16 phi delta kappan the students also became much more aware of when they were learning and when they were not one class which was subsequently taught by a teacher not emphasizing assessment for learning surprised that teacher by complaining look we ve told you we don t understand this why are you going on to the next topic while students who are in tune with their learning can create difficulties for teachers we believe that these are exactly the kinds of problems we should want to have subject differences from hearing about research and discussing ideas with other colleagues the teachers built up a repertoire of generic skills they planned their questions allowed appropriate wait time and gave feedback that was designed to cause thinking they ensured that students were given enough time during lessons to evaluate their own work and that of others however after a while it became clear that these generic strategies could go only so far choosing a good question requires a detailed knowledge of the subject but not necessarily the knowledge that is gained from advanced study


p. 9

in a subject a high level of qualification in a subject is less important than a thorough understanding of its fundamental principles an understanding of the kinds of difficulties that students might have and the creativity to be able to think up questions that stimulate productive thinking.9 furthermore such pedagogical content knowledge is essential in interpreting responses that is what students say will contain clues to aspects of their thinking that may require attention but picking up on these clues requires a thorough knowledge of common difficulties in learning the subject thus while the general principles of formative assessment apply across all subjects the ways in which they manifest themselves in different subjects may differ we have encountered such differences in making comparisons between teachers of mathematics science and language arts in mathematics students have to learn to use valid procedures and to understand the concepts that underpin them difficulties can arise when students learn strategies that apply only in limited contexts and do not realize that they are inadequate elsewhere questioning must then be designed to bring out these strategies for discussion and to explore problems in understanding the concepts so that students can grasp the need to change their thinking in such learning there is usually a well-defined correct outcome in more open-ended exercises as in investigations of the application of mathematical thinking to everyday problems there may be a variety of good solutions then an understanding of the criteria of quality is harder to achieve and may require joint discussion of examples and of the abstract criteria that they exemplify in science the situation is very similar there are many features of the natural world for which science provides a correct model or explanation however outside school many students acquire different ideas for example some students come to believe that animals are living because they move but that trees and flowers are not because they don t or students may believe that astronauts seem almost weightless on the moon because there is no air present many of these alternative conceptions can be anticipated for they have been well documented what has also been documented is that the mere presentation of the correct view has been shown to be ineffective the task in such cases is to open up discussion of such ideas and then provide feedback that challenges them by introducing new pieces of evidence and argument that support the scientific model there are other aspects for which an acceptable outcome is less well defined as in mathematics open-ended investigations call for different approaches to formative assessment even more open are issues about social or ethical implications of scientific achievements for there is no answer thus such work has to be open in a more fundamental way then the priority in giving feedback is to challenge students to tease out their assumptions and to help them to be critical about the quality of any arguments peer assessment and self-assessment have a long history in language arts both the nature of the subject and the open outcome of many of the tasks characteristically make such practices central to one of the overall aims of the discipline which is to enhance the critical judgment of the students the priority in giving feedback is to challenge students to tease out their assumptions and to help them be critical about the quality of arguments a second important function of peer assessment and self-assessment was introduced by royce sadler who argued that criteria alone are unhelpful in judging the quality of a piece of work or in guiding progression because there will always be too many variables.10 the key lies in knowing how to interpret the criteria in any particular case which involves guild knowledge teachers acquire this knowledge through assessing student work and it is this process that allows them to differentiate between grades and to gain a sense of how progress is achieved peer assessment and self-assessment provide similar opportunities for students to be apprenticed into the guild provided the criteria of quality are clearly communicated in language arts as in science and mathematics attention needs to be paid to the central activities those that are the most successful are those rich tasks that provide students with an opportunity either to extend their understanding of a concept within the text or to scaffold their ideas before writing characteristically these include small-group and pair work with the results often being fed back into a whole-class discussion again this type of work is not uncommon in language arts the skill being to make the task sufficiently structured to scaffold learning but not so tightly defined as to limit thinking such activities not only provide students with a chance to develop their understanding through talk but they also provide the teacher with the opportunity to give feedback during the course of a lesson through further questioning and guidance the better the quality of the task the better the quality of the interventions september 2004 17


p. 10

differences between learning tasks can be understood in terms of a spectrum at one end are closed tasks with a single well-defined outcome at the other are open tasks with a wide range of acceptable outcomes tasks in language arts for example the writing of a poem are mainly at the open end but there are closed components even for such tasks for example the observance of grammatical or genre conventions tasks in say mathematics are more often closed but applications of mathematics to everyday problems can require open-ended evaluations thus in varying measure the guidance needed for these two types of learning work will be needed in all subjects despite these differences experience has shown that the generic skills that have been developed do apply across subjects one of the project s science teachers gave a talk to the whole staff about his experiences using some of the generic skills that we ve been discussing and subsequently found how such practices distributed themselves throughout the disciplines art and drama teachers do it all the time so do technology teachers something to do with open-ended activities long project times and perhaps a less cramped curriculum but an english teacher came up to me today and said yesterday afternoon was fantastic i tried it today with my year 8s and it works no hands up and giving them time to think i had fantastic responses from kids who have barely spoken in class all year they all wanted to say something and the quality of answers was brilliant this is the first time for ages that i ve learnt something new that s going to make a real difference to my teaching james two bishops school you do tells us how smart you are and what grades you ll get the difference is greatest for low achievers.11 · students given feedback as marks are likely to see it as a way to compare themselves with others ego involvement those given only comments see it as helping them to improve task involvement the latter group outperforms the former.12 students given marks are likely to see it as a way to compare themselves with others those given only comments see it as helping them to improve the latter group outperforms the former · in a competitive system low achievers attribute their performance to lack of ability high achievers to their effort in a task-oriented system all attribute performance to effort and learning is improved particularly among low achievers.13 · a comprehensive review of research studies of feedback found that feedback improved performance in 60 of the studies in the cases where feedback was not helpful the feedback turned out to be merely a judgment or grade with no indication of how to improve.14 in general feedback given as rewards or grades enhances ego involvement rather than task involvement it can focus students attention on their ability rather than on the importance of effort thus damaging the self-esteem of low achievers and leading to problems of learned helplessness 15 feedback that focuses on what needs to be done can encourage all to believe that they can improve such feedback can enhance learning both directly through the effort that can ensue and indirectly by supporting the motivation to invest such effort.16 motivation and self-esteem learning is not just a cognitive exercise it involves the whole person the need to motivate students is evident but it is often assumed that offering such extrinsic rewards as grades gold stars and prizes is the best way to do it however there is ample evidence to challenge this assumption students will invest effort in a task only if they believe that they can achieve something if a learning exercise is seen as a competition then everyone is aware that there will be losers as well as winners and those who have a track record as losers will see little point in trying thus the problem is to motivate everyone even though some are bound to achieve less than others in tackling this problem the type of feedback given is very important many research studies support this assertion here are a few examples · students who are told that feedback will help you to learn learn more than those who are told that how 18 phi delta kappan the big idea focus on learning our experiences in the project all point to the need to rethink a teacher s core aim enhancing student learning to achieve this goal calls for a willingness to rethink the planning of lessons together with a readiness to change


p. 11

the roles that both teacher and students play in supporting the learning process a learning environment principles and plans improvement in classroom learning requires careful forethought actually thinking about teaching has meant that i have been able to come up with ideas and strategies to cope with whatever has arisen and has contributed greatly to my professional development i now think more about the content of the lesson the influence has shifted from what am i going to teach and what are the pupils going to do toward how am i going to teach this and what are the pupils going to learn susan waterford school the learning process more transparent to the students indeed i now spend my time looking for ways to get students to take responsibility for their learning and at the same time making the learning more collaborative tom riverside school collaboration between teachers and students and between students and their peers can produce a supportive environment in which students can explore their own ideas hear alternative ideas in the language of their peers and evaluate them one technique has been to put the students into small groups and give each student a small part of the unit to explain to [his or her colleagues they are given a few minutes preparation time a few hints and use of their exercise books then each student explains [his or her chosen subject to the rest of the group students are quick to point out such things as i thought that the examples you chose were very good as they were not ones in our books i don t think i would have thought of those or i expected you to mention particles more when you were explaining the difference between liquids and gases these sessions have proven invaluable not only to me in being able to discover the level of understanding of some students but to the students too philip century island school one purpose of a teacher s forethought is to plan to improve teaching actions so for example the planning of questions and activities has to be in terms of their learning function i certainly did not spend sufficient time developing questions prior to commencing my formative training not until you analyze your own questioning do you realize how poor it can be i found myself using questions to fill time and asking questions which required little thought from the students when talking to students particularly those who are experiencing difficulties it is important to ask questions which get them thinking about the topic and will allow them to make the next step in the learning process derek century island school of equal importance is concern for the quality of the responses that teachers make whether in dialogue or in feedback on written assignments effective feedback should make more explicit to students what is involved in a high-quality piece of work and what steps they need to take to improve at the same time feedback can enhance students skills and strategies for effective learning there is also a deeper issue here a learning environment has to be engineered to involve students more actively in the learning tasks the emphasis has to be on students thinking and making that thinking public as one teacher put it there was a definite transition at some point from focusing on what i was putting into the process to what the students were contributing it became obvious that one way to make a significant sustainable change was to get the students doing more of the thinking i then began to search for ways to make an additional advantage of such an environment is that a teacher can work intensively with one group challenging the ideas and assumptions of its members knowing that the rest of the class members are also working hard so the main actions to be taken to engineer an effective learning environment are · plan classroom activities to give students the opportunity to express their thinking so that feedback can help develop it · formulate feedback so that it guides improvement in learning · use activities that demand collaboration so that everyone is included and challenged and train students to listen to and respect one another s ideas and · be sure that students are active participants in the lessons and emphasize that learning may depend less on their capacity to spot the right answer and more on their readiness to express and discuss their own understanding a learning environment roles and expectations it is one thing to plan new types of classroom activity and quite another to put them into practice in ways that are faithful to the aims they were developed to serve here september 2004 19


p. 12

there are no recipes to follow in a uniform way inside the black box was clear in stating that the effective development of formative assessment would come about only if each teacher finds his or her own ways of incorporating the lessons and ideas that are set out above into her or his own patterns of classroom work a second principle is that the learning environment envisaged requires a classroom culture that may well be unfamiliar and disconcerting for both teachers and students the effect of the innovations implemented by our teachers was to change the classroom contract between the teacher and the student the rules that govern the behaviors that are expected and seen as legitimate by teachers and students the students have to change from behaving as passive recipients of the knowledge offered by the teacher to becoming active learners who can take responsibility for and manage their own learning for the teachers courage is necessary one of the striking features of the project was that in the early stages many participants described the new approach as scary because they felt they were going to lose control of their classes toward the end of the project they spoke not of losing control but of sharing responsibility for the students learning with the class exactly the same process but viewed from two very different perspectives in one perspective the teachers and students are in a delivery/recipient relationship in the other they are partners in pursuit of a shared goal what formative assessment has done for me is made me focus less on myself but more on the children i have had the confidence to empower the students to take it forward robert two bishops school end the improvement of everyone s learning · by empowering students to become active learners thus taking responsibility for their own learning · by incorporating the changes in the teacher s role one step at a time as they seem appropriate and · by sustained attention to and reflection on ways in which assessment can support learning what you can do to incorporate some of the ideas about formative assessment into your own practice the first step is to reflect on what you are now doing discussion with colleagues and observation of one another s lessons can help spark such reflection a next step must be to try out changes wholesale change can be too risky and demanding so it is often best to think of one thing you feel confident to try be it traffic lights peer assessment improved questioning whatever and simply try it if you are a teacher in a middle school or high school try it with just one group or if you are an elementary teacher try it in just one subject area we found that as teachers gained confidence in the power of allowing students to say what they know and what they need to know the teachers decided that they should extend assessment for learning to the whole of their teaching taking on further strategies will then lead to further progress when several colleagues are collaborating each starts with a different strategy and then shares findings this process should lead to the explicit formulation of an action plan comprising a range of strategies to be used in combination preferably starting with a class at the beginning of the school year the first reason to start at the beginning of the year is so that there can be time to accustom both teacher and students to a new way of working the second is that it can be very difficult to change the established habits and routines in the middle of a year the experience of a year s sustained work with only a few classes preferably alongside similar efforts by colleagues can provide a firm basis for subsequent adoption of new practices on a wider scale collaboration with a group trying out similar innovations is almost essential mutual observation and the sharing of ideas and experiences about the progress of action plans can provide the necessary support both with the tactics and at a strategic level support for colleagues is particularly important in overcoming those initial uncertainties when engaging in the risky business of changing the culture and expectations in the classroom as for any innovation support from administrators is essential one way administrators can support change of this what has been happening here is that everybody s expectations that is what teachers and students think that being a teacher or being a student requires you to do have been altered while it can seem daunting to undertake such changes they do not have to happen suddenly changes with the kmofap teachers came slowly and steadily as experience developed and confidence grew in the use of the various strategies for enriching feedback and interaction for example many teachers started by using questions to encourage thinking then they improved their oral and written feedback so that it brought thinking forward and went on to develop peer and self-assessment to summarize expectations and classroom culture can be changed · by changing the classroom contract so that all expect that teacher and students work together for the same 20 phi delta kappan


p. 13

kind is to help peer groups of teachers find time to meet on a regular basis opportunities should also be found for teachers to report to faculty and staff meetings the work of any group experimenting with innovations is an investment for the whole school so that support should not be treated as indulgence for idiosyncratic practices indeed such work should be integrated into a school improvement plan with the expectation that the dissemination of fruitful practices will follow from the evaluation of a group s experiences perience and expertise for disseminating their ideas throughout the school and for supporting colleagues in making similar explorations for themselves 1 paul black and dylan wiliam inside the black box raising standards through classroom assessment phi delta kappan october 1998 pp 139-48 a version of this article has been published and widely sold in the united kingdom a booklet published in 2002 has also been widely distributed in the uk it covers the same issues as the article and bears the same title with the same authors both booklets and further booklets in this series are published by nfer-nelson only a few references to the literature are given here further information about publications and other resources can be obtained on the king s college website in the pages of the king s formative assessment group some of the publications can be downloaded from this site 2 paul black and dylan wiliam assessment and classroom learning assessment in education march 1998 pp 7-71 3 paul black christine harrison clare lee bethan marshall and dylan wiliam assessment for learning putting it into practice buckingham u.k open university press 2003 4 mary budd rowe wait time and rewards as instructional variables their influence on language logic and fate control journal of research in science teaching vol 11 1974 pp 81-94 5 ruth butler enhancing and undermining intrinsic motivation the effects of task-involving and ego-involving evaluation on interest and performance british journal of educational psychology vol 58 1988 pp 1-14 6 royce sadler formative assessment and the design of instructional systems instructional science vol 18 1989 pp 119-44 7 royce sadler formative assessment revisiting the territory assessment in education vol 5 1998 pp 77-84 8 see for example paul w foos joseph j mora and sharon tkacz student study techniques and the generation effect journal of educational psychology vol 86 1994 pp 567-76 and alison king facilitating elaborative learning through guided student-generated questioning educational psychologist vol 27 1992 pp 111-26 9 see for example mike askew et al effective teachers of numeracy final report london king s college london school of education 1997 in this study there was no correlation between the progress made by elementary school students in arithmetic and the highest level of mathematics studied by the teacher indeed there was a nonsignificant negative correlation between the two the students who made the most progress were taught by teachers without high levels of subject knowledge but who emphasized the connections between mathematics concepts 10 sadler formative assessment and the design of instructional systems 11 richard s newman and mahna t schwager students help seeking during problem solving effects of grade goal and prior achievement american educational research journal vol 32 1995 pp 35276 12 ruth butler task-involving and ego-involving properties of evaluation effects of different feedback conditions on motivational perceptions interest and performance journal of educational psychology vol 79 1987 pp 474-82 13 rhonda g craven herbert w marsh and raymond l debus effects of internally focused feedback on enhancement of academic selfconcept journal of educational psychology vol 83 1991 pp 17-27 14 avraham n kluger and angelo denisi the effects of feedback interventions on performance a historical review a meta-analysis and a preliminary feedback intervention theory psychological bulletin vol 119 1996 pp 254-84 15 carol s dweck motivational processes affecting learning american psychologist special issue psychological science and education vol 41 1986 pp 1040-48 16 carol s dweck self-theories their role in motivation personality and development philadelphia psychology press 2000 k there may be a need to review school policies an example would be a policy that by demanding a grade on every piece of homework prevents the serious use of comments at the same time there may be a need to review current school policies because such policies can actually constrain the use of formative assessment a notable example would be a policy that by demanding that a score or grade be given on every piece of homework prevents the serious use of comments five of the six schools in the kmofap project have following the experience of their science and mathematics teachers modified their policies to allow comment only marking for two of these the modification was that no scores or grades be given on homework throughout the school in another example a target setting system that required very frequent review was inhibiting any change in learning methods that might slow down immediate progress in order to produce medium to long-term gains in learning skills those engaged in innovations may need formal exemption from such policies thus support evaluation and subsequent dissemination of innovation in assessment for learning will be planned in a coherent way only if the responsibility for strategic oversight of the development is assigned to a member of the school leadership team our experience supports the view that to realize the promise of formative assessment by leaving a few keen individuals to get on with it would be unfair to them while to do it by imposing a policy that requires all teachers to immediately change their personal roles and styles would be absurd what is needed is a plan extending over at least three years in which a few small groups are supported for a twoyear exploration these groups then form a nucleus of ex september 2004 21


p. 14

file name and bibliographic information k0409bla.pdf paul black christine harrison clare lee bethan marshall and dylan wiliam working inside the black box assessment for learning in the classroom phi delta kappan vol 86 no 1 september 2004 pp 9-21 copyright notice dylan wiliam holds copyright to this article and has given phi delta kappa permission to distribute it all further use must comply with u.s law governing fair use multiple copies in print and electronic formats may not be made or distributed without express permission from phi delta kappa international inc all rights reserved note that photographs artwork advertising and other elements to which phi delta kappa does not hold copyright may have been removed from these pages please fax permission requests to the attention of kappan permissions editor at 812/339-0018 or e-mail permission requests to



no comments yet