Developing Quality Open Response & Multiple Choice Items for the Classroom

 

Embed or link this publication

Popular Pages


p. 1

developing quality open response and multiple choice items for the classroom office of teaching learning

[close]

p. 2

developing quality open response and multiple choice items for the classroom planning guide introduction the purpose of this manual is to help you develop and incorporate questions like those on the kentucky core content test kcct into your classroom instruction the planning guide will take you through the basic steps of development throughout this manual you will find several links to sites with more information about the section you are reading the example questions included in this manual reflect attributes of kcct-like items multiple choice and open response questions while kcct items are intended to assess only core content standards statewide classroom assessments are meant to assess the learner goals in a particular standards based unit of study for a specific group of students the graphic comparing kcct items for state assessment and classroom use represents both the similarities and the differences of these two different applications click here for a text-based version comparing kcct items for state assessment and classroom use multiple choice and open response items state · addresses state expectations academic expectations and core content for assessment · annual measurement · results measure school performance · apply statewide sensitivity guidelines · evaluate long-term learning · encourage higher-level thinking and application of content · allow response flexibility · engage student interest · align with academic expectations · align with core content for assessment · instructions are age appropriate clear in language and graphics achievable and scoreable classroom · address unit and lesson objectives · ongoing measurement · results inform instructional planning/practice · show awareness of local sensitivity issues · embed into curriculum and instruction · develop in classroom context open response or questions and multiple choice mc questions are not the only means of assessing student learning it should not be assumed that this manual signifies that or and mc are the only types of assessments to be used in the classroom teachers should allow for all types of assessment such as performance tasks benchmark assessments constructed response etc when designing a unit of study teachers should consider multiple types of assessments for both formative and summative purposes while multiple choice and open response items are certainly needed and students should be exposed to them often they should also be given the opportunity to demonstrate their learning in a variety of ways 6/14/07 kentucky department of education 1

[close]

p. 3

developing quality open response and multiple choice items for the classroom table of contents planning an assessment 3 writing an open response question 5 the five basic open response question types 7 developing a scoring guide 11 classroom practices for improving student responses 15 writing a multiple choice question 17 a note about web links included in this document revisions to the structure of kde s web site may possibly result in the disruption of some of the web links in this document should a link no longer work please download the most recent version of this document or search by keyword for the appropriate linked document from the kde web site 6/14/07 kentucky department of education 2

[close]

p. 4

developing quality open response and multiple choice items for the classroom planning an assessment open response and multiple choice questions should be included within the unit of study as assessments of the content and skills of that unit and not as isolated activities refer to the big picture curriculum instruction assessment see appendix a developing quality assessments is an important step in constructing an instructional unit so this document should be considered as a companion piece to the unit of study manual all questions should be authentic and directly related to the standards e.g academic expectations program of studies and core content for assessment addressed in the unit of study academic expectations program of studies core content for assessment 4.1 combined curriculum document the core content for assessment has depth of knowledge dok levels attached to each standard dok measures the cognitive complexity of an individual item or standard kentucky s core content for assessment contains dok ceilings for the purpose of ensuring alignment between the standards and the items used to assess them the dok levels indicated by the core content for assessment set a ceiling or upper limit for the cognitive complexity of questions included in the kcct while the questions on the kcct must not exceed the dok levels indicated by each individual standard assessments written for the classroom are free to exceed this ceiling more information can be found about dok in the support material in kde s web page each content area has dok charts and examples in their support materials creating student learning targets to plan assessments while both or and mc item can potentially be used to assess any standard the decision as to which is most appropriate depends upon the specific learning target you wish to assess most content standards need to be `deconstructed into learning targets that specify what student learning and skill/ability should be demonstrated for example in the combined curriculum document for science at grade 4 structure and transformation of matter one of the skills/concepts is sc-4-stm-s-4 students will conduct tests compare data and draw conclusions about physical properties of matter including states of matter conduction and buoyancy obviously a single item either mc or or could not be constructed to effectively assess that entire statement however by deconstructing you can develop learning targets such as compare the physical properties of matter to make determinations about whether the material will sink or float draw conclusions about the physical properties of [some type of matter after learning the most common uses of that type of matter etc an individual or or mc item could be constructed around each of those learning targets the decision about which is most appropriate is influenced by several factors 6/14/07 kentucky department of education 3

[close]

p. 5

developing quality open response and multiple choice items for the classroom considerations for choosing the appropriate question type or or mc what is an open response question an open response item is a question that requires students to both demonstrate content knowledge and to apply that knowledge in some way it is the application component of an open response question that distinguishes it from the more familiar essay or constructed response question for all or questions content is the foundation of student answers students must demonstrate content knowledge in order to successfully support their answers to communicate clearly what they know and are able to do or items often allow for more depth of knowledge to be demonstrated than in mc items students can be asked to demonstrate more complex cognitive behaviors such as comparing relating analyzing inferring concluding predicting generalizing solving and/or applying advantages of open response items q or items allow for more depth of knowledge to be demonstrated than do mc items q or items allow students to demonstrate more complex cognitive behaviors such as comparing relating analyzing inferring concluding predicting generalizing solving and/or applying disadvantages of open response items q or items are more difficult and more time consuming to score q because of the time required to answer them there must usually be fewer open response items on an assessment than mc items q effectiveness of or items is dependant on the scoring guide and answer information provided what is a multiple choice question multiple choice questions are considered `selected response items students are given three or more possible answers 4 on the kcct and are asked to choose the correct answer or the best answer the item begins with an item stem followed by response choices the `key or correct answer and `distractors mc questions can be used to measure knowledge recall as well as higher order thinking they are appropriate for use with objectives that call for the students to do tasks such as recognize distinguish between select estimate infer predict relate categorize etc advantages of multiple choice items q can be used to measure a wide variety of learning outcomes q permit wide sampling and broad coverage of a content domain q are reliable and efficient to score q can provide useful diagnostic information about the learning of individual students or groups of students disadvantages of multiple choice items q mc items are difficult to write well q mc items cannot measure certain types of skills e.g the ability to organize and express ideas in writing conduct a scientific investigation q performance on mc items can be influenced by student characteristics unrelated to the subject of measurement such as reading ability and test-wiseness the chart in appendix b illustrates the planning process when designing kcct-like questions 6/14/07 kentucky department of education 4

[close]

p. 6

developing quality open response and multiple choice items for the classroom writing an open response question while this document primarily addresses kcct-like or question development it is also designed to help in the construction of assessment items for classroom use unlike items developed for the kcct items developed for classroom use are not limited to addressing core content for assessment standards items intended for classroom use are free to address broader standards such as the program of studies national standards or local curriculum guides in contrast items that are developed for the kcct must by definition be aligned to the core content for assessment for items based on core content depth of knowledge dok ceilings are not an absolute limit for local assessment unlike item developers for the kcct teachers are free to exceed the dok ceilings for any core content for assessment standard when developing items characteristics of a kcct-like open response qdirectly tied to one or more content standards qconsists of an item name prompt and directions question item name prompt native american influences native american cultures have influenced many parts of american life including our government for example in the iroquois culture each tribe would send a representative to meet with the other tribes representatives to discuss problems and make decisions for the entire iroquois league a explain how our government uses this idea of representative government b explain two reasons why this is a good way to govern our country directions q specifies exactly what a student is required to do in order to achieve the maximum score no extension is q requires a student to required beyond what the question specifies see common misconceptions on page 14 · demonstrate content knowledge · apply that knowledge and · communicate an answer in no more than a one-page written response mathematics and science response pages are printed on graph paper other content area response sheets are printed on lined paper sample response books are available here q scored by the use of a question-specific rubric on a 0-4 scale but a general scoring guide is available to guide student responses to download the general scoring guide click here 6/14/07 kentucky department of education 5

[close]

p. 7

developing quality open response and multiple choice items for the classroom q written in one of five basic question formats q or questions may · have a correct answer which students can determine and explain through a variety of methods or in varying degrees of correctness · have multiple successful answers for which students must apply their analytical skills to a response or · combine requirements one part requires a student to provide a single correct answer and a subsequent part asks the student to extend his/her knowledge in another way such as applying the knowledge to another situation or by predicting an outcome in general an or differs from an essay or short answer question in that it requires some component of application below are some examples illustrating the difference between a kcct-like question and other forms of constructed response questions click here to download the word version essay question assesses content knowledge only open response question assesses content knowledge and application list and describe the three steps of the water cycle john s class is studying the water cycle by observing a sealed jar of water on the windowsill a list and describe the three steps of the water cycle b choose one of these steps and predict how john might see it occurring inside the jar naep extended-response open-ended parameters what exactly is meant by `evaluate open response question assesses content knowledge and application union and confederate resources as percentages of total united states resources resource population railroads farm acreage factory workers north 71 71 65 9 south 9 9 35 8 the union and confederate states had significant differences in the resources available to them during the civil war union and confederate resources as percentages of total united states resources north 71 71 65 9 south 9 9 35 8 resource population use the information in the table above to evaluate the railroads statement the south could never have won the civil farm acreage factory workers war a choose two of the resources listed and describe why they were important b explain what advantages an abundance of those resources would have provided during the civil war 6/14/07 kentucky department of education 6

[close]

p. 8

developing quality open response and multiple choice items for the classroom the five basic open response question types 1 scaffolded questions scaffolded questions have multiple parts with each question or direction the student is to address presented and labeled separately e.g a b c the order is arranged so that successive questions depend upon the response to the previous question often each part becomes increasingly more difficult or complex scaffolded question example the framers of the u.s constitution wanted to prevent the new federal government from becoming a dictatorship to keep the government from becoming too powerful they divided its powers among three branches a for each of the three branches of government identify one power given to it by the constitution b explain why each power you identified in part a is important to our system of government support your answer with real-life examples note answering part b of this question requires that the student be able to list branches of government in part a 2 single dimension/component questions single dimension/component forms ask a straight-forward question which requires explanation examples description or evidence as support single dimension/component example rivers provide several advantages to cities many kentucky cities are located near large rivers describe three important advantages that the rivers provide these cities explain why each advantage is important 3 two or more relatively independent components questions two or more relatively independent components are signaled by a b c parts the parts may be about the same prompt but have little relation to each other in that a correct response to one question is not dependent upon the response to the other questions two or more relatively independent components example fossils provide important clues about things that have lived in the past a describe two ways that fossils can form b explain one way that fossils can help us understand how livings things have changed over time note answering part b of this question does not require the student be able to successfully answer part a he reverse is also true 6/14/07 kentucky department of education 7

[close]

p. 9

developing quality open response and multiple choice items for the classroom 4 student choice topics/options provided questions student choice with provided topics or options that force students to choose from the selections they offer students more opportunities to demonstrate their individual learning but may provide more scoring difficulty because there are many more correct answers student choice example some of earth s materials are listed below soil water gases of the atmosphere rocks a choose two materials from the list explain how a plant uses each of these materials to live b choose two materials from the list explain how an animal uses each of these materials to live 5 response to provided information questions students must be able to manipulate raw materials such as data readings or graphics in order to respond to specific questions this question type is combined with another type of question in the example below the student is responding to a text passage but the question is scaffolded as well response to provided information example note the student was required to read a text passage before completing this question in the story first light matthew woke up in another time period the 1850s a describe four things matthew discovered that were different from what he was used to in his present life b explain how each of those differences affected him use information from the story to support your answer 6/14/07 kentucky department of education 8

[close]

p. 10

developing quality open response and multiple choice items for the classroom steps in constructing an open response question 1 decide which program of studies core content or other standards you wish to assess 2 identify the major concepts within these standards you want to assess most content standards are too complex to be completely assessed with a single question so it is important to identify the specific learning targets within the standard your question will be focused on 3 decide how deeply you wish to assess this concept what depth of knowledge dok level is acceptable for the question must it be written to a specified dok level if so what characteristics must your question have to meet your chosen level of complexity remember dok levels on the core content are a ceiling for state assessment but classroom assessments are free to vary from that ceiling to download content-specific depth of knowledge guides click here 4 choose the most appropriate question type from the five possible choices 5 establish the situation by writing a question prompt that reflects the major concepts chosen the prompt should set the stage for the question that follows it even if the item has a graphic there still must be a prompt that describes or provides information related to the graphic and/or item directions try to be economical in your choice of words the prompt should help engage students the purpose of the question prompt is not to assess reading ability present the prompt in paragraph form but use bullets to emphasize important details if the question has a graphic the graphic should be preceded by an introductory statement 6 design directions that tell the student how to demonstrate knowledge making sure the question requires application of knowledge and not merely recall if the question is in multiple parts at least one of the parts should require students to apply their knowledge tips for designing directions q specify exactly what you want students to answer specify numbers of responses if appropriate specify three examples if you require three as opposed to ambiguous terms like some or several determine if they can decipher the question q simple and direct language you are evaluating what students know rather than attempting to use q language that is both age and grade-level appropriate use q simple basic vocabulary when appropriate and technical vocabulary when you are assessing use the students knowledge of the meaning of the technical word/phrase for example when you plant a seed the roots grow downward this is called geotropism which factor is responsible for geotropism vs when you plant a seed the roots grow downward and the stem grows upward which factor is responsible for the roots growing downward 6/14/07 kentucky department of education 9

[close]

p. 11

developing quality open response and multiple choice items for the classroom q the item asks students to read a passage or examine a graphic and then give three ways or if explain two reasons based on the material given make sure there are at least three ways or two reasons found in the material provided or graphic to consult then there should be at least twice that number of possible answers q students are asked to generate a certain number of ideas on their own without having a passage if q if the students are required to respond to multiple parts of a question label each part separately a b c 7 create a question-specific scoring guide rubric to download released forms of the kcct assessment click here to download a sample collection of open response items from released kcct forms click here appendix c provides a graphic illustrating the steps in designing an open response item appendix d provides a template for writing open response questions 6/14/07kentucky department of education 10

[close]

p. 12

developing quality open response and multiple choice items for the classroom developing a scoring guide scoring guides are sets of criteria which describe the characteristics of responses at each identified level they provide the tool necessary to accurately evaluate student success for each individual question scoring guides are developed following the actual writing of the question this will allow the teacher to discover potential problems with the question constructing the scoring guide will also help ensure that the question is rich enough to support various levels of student responses a scoring guide should q include a clear explanation of what is expected in a quality student response q define the various levels of possible student responses and place a value on each level if there is more than one way a student may achieve a given score level the scoring guide should include those different possibilities q enable scoring to be consistent accurate and as objective as possible the scoring guide should provide a scorer with the details necessary to score a response q simple language and repeat significant descriptive words used in the question use q ensure that what is required for a top-level response is clearly indicated in the description the kentucky general scoring guide is a good example to follow when developing your own scoring guides it utilizes four distinct performance levels and serves as a template for constructing an item-specific scoring guide to download the general scoring guide click here establishing the parameters the first step in designing the scoring guide is to determine what a top-level response should say in other words what is the expectation for a response that fully and completely answers the question if answering the question well is difficult for the teacher is the question itself a reasonable task for students the question should be revised if you cannot answer it in writing yourself the next step is to write descriptions of each of the other levels while there is no perfect formula for distinctions between levels there should be appropriate and sequential differences between levels a good scoring guide helps make the scorer s task easier by clearly stating the differences between levels in discernible and important ways sometimes these distinctions will include quantity indicators for instance if the question asks for three examples a response with two well-defined examples might receive a score of three if you are using a fourlevel scoring guide simple numerical indicators should not be the only difference between levels quality of work must also be considered 6/14/07 kentucky department of education 11

[close]

p. 13

developing quality open response and multiple choice items for the classroom kcct scoring guides are written with four levels of performance as well as zero or blank non-performance levels common descriptors used to distinguish the four performance levels are listed below t the top or four level of a scoring guide may typically characterize responses as effective thorough complete successful insightful in depth efficient and/or sophisticated t a level three response will usually use terms such as adequate satisfactory understanding of major concepts completes most and/or clear t a response of level two will often be described as having gaps or leaps incomplete some important points demonstrates basic understanding and/or some errors t responses receiving a one are typically labeled as minimal completes only small part little understanding not logical unclear and/or major errors finally review your scoring guide to confirm that it is consistent with both the question and the standards it is intended to assess note the four levels of an open-response scoring guide do not directly correlate with the four performance levels n,a,p,d of the kcct nor with the four levels of dok reminders as you design your scoring guide remember to ask yourself q what does a top-level response contain q can i write a top-level response q is my guide consistent with the question academic expectations program of studies core content and unit instruction q is each level clearly different from other levels sample assessment item and scoring guide safety around strangers susie has been taught by her parents and her teachers not to talk to strangers when she is alone a describe three things in addition to not talking that susie might do if a stranger came up to her when she was alone b explain how each of these actions would help to keep her safe note when constructing a scoring guide to be used by someone other than the teacher who originally planned or taught the unit it may be necessary to create a list of plausible responses this step is usually taken when scoring items in groups or when sharing items with teachers less familiar with the content of the question examples of things susie might do if a stranger came up to her when she was alone · go inside · lock doors · tell an adult · get car license number · call police/911 · walk away/run away 6/14/07 kentucky department of education 12

[close]

p. 14

developing quality open response and multiple choice items for the classroom scoring guide score 4 description student describes three things in addition to not talking susie might do if a stranger came up to her when she was alone student clearly explains how each of these actions would help to keep her safe student describes three things in addition to not talking susie might do if a stranger came up to her when she was alone student generally explains how each of these actions would help to keep her safe or student describes three things in addition to not talking susie might do if a stranger came up to her when she was alone student clearly explains how one or two of these actions would help to keep her safe student describes three things in addition to not talking susie might do if a stranger came up to her when she was alone with limited or no explanation of how each of these actions would help to keep her safe or student describes three things in addition to not talking susie might do if a stranger came up to her when she was alone student generally explains how one or two of these actions would help to keep her safe student demonstrates minimal understanding e.g student describes one thing in addition to not talking susie might do if a stranger came up to her when she was alone with limited or no explanation of how the action would help to keep her safe student s response is totally incorrect or irrelevant no student response 3 2 1 0 blank sample 4-point student response a every child is taught one way or another how to deal with strangers susie has been taught by parents and teachers not to talk to strangers i will describe 3 things susie could do if a stranger approached her while she was alone susie could pretend she didn t hear the person yell for an adult or run as fast as possible b these things are what i would do if i was in her situation below i will tell you why if susie ignored the person they might just leave her alone if she called for an adult if the stranger had bad intentions they would probably leave her alone if she ran away as fast as possible chances are the stranger won t bother to chase her i was taught to stay away from strangers but this is what i was taught to do if one approached me i hope susie never faces that situation for more examples of annotated released items with scoring guides click here appendix e provides a template for creating a four point scoring guide similar to the one above 6/14/07 kentucky department of education 13

[close]

p. 15

developing quality open response and multiple choice items for the classroom common misconceptions about scoring of open response items on the kcct misconception restating the question is mandatory fact no it is not required and doing so will not gain additional points responses restating the question without further no additional information must be included in order to information will be given at least one point receive any credit a graphic organizer should be done on the response depending on the type of question being asked a page graphic organizer may not be the best way to record the answer best practice would be to use the organizer on scrap paper and then record answers must be in paragraph form scorers are trained to focus on content and not address the format of the response a response in any format bulleted labeled diagram or graphic organizer will be scored however the nature of graphic organizers is to outline or abbreviate rather than to give supporting information and/or explanations that are usually required by the questions doing more than required by the prompt will score a 4 a 4 will be assigned to a response that completely and accurately reflects the correct answer according to the rubric no additional information is required to score a 4 must use content specific vocabulary in order to not necessarily if the content can be adequately score a 4 expressed without the use of specific vocabulary appropriate credit will be given to the response content vocabulary is only required if the question specifically asks for its use three or more examples must always be given no the question will specify the number of examples required giving more will not increase the odds of receiving a higher score scorers only have 30 seconds to score each piece no scorers can take as much time as needed on each piece released items are bad items that have been thrown not true the items are good to use in instruction and out of the test are representative of the test release is not based on how the item performed strike thrus on an open response is better than not necessarily scorers will score what is written erasing on the page strike thrus may take up more space than erasing leaving less space to complete a proper response open response answers will be scored using the new each open response item has a specific scoring guide analytical scoring rubric the question is scored for content and not writing mechanics reading and mathematics at the new grades will be for grades 3-8 all reading and mathematics scores will counted for only nclb be used for nclb and cats calculations draft 6/14/07 kentucky department of education 14

[close]

Comments

no comments yet