2011-2012 CLA Report

 

Embed or link this publication

Popular Pages


p. 1

[cla kansas state university 2011-2012 cla institutional report

[close]

p. 2

2011-2012 results your 2011-2012 results consist of two components cla institutional report and appendices cla student data file report the report introduces readers to the cla and its methodology including an enhanced value-added equation presents your results and offers guidance on interpretation and next steps appendices the report appendices offer more detail on cla tasks scoring and scaling value-added equations and the student data file a 1 2 3 4 5 6 task overview p 20-23 diagnostic guidance p 24 task development p 25 scoring criteria p 26-28 scoring process p 29 scaling procedures p 30-31 introduction to the cla p 3 methods p 4-5 your results p 6-10 results across cla institutions p 11-14 sample of cla institutions p 15-18 moving forward p 19 bcdefg modeling details p 32-36 h i j percentile lookup tables p 37-42 student data file p 43 cae board of trustees and officers p 44 student data file your student data file was distributed separately as a password-protected excel file your student data file may be used to link with other data sources and to generate hypotheses for additional research 2 2011-2012 cla institutional report

[close]

p. 3

1 introduction to the cla assessing higher-order skills the collegiate learning assessment cla is a major initiative of the council for aid to education the cla offers a value-added constructedresponse approach to the assessment of higher-order skills such as critical thinking and written communication hundreds of institutions and hundreds of thousands of students have participated in the cla to date the institution not the student is the primary unit of analysis the cla is designed to measure an institution s contribution or value added to the development of higher-order skills this approach allows an institution to compare its student learning results on the cla with learning results at similarly selective institutions the cla is intended to assist faculty school administrators and others interested in programmatic change to improve teaching and the continuous improvement model requires multiple indicators beyond the cla because no single test can serve as the cla helps campuses follow a continuous improvement model that positions faculty as central actors in the link between assessment and the teaching and learning process included in the cla are performance tasks and analytic writing tasks performance tasks present realistic problems that require students to analyze complex materials several different types of materials are used that vary in credibility relevance to the task and other characteristics students written responses to the tasks are graded to assess their abilities to think critically reason analytically solve problems and write clearly and persuasively the signaling quality of the cla is important because institutions need to have a frame of reference for where they stand and how much progress their students have made relative to the progress of students at other colleges yet the cla is not about ranking institutions rather it is about highlighting differences between them that can lead to improvements the cla is an instrument designed to contribute directly to the improvement of teaching and learning in this respect it is in a league of its own learning particularly with respect to strengthening higher-order skills the benchmark for all student learning in higher education there are however certain skills deemed to be important by most faculty and administrators across virtually all institutions indeed the higher-order skills the cla focuses on fall into this category 2011-2012 cla institutional report 3

[close]

p. 4

2 methods cla methodology the cla uses constructed-response tasks and value-added methodology to evaluate your students performance reflecting the following higherorder skills analytic reasoning and evaluation writing effectiveness writing mechanics and problem solving schools test a sample of entering students freshmen in the fall and exiting students seniors in the spring students take one performance task or a combination of one make-an-argument prompt and one critique-an-argument prompt the interim results that your institution received after the fall testing window reflected the performance of your entering students your institution s interim institutional report presented information on each this report is based on the performance of both your entering and exiting students value-added modeling is often viewed as an equitable way of estimating an institution s contribution to learning simply comparing average achievement of all schools tends to paint selective institutions in a favorable light and discount the educational efficacy of schools admitting students from weaker academic backgrounds valueadded modeling addresses this issue by of the cla task types including means averages standard deviations a measure of the spread of scores in the sample and percentile ranks the percentage of schools that had lower performance than yours also included was distributional information for each of the cla subscores analytic reasoning and evaluation writing effectiveness writing mechanics and problem solving the cla value-added estimation approach employs a statistical technique known as hierarchical linear modeling hlm under this methodology a school s value-added score indicates the degree to which the observed senior mean cla score meets exceeds or falls below expectations established by 1 seniors entering academic ability eaa scores and 2 the mean cla performance of freshmen at that school which serves as a control for selection effects not covered by eaa only students with eaa scores are included in institutional analyses providing scores that can be interpreted as relative to institutions testing students of similar entering academic ability this allows all schools not just selective ones to demonstrate their relative educational efficacy note that the methods employed by the community college learning assessment ccla differ from those presented here a description of those methods is available upon request a description of the differences between the original ols model and the enhanced hlm model is available in the frequently asked technical questions document distributed with this report sat math critical reading act composite or scholastic level exam sle scores on the sat scale hereinafter referred to as entering academic ability eaa 4 2011-2012 cla institutional report

[close]

p. 5

2 methods continued when the average performance of seniors at a school is substantially better than expected this school is said to have high value added to illustrate consider several schools admitting students with similar average performance on general academic ability tests e.g the sat or act and on tests of higher-order skills e.g the cla if after four years of college education the seniors at one school perform better on the cla than is typical for schools admitting similar students one can infer that greater gains in critical thinking and writing skills occurred at the highest performing school note that a low negative value-added score does not necessarily indicate that no gain occurred between freshman and senior year however it does suggest that the gain was lower than would typically be observed at schools testing students of similar entering academic ability value-added scores are placed on a standardized z-score scale and assigned performance levels schools that fall between -1.00 and +1.00 are classified as near expected between +1.00 and +2.00 are above expected between -1.00 and -2.00 are below expected above +2.00 are well above expected and below -2.00 are well below expected value-added estimates are also accompanied by confidence intervals which provide information on the precision of the estimates narrow confidence intervals indicate that the estimate is more precise while wider intervals indicate less precision our analyses include results from all cla institutions regardless of sample size and sampling strategy therefore we encourage you to apply due caution when interpreting your results if you tested a very small sample of students or believe that the students in your institution s sample are not representative of the larger student body moving forward we will continue to employ methodological advances to maximize the precision of our valueadded estimates we will also continue developing ways to augment the value of cla results for the improvement of teaching and learning 2011-2012 cla institutional report 5

[close]

p. 6

3 your results 3.1 value-added and precision estimates performance level total cla score performance task analytic writing task near near near near near value-added score 0.66 0.48 0.75 0.42 0.81 value-added percentile rank 75 69 80 70 85 confidence interval lower bound 0.09 -0.17 0.11 -0.28 0.16 confidence interval upper bound 1.23 1.13 1.39 1.12 1.46 expected mean cla score 1199 1217 1179 1157 1207 make-an-argument critique-an-argument 3.2 seniors unadjusted performance number of seniors total cla score performance task analytic writing task 104 53 51 51 51 104 mean score 1232 1246 1217 1179 1254 1170 mean score percentile rank 81 81 76 65 86 87 25th percentile score 1147 1157 1126 1129 1120 1070 75th percentile score 1303 1338 1299 1255 1343 1260 standard deviation 130 148 107 112 154 153 eaa make-an-argument critique-an-argument 3.3 freshmen unadjusted performance number of freshmen total cla score performance task analytic writing task 96 44 52 52 52 96 mean score 1026 1031 1021 1013 1029 1053 mean score percentile rank 39 43 38 34 43 57 25th percentile score 910 894 918 938 902 949 75th percentile score 1107 1111 1100 1160 1123 1170 standard deviation 140 140 141 179 141 145 eaa make-an-argument critique-an-argument 6 2011-2012 cla institutional report

[close]

p. 7

3 your results continued 3.4 student sample summary average freshman percentage across schools n/a n/a average senior percentage aross schools 17 83 transfer transfer students non-transfer students gender male female decline to state primary language english primary language other primary language field of study sciences and engineering social sciences humanities and languages business helping services undecided other n/a race ethnicity american indian alaska native asian pacific islander black non-hispanic hispanic white non-hispanic other decline to state parent education less than high school high school some college bachelor s degree graduate or professional degree number of freshmen 2 94 freshman percentage 2 98 number of seniors 0 104 senior percentage 0 100 21 75 0 22 78 0 38 61 1 41 63 0 39 61 0 36 63 1 96 0 100 0 87 13 102 2 98 2 87 13 17 4 5 12 32 26 18 4 5 13 33 27 22 12 11 12 26 17 35 15 7 13 18 16 34 14 7 13 17 15 21 18 17 15 22 7 0 1 2 2 89 2 0 0 1 2 2 93 2 0 1 7 14 15 59 3 2 0 4 1 2 96 0 1 0 4 1 2 92 0 1 1 7 10 11 63 4 4 0 12 16 44 24 0 13 17 46 25 6 23 24 28 20 0 7 23 41 33 0 7 22 39 32 5 16 28 29 22 average percentages across schools are not reported by transfer status because institutions do not necessarily define freshman transfers the same way 2011-2012 cla institutional report 7

[close]

p. 8

3 your results continued performance compared to other institutions figure 3.5 shows the performance of all four-year colleges and universities relative to their expected performance as predicted by the value-added model the vertical distance from the diagonal line indicates the value added by the institution institutions falling above the diagonal line are those that add more value than expected based on the model your institution is highlighted in red see appendix g for details on how the total cla score value-added estimates displayed in this figure were computed 3.5 observed cla scores vs expected cla scores 1400 1300 observed mean senior cla score 1200 1100 1000 your institution other cla institutions 900 observed performance equal to expected performance 800 800 900 1000 1100 1200 1300 1400 expected mean senior cla score due to the low statistical reliability of small sample sizes schools that tested fewer than 50 students are not included in figure 3.5 8 2011-2012 cla institutional report

[close]

p. 9

3 subscore distributions your results continued figures 3.6 and 3.8 display the distribution of your students performance in the subscore categories of analytic reasoning and evaluation writing effectiveness writing mechanics and problem solving the numbers on the graph correspond to the percentage of your students that performed at each score level the distribution of subscores across all schools is presented for comparative purposes the score levels range from 1 to 6 note that the graphs presented are not directly comparable due to potential differences in difficulty among task types and among subscore categories see diagnostic guidance and scoring criteria for more details on the interpretation of subscore distributions tables 3.7 and 3.9 present the mean and standard deviation of each of the subscores across cla task types for your school and all schools 3.6 seniors distribution of subscores analytic reasoning and evaluation 20 40 60 80 20 40 60 80 writing effectiveness 20 40 60 80 writing mechanics 20 40 60 80 problem solving performance task 45 28 2 1 4 2 3 4 5 21 0 6 45 28 2 1 4 2 3 4 15 5 6 6 55 23 0 1 2 2 3 4 71 5 21 0 6 38 42 17 0 1 2 2 3 4 5 2 6 0 0 0 20 40 60 80 20 40 60 80 make-an-argument 47 29 0 1 8 2 3 4 16 0 5 6 51 24 0 1 6 2 3 4 18 2 5 6 20 40 60 80 14 0 1 0 2 3 4 16 0 5 6 0 0 20 40 60 80 20 40 60 80 20 40 60 80 0 critique-an-argument 43 27 2 1 6 2 3 4 16 31 0 1 4 2 3 43 22 0 4 5 6 55 35 0 1 0 2 10 3 4 5 0 6 6 your school 0 6 5 all schools 0 0 3.7 seniors summary subscore statistics analytic reasoning and evaluation your school performance mean task standard deviation make-an mean argument standard deviation critique-an mean argument standard deviation 3.8 0.9 3.7 0.8 3.8 1.0 all schools 3.4 0.9 3.6 0.8 3.4 0.9 writing effectiveness your school 3.8 1.0 3.9 0.8 3.8 0.8 all schools 3.5 0.9 3.7 0.9 3.5 0.9 0 writing mechanics your school 3.9 0.7 4.0 0.5 4.3 0.6 all schools 3.7 0.8 3.8 0.7 3.9 0.7 problem solving your school 3.8 0.8 all schools 3.3 0.9 2011-2012 cla institutional report 9

[close]

p. 10

3 your results continued 3.8 freshmen distribution of subscores analytic reasoning and evaluation 20 40 60 80 20 40 60 80 writing effectiveness 20 40 60 80 writing mechanics 20 40 60 80 problem solving 52 30 5 1 2 3 14 0 4 5 0 6 performance task 34 5 1 2 39 23 0 3 4 5 0 6 45 32 5 1 16 2 2 3 4 5 0 6 39 48 0 0 0 0 2 1 2 3 11 4 0 5 0 6 20 40 60 80 20 40 60 80 make-an-argument 46 23 2 1 2 3 4 27 2 5 0 6 50 23 4 1 2 3 4 19 4 5 0 6 20 40 60 80 48 35 0 1 12 2 3 4 6 5 0 6 0 0 20 40 60 80 20 40 60 80 20 40 60 80 0 56 29 0 1 6 2 3 4 10 5 critique-an-argument 42 42 10 2 3 4 42 40 4 1 2 3 12 4 4 1 0 0 5 6 5 6 0 2 0 2 your school 0 6 0 all schools 3.9 freshmen summary subscore statistics analytic reasoning and evaluation your school performance mean task standard deviation make-an mean argument standard deviation critique-an mean argument standard deviation 2.8 0.8 3.0 0.8 2.6 0.8 all schools 2.9 0.8 3.2 0.8 2.8 0.9 writing effectiveness your school 2.8 0.9 3.0 0.9 2.7 0.8 all schools 2.9 0.9 3.2 0.9 2.8 0.8 writing mechanics your school 3.1 0.9 3.5 0.8 3.4 0.8 all schools 3.2 0.8 3.4 0.8 3.4 0.8 problem solving your school 2.7 0.7 all schools 2.7 0.8 10 2011-2012 cla institutional report

[close]

p. 11

4 results across cla institutions performance distributions tables 4.1 and 4.2 show the distribution of performance on the cla across participating institutions note that the unit of analysis in both tables is schools not students figure 4.3 on the following page shows various comparisons of different groups of institutions depending on which factors you consider to define your institution s peers these comparisons may show you how your institution s value added compares to those of institutions similar to yours 4.1 seniors number of schools total cla score performance task analytic writing task make-an-argument critique-an-argument eaa 172 171 172 172 172 172 mean score 1162 1165 1157 1142 1170 1062 25th percentile score 1108 1115 1107 1084 1126 1009 75th percentile score 1220 1229 1214 1201 1226 1115 standard deviation 87 95 84 86 91 102 4.2 freshmen number of schools total cla score performance task analytic writing task make-an-argument critique-an-argument eaa 169 167 169 169 169 169 mean score 1048 1048 1048 1047 1046 1031 25th percentile score 991 985 995 997 987 968 75th percentile score 1110 1117 1106 1110 1102 1094 standard deviation 93 98 89 96 88 110 158 institutions tested both freshmen and seniors 2011-2012 cla institutional report 11

[close]

p. 12

4 peer group comparisons results across cla institutions continued 4.3 1400 observed mean senior cla score 1300 1200 1100 1000 insitution size number of fte undergraduates 900 v small up to 3,000 midsized 3,001-10,000 large 10,001 or more 800 800 900 1000 1100 1200 1300 1400 expected mean senior cla score 1400 observed mean senior cla score 1300 1200 1100 1000 900 minority-serving institutions non-minority-serving institutions minority-serving institutions 800 800 900 1000 1100 1200 1300 1400 expected mean senior cla score 12 2011-2012 cla institutional report

[close]

p. 13

4 results across cla institutions continued 4.3 peer group comparisons continued 1400 observed mean senior cla score 1300 1200 1100 1000 insitution type doctoral masters 900 800 800 900 1000 1100 1200 1300 1400 bachelors expected mean senior cla score 1400 observed mean senior cla score 1300 1200 1100 1000 900 sector public private 800 800 900 1000 1100 1200 1300 1400 expected mean senior cla score 2011-2012 cla institutional report 13

[close]

p. 14

4 results across cla institutions continued sample representativeness cla-participating students appeared to be generally representative of their classmates with respect to entering ability levels as measured by entering academic ability eaa scores specifically across institutions the average eaa score of cla seniors as verified by the registrar was only 23 points higher than that of the entire senior class 1070 versus 1047 n 155 institutions further the correlation between the average eaa score of cla seniors and their classmates was high r 0.85 n 155 institutions the pattern for freshmen was similar the average eaa score of cla freshmen was only 6 points higher than that of the entire freshman class 1032 versus 1026 over n 156 institutions and the correlation between the average eaa score of cla freshmen and their classmates was similarly high r 0.92 n 156 institutions these data suggest that as a group cla participants were similar to all students at participating schools this correspondence increases confidence in the inferences that can be made from the results with the samples of students that were tested at a school to all the students at that institution as reported by school registrars 14 2011-2012 cla institutional report

[close]

p. 15

5 sample of cla institutions carnegie classification table 5.1 shows cla schools grouped by basic carnegie classification the spread of schools corresponds fairly well with that of the 1,587 fouryear not-for-profit institutions across the nation table 5.1 counts exclude some institutions that do not fall into these categories such as special focus institutions and institutions based outside of the united states 5.1 carnegie classification of institutional sample nation n 1,587 carnegie classification doctorate-granting universities master s colleges and universities baccalaureate colleges number 275 619 693 percentage 17 39 44 cla n 161 number 30 81 50 percentage 19 50 31 source carnegie foundation for the advancement of teaching carnegie classifications data file february 11 2010 2011-2012 cla institutional report 15

[close]

Comments

no comments yet