Scholarship of Assessment Grant:
Pilot of Collegial Review of Student Writing
Report of Activities to Date: Feb 24, 2005
By Barbara Price
Recommendations for Future Actions
Truman State University has a long history of direct assessment of student writing beginning in the late 1970’s. After initial attempts to assess writing using locally designed and nationally normed instruments, the University implemented in 1989 a locally designed writing assessment, the Sophomore Writing Experience (SWE), which remained in place as a graduation requirement for all students until 2002. One of the major strengths of the SWE was the opportunity provided for faculty to come together and discuss student writing. Faculty involved cited many personal and professional benefits from this experience, including a wider perspective of student writing across the University, as well as ideas for responding to their own students’ writing and insights gained from working with colleagues from other disciplines in an informal, relaxed atmosphere.
The current project, as part of a larger proposal for a multiple-component writing assessment program at Truman, aims to continue the tradition of collaborative review of student writing begun with the SWE. The purpose of this Pilot Collegial Review Workshop project was to develop a protocol for a collegial review of student writing at Truman that builds upon the tradition of the SWE while incorporating recent innovations in practitioner inquiry and other forms of collaboration. In addition to ongoing planning work of the Writing Assessment Committee and various staff development activities, this development work was planned to include a one-day Pilot Collegial Review Workshop coordinated by a Truman faculty member with experience in developing and implementing a similar review for the Iowa Writing Project.
A primary goal of assessment is to produce evidence that the work of teachers and students is effective in producing improved learning. Evidence in the form of quantitative data has much to offer, including that it is relatively easy to present. Many educators have expressed concern, however, that the data available from quantitative assessment is not sufficiently rich to alone inform instructional or programmatic decisions. In response, a number of qualitative approaches to assessment have been developed, in the hope that the evidence collected through such a process can complete the picture painted by quantitative data.
Variously referred to as collaborative assessment, practitioner inquiry, descriptive inquiry, descriptive review, and, for this proposal, collegial review, these approaches typically: foreground close reading of samples of student work; engage practitioners in facilitated discussion of the student work, as well as of their individual and institutional goals for student learning; and provide opportunity for practitioners to reflect upon their own practice in discussion with colleagues. The facilitated professional discussion at the heart of collegial review often employs protocols—agreed upon guidelines for conversation—to provide structure. A narrative report is the usual form in which the results of practitioners’ observations are presented. Benefits of such practitioner inquiry include that it: 1) is context based, looking at student work within the reality of the local educational setting, 2) builds community among practitioners within and across programs and, potentially, across similar programs in other locations, 3) generates new knowledge through discovery and shared craft knowledge, 4) creates improved understanding of institutional goals through discussion of various ways those goals can be met.
Examples of approaches to practitioner inquiry include Harvard’s Project Zero Descriptive Review Process, the Prospect Archive and Center for Education and Research’s Descriptive Review Process, the Tuning Protocols developed by the Coalition of Essential Schools and the Annenberg Institute for School Reform. The Academy for Educational Development, following a three-year study of National Writing Project teachers and student achievement instituted a similar series of collaborative workshops last year (for thirty years, the National Writing Project has provided highly regarded professional development opportunities for teachers at all educational levels). Sixteen National Writing Project sites across the country, including Missouri’s Gateway Writing Project, participated in weeklong institutes in June, 2003, in which teachers learned together by carefully examining student work.
While most of these models of collaborative assessment or collegial review have been developed by university researchers to be implemented in the schools and replicated from site to site, the Iowa Writing Project’s collegial review was developed locally and designed to support the principles of teaching and learning advocated by the Writing Project. The result is a multi-layered approach to collecting and studying evidence of student learning that supports individual teacher reflection and growth as well as providing rich, complex data to Iowa Writing Project leaders and stakeholders.
This proposal suggested developing an approach to collegial review of student writing that would be specific to our campus’s needs—one based upon the Iowa Writing Project work and incorporating aspects of other new approaches to collaborative assessment, along with key elements of Truman’s Sophomore Writing Experience.
The objectives of this Pilot Project were to develop, try out, and refine protocols and procedures for a collegial review of student writing at Truman, as well as to make suggestions for implementation of a full collegial review in the future. A descriptive assessment of this sort will provide valuable qualitative data to complement the quantitative data provided by other assessment approaches in place at Truman. Such a qualitative review has been proposed as one part of a comprehensive program for writing assessment approved by the Undergraduate Council.
When fully implemented, the collegial review will have as objectives: 1) supporting professional development of faculty who teach writing within their courses, 2) providing constructive feedback to both faculty who teach writing and student writers concerning the quality of student writing, 3) facilitating the improvement of teaching and learning of writing at Truman, 4) creating an environment that supports faculty inquiry into what constitutes compelling evidence our learning outcomes are being met, and 5) establishing a method for providing that evidence to all University stakeholders.
Activities in Preparation of the Workshops
The Writing Assessment Committee refined and revised proposed methods and activities as part of their ongoing work during the spring semester, 2004.
In particular, members of the committee presented an abbreviated version of the Collegial Review to colleagues at the Center for Teaching and Learning’s Faculty Development Lunch on March 17. In this mini-workshop, project coordinator Barbara Price introduced the principle of using a guiding question to frame collegial inquiry. As an example of how this process can work, she gave participants a set of papers written by students in one of her literature classes (with students’ names removed). She asked them to read and annotate these papers, using as a guiding question, “What evidence do you see in these samples of my students’ writing that they are meeting Truman expectations for the Aesthetic Mode of Inquiry—Literature?” That question and a copy of the corresponding Aesthetic Mode of Inquiry expectations were distributed for participants’ reference (Appendix A). The resulting discussion was interesting and informative. Asked to consider the possibility of a university-wide implementation of such a collegial review, the participants’ primary expressed concern was whether and how adequate time could be found for both the reading and discussion. The committee reviewed faculty responses to this workshop, as well as faculty feedback from various informational activities.
Plans for the Pilot Workshop were necessarily revised when the announcement of the grant occurred later than the planned beginning of preparatory activities. Keeping to the intention of the original proposal, plans were modified as circumstances allowed. Rather than one large Pilot Workshop, the project coordinator, working with the committee, proposed two smaller workshops, one to be held during the summer, the second during fall semester, 2004. The first workshop was completed, with further modifications; the second of the planned smaller workshops has not yet been conducted.
Activities in Preparation for the Pilot Workshop
In preparation for the May Pilot Workshop the project coordinator and committee members invited professors teaching writing-intensive courses to apply to participate in Pilot Workshop. Three participants were selected for the May Pilot Workshop, with the goal of representing as many university divisions as possible. Those faculty members, Wendy Miner (Education), Diane Johnson (Language and Literature), and Elaine McDuff (Social Science), met in two separate half-day sessions during the summer of 2004 (May and August) to explore the desirability of initiating a collegial review of student writing at Truman. Professors Miner, Johnson, and McDuff shared copies of student writing from their writing-intensive classes, as did the project coordinator, Barbara Price (Language and Literature). Also joining the discussion was TSU Assessment Specialist, Sue Pieper.
Despite the small size of the group, there was considerable difficulty in finding a mutually agreeable time to meet. When one member of the group had to change plans because of a funeral, the group was unable to find a single day when they could all meet, so they decided to meet in two half-day sessions.
Project participants were asked to collect and contribute samples of their student work representing a range of demonstrated student accomplishment. In addition, they were asked to include contextualizing information such as syllabi, assignments, and rubrics or other articulation of grading criteria.
Given the small size of this initial group, participants did not always remove names and other identifying marks from student papers. They did speak of the necessity of treating students’ work with respect and recognizing it as the students’ property, and they concluded that, in the future, removing students’ names would make the process more objective.
A photocopy of each of set of papers was prepared for each project participant.
Activities During the Pilot Workshop
All activities planned to occur during the pilot workshop were completed, although with fewer than the planned number of participants:
· review workshop objectives
· establish draft protocols
· form questions to guide inquiry, with a focus on discovering evidence that students’ writing demonstrates competency in meeting the University goals
· following draft protocols, participants read, discuss, and write brief responses to student writing samples
· participants provide feedback about the process, particularly the effectiveness of the protocols
Work began with a discussion of the Pilot Project goals and the relation of this project to the recommendations of the Writing Assessment Committee for a three-part approach to writing assessment at Truman State University. The project coordinator also provided materials describing other professional inquiry projects in various settings.
Establishment of agreed upon protocols is essential to any collegial review. Accordingly, the following protocols were discussed and implemented for this Pilot Project:
1) Discussion will be contained by the focusing question: “What evidence do you see in these samples of my students’ writing that they are meeting Truman’s learning outcomes for writing-enhanced Liberal Studies Program courses?” (Appendix B)
2) Our reading will be seen as a search for evidence of accomplishment (what students do, rather than what they fail to do).
3) As we work, we will keep in mind the dual goals of the project and think about the challenges to large-scale implementation of this process. How can Truman faculty use this process to:
· inform instruction (focus on one instructor’s writing samples)
· collect evidence for a larger audience and purpose (using samples for many and various classes)
4) Participants will limit our explanations/apologies/explanations of the materials presented from their classes, allowing those materials to speak (or not) for themselves.
As participants read, we color-coded the three general outcome areas of the outcomes for writing-enhanced courses: cognition, process, and product. In general, we found it much easier to discover evidence of cognition and product in the student papers than of process. Another class, or another set of papers, might have included multiple drafts or other evidence of revision, but none of the papers we examined included this evidence of process. On the other hand, even with only one draft, we believed we could discern evidence of copy-editing, but we had no way of tracing improvement in mechanics from an earlier draft to the one we read.
The papers Wendy Miner provided illustrate very well one of the challenges we face in conducting a university-wide Collegial Review: many of the papers are long, and reading them takes time. When we read together, this means some people are waiting for others to finish, while the slower readers may feel awkward or rushed. It also means that much of our shared time is taken up by reading. Reading ahead of time is one possible solution, but that means losing the freshness of reading just prior to discussion.
We experimented with reading only a portion of the longer papers. Participants received the following suggestion: “Please read only as much of the papers as you believe you need to in order to meet our objective: discovering evidence that the university’s objectives for WE classes are being met. (Perhaps that’s 3 of 25 pages, with the remaining pages only being skimmed; perhaps you’ll find you need to read far more.) Please annotate the papers, so you can point easily to that evidence. And please make some notes about how you read: how much of the papers you found you needed to read, the procedures you used.”
As we discussed the reading, some readers insisted that reading only a portion of each paper made them very uncomfortable, and we also acknowledged that we find ourselves wanting to complete any interesting paper. One reader reported that she found she needed to read both the opening and conclusion of each paper carefully, but could read the middle section more quickly.
A key discovery within our discussions was that important instructions about writing were often delivered outside the syllabus, leaving no record of connections between the instructor’s goals and the university WE Outcomes. Participants quickly realized that making connections to the WE Outcomes explicit could often be accomplished quite easily. A few simple revisions or additions to the syllabi could make the expectations concerning student writing much clearer, as well as clarifying the relation between the expectations of the class and those of the university.
The following represent the kind of comments participants offered in response to the project’s focusing question, “What evidence do you see in these samples of my students’ writing that they are meeting Truman’s learning outcomes for writing-enhanced Liberal Studies Program courses?”
Examples of Comments from Review of Syllabi
Discussion with process
Journaling
Applied paper
Recommendation students seek Writing Center help
Deeper level of analysis in applied papers
Assignments build on each other—journal leads to applied papers leads to final paper
Examples of Comments from Review of Papers
Paper W
Readable style
Reasonably well organized
Second paragraph was organized well, flowed within
Supported some of the claims with reasons, especially in support of discussion
of. . .
Some risks with punctuation
Some variety in sentence openers
Paper T
Sophisticated construction: from general to specific using writer’s own experience
Well developed explication of . . .
Uses connectives
Summarizes ideas as paragraph openers or closers
General example, supported by specific example
Audience would get a sense of the writer as a person
Paper F
Use of connective phrases
Use of clear graph to illustrate point
Use of supporting materials from popular culture
Specific examples to support claims
Use of solid academic resources with frequent references
Clear thesis
Variety of references from the Web and academic journals
Connective phrases within the paper join topics clearly
Paper is clearly organized with articulate beginning and ending that tie it together
Strong opening sentence
Thorough coverage of topic
Well organized
Analysis and synthesis
Evaluation of what was important to the argument
Clearly making an argument
Clear thesis
Lack of major mechanical problems suggests revision
Very complete bibliography
Conclusion seemed to bring the paper back together
Audience—written so a reader outside the field could understand
APA style was handled with skill
Paper M
Straightforward report style was maintained consistently
Organization—clear
Good command of mechanics and APA style
Some analysis and evaluation early in the paper—reviewing different definitions
Thesis is pretty clear, although the claim was. . .
Examples are a little “off”
Interesting
[More analysis of the topics, more student questions of sources would strengthen her analysis]
[Student development of her own thesis from these materials would strengthen the paper.]
As a lit review, there is analysis of each source.
Paper J
The abstract as a task was handled well.
Good rationale for studying the topic
Good statistic support
Definition; considered alternative definitions, but without examples
Organized at the general level
Good connectives between topics
APA style under control
Fairly thorough bibliography
While these observations in response to the focusing question were intrinsically valuable to the participants of the workshop, their value to the University would be increased by collecting them in systematic fashion. Beginning the design of such a collection/presentation procedure is an intended goal of the next Pilot Workshop.
Observations from Pilot Project Workshop
Participants were generally enthusiastic about the experience, citing the satisfaction of learning what is happening in others’ writing-intensive classes and receiving useful feedback about their work and their students’ work.
The professional growth afforded by this sort of activity was evident. Also evident was the difficulty of providing adequate time for both the reading and the discussion.
Although this first workshop did not deal directly with the challenge of collating and presenting the qualitative data describing students’ writing, participants did express awareness that this would be far from a small task.
Participants were struck by the distinction between grading and reviewing papers for other purposes.
Participants also found a sharp distinction between grading within the context of a class and reading papers as individual, decontextualized artifacts.
Readers were aware of separating the writer from the artifact in a way not possible in their classes.
Readers remarked that seeing instructor comments on some papers influenced the way they read those papers.
Removing student names and other identifying marks was seen as desirable; doing so “allows us not to be influenced by gender, ethnicity, knowledge of the person.”
Some participants had difficulty focusing on searching for evidence of accomplishment, rather than identifying errors or other weakness in student work.
Participants expressed the desirability of including multiple drafts of papers when available, particularly as evidence that the WE Outcomes process goals are being met.
In review of the Pilot Project, one participant offered the following:
“I
wonder if it would be helpful to have people bring papers for a class that they
are going to teach next so they can go back energized ready to make positive
changes.”
“I thoroughly enjoyed
seeing the process of people reading and critiquing etc. I think the process is
very valuable for several reasons:
1. it shows a commitment on the part of the university to ensuring writing
enhanced classes make sense to instructors and students alike.
2. it familiarizes instructors with the goals of writing enhanced courses (once
again)
3. I loved the feedback that I got on my papers and am going to try and create
change for both courses.
4. it validated what I thought about the writing
5. it made all my hard work seem more important, because it does reflect in my
students' products
6. I got ideas about connectives, editing and critiquing and other assignments,
seeing others' expectations about writing assignments, and got ideas to improve
my writing requirements/syllabus, etc.
7. it was wonderful seeing what other disciplines require with writing
8. I got to catch up with old friends I had not seen in awhile.”
The results of this small Pilot Workshop suggest that Collegial Review has much to offer Truman faculty. However, the difficulty of finding a meeting time for even such a small group illustrates the primary challenge to instituting a large-scale Collegial Review at Truman State University: scarcity of time and other resources.
Recommendations for Future Actions
With the creation of the recently constituted Writing across the University Committee, the mechanism is in place for continued exploration of the most effective ways of implementing Collegial Review practices on the Truman State University campus.
Working within this committee and in consultation with its members, and with the support of the University’s Vice President for Academic Affairs, the project coordinator will complete the remaining portions of the Pilot Project. One immediate task is to provide information about this approach to assessment to the new committee members, while preserving the recommendations of the previous committee. With that in mind, this report has been shaped to incorporate much of the language of the original proposal, hoping to give the new members of the committee, as well as any other readers, a sense of the intention of the project. In addition, the Chair of the WAU Committee and I would like to engage new committee members in the experience of a Collegial Review to fully inform their understanding of the possibilities such an approach holds.
As reflected in the revised timeline (Appendix C) I request, with the support of the Chair of the Writing across the University Committee, the University’s support, via extension of the grant’s calendar, in conducting two specific workshop activities: (1) a Pilot Workshop in which we apply the practices and principles of the Collegial Review to the needs of a particular academic program, with new members of the WAU committee as participants and (2) a second summer Pilot Workshop, refining the procedures and protocols drawing from the experience of the first summer workshop, with faculty representing other disciplines and divisions specifically invited to participate. The information gleaned from these two Pilot Workshops will then be made available to the Writing across the University Committee for their consideration and further recommendations, and a summary report forwarded to both Dean Heinz Woehlk and Vice President Garry Gordon. Results of the Pilot Project will be shared within the University community and proposals offered for sharing the results with audiences beyond Truman.
This approach to professional inquiry has much to offer Truman, supports many of our ongoing activities, is consistent with Truman goals, and has the potential of expanding our sense of the meaning and range of assessment. The challenges are largely ones of time and resources.
Brown, Pamela U. “Looking at Student and Teacher Work Collaboratively.” The Voice (the newsletter of the National Writing Project) 8.5 (2003): 18-19.
Carini, P. “Building from Children’s Strengths.” Journal of Education 168 (1986) 13-24.
Cochran-Smith, M. and S. Lytle, eds. Inside/Outside: Teachers Research and Knowledge. New York: Teachers College Press, 1993.
Gore, J. and K. Zeichner. “Action Research and Reflective Teaching in Preservice Teacher Education: A Case Study from the United States.” Teaching and Teacher Education 7 (2) 119-136.
Huot, Brian. (Re)Articulating Writing Assessment for Teaching and Learning. Logan: Utah State, 2002.
Little, Judith Warren, Maryl Gearhart, Marnie Curry, and Judith Kafka. “Looking at Student Work for Teacher Learning, Teacher Community, and School Reform.” Phi Delta Kappan 85.3 (2003): 185-192.
McDonald, J. P., N. Mohr, A. Dichter, and E. D. McDonald, eds. The Power of Protocols: An Educator’s Guide to better Practice. New York: Teachers College Press, 2003.
McEntee, G. H., J. Appleby, J. Dowd, J. Grant, S. Hole, and P. Silva, with J. Check. At the Heart of Teaching: A Guide to Reflective Practice. New York: Teachers College Press, 2003.
Price, Barbara. Portfolio Review: Dialogue Informing Practice. Cedar Rapids: Iowa Writing Project, 1996.
Descriptive Review Process
Prospect Archive and Center for Education and Research
North Bennington, Vermont
The Evidence Project, Project Zero
Harvard
The Philadelphia Education Fund
Tuning Protocols developed by the Coalition of Essential Schools and the Annenberg Institute for School Reform
New Standards documentation of student work
Protocols
Harmony School Education Center
Bloomington, Indiana
Back to Top
Appendix A
What evidence do you see in these samples of my students’ writing that they are meeting Truman expectations for the Aesthetic Mode of Inquiry—Literature?
Students who successfully complete the Aesthetic Mode of Inquiry--Literature will develop:
· the ability to interpret a text by drawing on some of the following techniques: close, active, reflective reading; past experiences; primary and secondary sources; other critical approaches; and
· the ability to analyze the structural elements and relationships within a text or between various literary genres in order to explain how authors create responses in readers.
In addition, students who successfully complete this Mode of Inquiry will show some of the following features in their writing, observations, questions, and discussions:
· familiarity with a significant number of influential and representative works OR familiarity with a significant number of works of an influential author(s);
· understanding of the diversity of human experience and creative expression presented in literature;
· situating works into historical, cultural or intellectual contexts OR seeing literature's connections to other disciplines OR seeing how other disciplines can inform the reading of literature;
· analyzing the values in the literature read; and
· recognizing how our own culturally and experientially derived assumptions shape our reading of a literary text.
Back to Top
Writing-Enhanced Courses
The projected outcomes of students' skills, habits, and attitudes, while distinguishable, are not separable; they blend together to produce the ability to write well and think critically. Cognition, writing process, and the written product interact and mutually reinforce one another.
As a result of Writing-Enhanced Courses, students will:
1. Cognition
A. use writing as a mode of learning as well as a method of communicating what was learned;
B. be able to generate, organize, and communicate information and ideas fully, clearly, and cogently;
C. exhibit critical thinking such as the ability to analyze, synthesize, evaluate, and reflect;
D. show audience awareness;
2. Process
A. engage in deep revision, closely examining and further developing the reasoning in the writing;
B. assess their own writing to uncover strengths and concerns, and be able to generate strategies for improvement;
C. solicit external critiques of their writing to guide revision;
D. as a regular habit of their writing process, copy-edit their own work for mechanics, style, and coherence;
3. Product
A. be able to write clear, coherent, and well organized prose for a targeted audience;
B. demonstrate a command of syntax, style, and tone appropriate to the task; and
C. exhibit mastery of punctuation, usage, and formatting conventions.
Back to Top
Timeline (revised)
Ongoing Regular meetings of Writing Assessment Committee/Writing across the University Committee
March 17, 2004 Faculty Development Luncheon Series Mini-workshop
April 22, 2004 Grant award
May and August
2004 Pilot Workshop 1 (complete)
February 2005 Draft report, Pilot Workshop 1, prepared, submitted to WAU for review and revision (complete)
February 2005 Report, Pilot Workshop 1, submitted by WAU to DIG, VPAA
Spring semester
2005 Pilot Workshop 2: review of student writing from Nursing Program; new WAU committee members as participants
Summer term
2005 Pilot Workshop 3: review of student writing; faculty from across University invited as participants
August 2005 Report of all Pilot activities prepared, submitted to WAU for review and revisions
September 2005 Report of all Pilot activities submitted by WAU to DIG, VPAA
AY 2005-06 Presentation of results for Truman audiences and at various regional and national professional meetings