2003 Assessment Grant Report
Candy Young, Ph.D., Professor, Political Science
Debi Cartwright, Ph.D – Associate Professor, Marketing
TABLE OF CONTENTS
Truman State University has been observing lower than expected junior exam results from its students for at least the last decade. The Assessment Committee attributes these results to a lack of student motivation. Using data from all students taking the junior exam in Spring 2003 (approximately 550) and a sample of 150 students who took the exam in Fall 2002, this study will examine the influence of new testing processes put in place for Spring 2003. The overall effect of the new initiatives will be analyzed by comparing 2002 and 2003 data. The specific effects of taking the test early in the semester, during the day rather than in the evening, and in the presence of faculty proctors will also be analyzed. Results will inform the University’s continuing efforts to enhance student performance on the junior exam. As these results have been directly tied to University funding in many years over the last decade, the results could have direct effect on University appropriation levels. Results of this study will also be of interest nationally as more universities participate in low-stakes assessment of student learning. Back to top
Truman’s Assessment Committee has been concerned about student motivation on standardized examinations for more than a decade. Several attempts have been made over the years to examine student motivation through a subcommittee of the assessment committee, an analysis of student course-taking patterns associated with performance on the junior test, and student responses to questions during the annual interview project. While the committee concluded that student motivation was less than desired, there was little agreement about how to improve it. Back to top
Faculty concern emanated from review of junior test data that showed performance levels were consistently below estimates of student ability as indicated by student performance on entering test scores. The ability of the University to use the junior test data for evaluation of student learning and the effectiveness of the Truman curriculum was compromised by data that was not genuinely reflective of student abilities. Furthermore, this data was and still is publicly available to education and political officials in the state of Missouri. For over a decade every state university has been expected to report student performance on a nationally recognized, general education exam, and student performance on the exam has been a significant component of the state’s performance funding system. Although the state has not funded the “Funding for Results” program during the recent years of fiscal crisis, it should be noted that other state universities out-perform Truman on general education assessment and that this has been noted by at least one president of another state university as a way to suggest that his institution was relatively underfunded.
Discussions with students about their motivation on the Junior Exam suggested that students thought the assessment was solely for purposes of the and that the assessment provided them with no benefits. They also thought the exam results were not part of their academic record and had no consequence for them personally. Students reported that faculty members, advisors, and/or student proctors communicated to them that the exam had no consequences for them.
In January 2003, the Assessment Committee decided to launch a set of pilot initiatives to encourage enhanced student motivation. This program included the:
1. Scheduling of more opportunities for students to take the exam during the day, especially by utilization of classrooms on January Conference Day.
2. Encouragement of faculty members to proctor examinations, especially on January Conference Day.
3. Reduction of the number of sections taken on the CAAP examination from four to three.
4. Recognition of students who do well on the examination by:
· Special designation on student transcripts for those students who achieve “distinction” (above the 50th percentile) and “high distinction” (above the 80th percentile) on general education assessment.
· Reference to their exam performance on job and graduate school applications and in faculty letters of recommendation.
· Favorable consideration on appeals for scholarship renewals from students with high performance on the exam.
5. Receipt of a letter from the university president inviting students to sit for the exam and reinforcing the exam’s importance to the university and the benefits for students. (See Appendix A)
6. Encouragement of students to use the examination as an opportunity to practice for similar examinations they will take when they are applying to graduate and professional schools in the future.
7. Pre-test reminder of various reasons to take the test seriously in a statement that was read to students at the beginning of each testing session. (See Appendix B)
This research project was designed to identify the effectiveness of the initiatives in the motivation plan. The 2002-2003 academic year provided an outstanding opportunity to student the motivation strategies. This was true because fall students took the Junior Exam prior to implementation of the motivation plan and their performance could be compared with those who took the Junior Exam in the spring of 2003. In addition, some of the students who took the Junior Exam in spring 2003 did so with faculty proctors and others took it without. Similarly, some took the exam during the day and others at night. The rest of the initiatives could be studied by comparing student performance for fall prior to adoption of the motivation strategies with student performance for spring. Back to top
Figure 1: Motivation Model
The above model depicts the independent and intervening variables hypothesized to affect student performance on the junior tests. This model suggests that the effect of the treatment is primarily a change in the student’s effort on the exam and this increased effort leads to higher performance on the exam. The following hypotheses were tested using correlation data and a multiple regression model:
Ø Students who took the exam with a faculty proctor present will indicate a higher level of effort given on the test.
Ø Students who took the exam with a faculty proctor present will score higher than students who took the exam with only student proctors present.
Ø Students who took the exam during the day will indicate a higher level of effort given on the test.
Ø Students who took the exam during the day will score higher than students who took the exam in the evening.
Ø Students who took the exam early in the semester will indicate a higher level of effort given on the test.
Ø Students who the exam early in the semester will score higher than students who took the exam later in the semester.
Ø Students who took the exam after the motivation plan was put into place will indicate a higher level of effort given on the test.
Ø Students who took the exam after the motivation plan was put into place will score higher than students who took the exam prior to implementation of the motivation plan.
The data for this study includes almost the entire population of students taking the junior exam during the 2002/2003 school year (N=539) because we had access to all of the scores and the numbers were manageable. However, we were only able to obtain the ACT scores for 504 of these students. We did not include the data points for cases with missing values. Therefore, we conducted the analysis based on the 504 complete cases.
In the multiple regression model, our independent variables consisted of:
Ø Year – This variable was coded as a dummy variable. This distinguished the students who had taken the exam under the control conditions (before the motivation plan was implemented) and those under the experimental conditions (after the motivation plan was implemented)
Ø Time of day – Set up as another dummy variable, this variable identified those students who took advantage of the daytime test-taking opportunities versus those who took the exam at night.
Ø Week – Scores from 1 – 10 were assigned depending on which week in the semester the students took the exam. This variable was later dropped from the analysis because it was too highly correlated with time of day. Most of the daytime opportunities were presented at the beginning of the year during the January conference.
Ø Proctor – This was also coded as a dummy variable differentiating the students who took the exam with a faculty proctor present versus those who took the exam with student proctors present.
Ø Motivation Plan – This was coded as a dummy variable and separated those who took the exam in the fall from those who took the exam in the spring after all of the motivation strategies were in place.
We used two control variables in the model, ACT score and gender, to control for factors known to influence scores on general education assessments. This helped us rule out these variables as alternative explanations of our findings.
There were two dependent variables analyzed: effort and performance. We looked at the effects of our independent variables on the students’ effort. This effort score was a one question self-reported 4-point Likert scale. Students indicated whether or not they had given the test their best effort, moderate effort, little effort or no effort. During the CAAP tests this question was asked after the completion of each subtest. The student’s test score measured performance. Back to top
Each of the hypotheses was supported. The self-reported effort score on the exams is the best predictor of the resultant scores. ACT is also an important predictor. The motivation strategies in general led to greater effort and higher test score and were statistically significant.
It proved beneficial to include more options to take the exams during the day, as taking the exam during the day correlated with higher test scores and was statistically significant in the model.
Faculty proctors were beneficial as having a faculty proctor present was correlated with higher self-reported effort and higher scores and was statistically significant in the model.
To put the findings in a more applied form, the following chart shows the substantial improvement made in student performance on the junior exam following the implementation of the motivation plan. The bar chart below demonstrates that gains were made for every test (Academic Profile and the Writing, Math, Reading, Critical Thinking and Science subscores on the CAAP) increasing the percentage of students scoring above the 50th and 80th percentiles. Back to top
Table 1: Fall 2002/Spring 2003 Comparison
Motivated students’ test scores are more useful for evaluation of student learning by the university, its faculty and advisors, and individual students. Aggregate performance reports to state officials are also more reflective of Truman’s curriculum, learning environment, and student achievement levels. Ideally, students will be able to see exam results as a complement to course grades and will use the exams as practice for the standardized exams required for graduate and professional schools. Faculty can also make better use of the results in conjunction with other assessments to inform discussions and decisions regarding curriculum and to increase the evidence provided in letters of recommendation written for students.
Given the results of this study, it becomes prudent for the university to offer more opportunities for daytime testing and to integrate more testing times with faculty proctors. Having the president of the university introduce the nature and purposes of the exam and having a faculty member remind students of them prior to taking the exam should also be continued.
Ultimately, this study yields evidence to support the importance of student motivation for assessment testing. The measures of the pilot project can only be seen as a small step toward enhanced student motivation. The larger campus culture is much more complex, and the University needs to explore means to make the use and importance of assessment more visible to students, faculty, and staff. If the data is not valued, integrated into campus processes, discussions, and decisions; the students are not likely to take the exams as seriously as is necessary to generate accurate results. This in turn, undermines the whole assessment system and causes the University’s education programs to appear less impressive to the public officials in the state who do review this data. Back to top
For a printable version of this report, please click here.