Dan Hite, Jeanne Mitchell, Don Kangas
Table of Content
Reliability of Current Biology MFAT Assessment Data
Reliability and Validity of Senior Portfolio Project Assessment Data
The Appropriateness of the Biology MFAT for Assessment of the Biology Core
Appendix II: Biology Portfolio Prompt
The Biology discipline has established three pillars, areas of learning, for Biology majors: Scientific Knowledge, Scientific Process, and Oral and Written Communication. To assess student development within these areas, a reliable comprehensive mechanism matching clearly stated and measurable curricular outcomes with appropriate and reliable assessment tools is needed. Among the assessment tools currently available, the Major Field Achievement Test (MFAT) and the Senior Portfolio Project are likely candidates for evaluation of the development of our Biology majors regarding these pillars. Determining the reliability of data generated by these assessments and the suitability of these assessment tools to measure specific outcomes within our pillars is a logical first step toward development of an efficient and comprehensive assessment mechanism. We foresee test results from the Biology MFAT being useful for evaluation of outcomes within the Scientific Knowledge and possibly the Scientific Process pillars; however, reliability of test results must first be established statistically. Recently, measures have been taken by the Biology discipline to improve student performance and accompanying these changes are survey data that can be analyzed to determine whether attitude and performance have changed positively in response to these measures. Examination of the general makeup of the Biology MFAT with regard to question type, numbers of questions of different type, and question format will allow determination of the applicability of exam results to our specified outcomes. Samples of scientific-reasoning submissions and Portfolio cover letters of our students from the Senior Portfolio Project will be examined for reliability by correlating quality of submissions with indicators of student attitude and understanding as reflected in cover letters and statements accompanying submissions. Submission content and scores will be evaluated separately to determine the applicability of submissions and scoring results to our specified outcomes. Back to top
2004 Report of the Current State of Assessment within the Biology Discipline
Submitted by Daniel Hite, Don Kangas and Jeanne Mitchell
This report outlines the purpose and major objectives of our review of the Biology discipline's assessment process, preliminary findings of the authors based on this review, and recommendations to the Biology discipline in light of our findings.
Our overall purpose is to develop processes that will allow curricular practice in Biology to be driven by performance indicators offered through analysis of assessment data. The processes should match clearly stated and measurable curricular outcomes with valid and reliable assessment tools. Our immediate purpose is two-fold: 1) to evaluate the reliability and validity of existing assessment data from two sources, the Biology Major Field Achievement Test (MFAT) and the University-wide, Senior Portfolio Project; and 2) to evaluate whether existing student performance data from these two sources can effectively measure specific curricular outcomes in three critical learning areas, or pillars, previously established by the Biology discipline: Scientific Knowledge, Scientific Process, and Oral and Written Communication.
Below, we separate our presentation into three elements: 1) establishing the validity of the MFAT and portfolio, 2) determining if these tools are appropriate for assessing the Biology core curriculum, and 3) matching these tools, where appropriate, to the pillars. Perhaps we should rephrase this paragraph to mirror the way it is presented below? It is our hope that these evaluations will allow the Biology discipline to determine whether current assessment tools may need to be adjusted, how they may need to be adjusted, and whether other (additional) assessment tools are needed to match our established curricular outcomes.
Objectives and Preliminary Findings
Below are given the three stated objectives of the proposal submitted to the review committee. Following each objective, stated in italics, is a summary of our action, findings and recommendations, where appropriate. Back to top
We plan...
To determine the reliability of current University assessment data collected on Biology majors through the Biology MFAT and the Senior Portfolio Project
Reliability of Current Biology MFAT Assessment Data:
Action: A survey addressing student attitude toward the Biology MFAT has been conducted by the Curriculum committee of the Biology discipline for three consecutive years for each fall and spring semester administering of the exam (2000/2001, 2001/2002, and 2002/2003). The purpose for conducting the survey was to determine whether student attitude was impacting student performance on the Biology MFAT. (Reports of behavior consistent with poor student attitude (e.g., simply filling in test sheets, leaving the exam early, and comments regarding effort) prompted this faculty evaluation of student attitude and effort.) The survey was conducted immediately following completion of the exam (fall 2001-2003 test dates), or within a week of taking the exam (2000 and spring 2001 test dates). The survey was primarily a series of statements that required responses graded along a Likert scale ranging from strongly agree to strongly disagree (see Appendix I: Biology MFAT Survey). Responses were converted to numerical values that could then be analyzed statistically to determine whether correlations we believed to be important were statistically significant (e.g., student GPA and MFAT score, MFAT score and student attitude indicators).
Concomitantly,
beginning in the spring of 2001, the Biology faculty began proctoring the exam
in Magruder Hall, following the exam with a pizza social, also attended by
Biology faculty. This latter initiative was undertaken to demonstrate
faculty interest in the quality of student performance on the exam. To
further encourage quality student performance on the Biology MFAT, the Biology
discipline sought and gained approval from the Vice President of Academic
Affairs office to establish a minimum percentile score requirement of twentieth
percentile as a requirement for graduation of all Biology majors.
Findings:
Concurrent Results of MFAT Survey Responses and MFAT Performance
Statistical correlations between MFAT scores and GPA and between MFAT scores and question eleven (11) of the MFAT survey, "If the University listed my MFAT score on my transcript I would have worked harder while taking the MFAT exam.", changed coincident with faculty intervention in the MFAT testing process. See below figure and table.
significant correlations are encircled
* Correlation coefficients are significant at 0.05 > p > 0.0?, N = number of students
Intervention began during the Spring 2001 testing period and presently continues. Significant positive correlations between MFAT score and GPA have been recorded for each testing period since the intervention began, excepting results from Fall 2001, which were not assessed. The trend for correlation coefficients between MFAT score and question eleven of the Biology MFAT Survey was counter. To understand the importance of this trend, it is important to realize that a numerical rating of one indicates strong agreement with the statement of question eleven and a numerical rating of nine, a strong disagreement. Therefore, a strong positive correlation could be interpreted as students with higher scores would not have worked harder while students with lower scores would have worked harder if the scores went on their transcripts. (Note: This action was considered by the Biology faculty, but not taken; however, it was thought that such a consideration by the students would prompt a sincere response regarding their level of effort.) A lack of significant correlation between these factors is an indication that student attitude no longer is affecting MFAT scores. Both trends indicate that our intervention was efficacious.
Although the correlation coefficients for MFAT and GPA account for less than or approximately equal to twenty-five percent of the variability in MFAT scores, we are satisfied with this level of correlation. Indeed, a very high correlation would be a red flag to us given that the Biology MFAT tests primarily knowledge retention and not higher-order thinking (e.g., process and comprehension). (Our analysis of the Biology MFAT is addressed later in this report.) Our curriculum has three emphases, one of which is a sound knowledge base and the other two being communication skills and skills related to the scientific process (e.g., data analysis, experimental design). A high correlation for MFAT score and GPA would be an indication that their grades were being driven predominately by their ability to retain knowledge, and may suggest a weakness in our other two areas of emphasis. The following student survey data (Q.1 and 5) seem consistent with the idea that knowledge retention is only a part of the overall education experience as perceived by our majors.
Q. 1: Biology core curriculum prepared me for the MFAT
1= strongly agree 5 = unsure 9 = strongly disagree
Q. 5: MFAT score will reflect the quality of education in my major that I have received
1= strongly agree 5 = unsure 9 = strongly disagree
Student response to question one clearly indicates that students agree that the Biology curriculum prepared them for the Biology MFAT (s.e. ≤ 0.15 in s03) and their response to question five indicates that students are unsure that the MFAT reflected the quality of education offered in the Biology major (SE ≤ XX). Question one indicates that from a student perspective, the Biology curriculum, which emphasizes a knowledge foundation, communication skills and understanding of the scientific process, provided the necessary background to perform satisfactorily on the Biology MFAT (a predominately knowledge-base assessment tool); however, students also indicate that the Biology MFAT does not reflect the quality of education received through the Biology curriculum. When taken together, it would seem that from the student perspective there is more to the Biology curriculum than simply knowledge retention. The validity of this interpretation could be determined once additional assessment tools are in place to assess communication skills and skills related to understanding/application of the scientific process. If valid, one would predict that questions similar to questions one and five, but directed to the other two areas of emphasis would yield appropriate responses.
Biology MFAT Average Percentile Scores by Semester:
The above chart demonstrates that recent testing periods have yielded more stability in test results based on mean percentile performance of our students. The data, preceding f01 appears to have a slight negative slope but there is no significant negative correlation (the slope was not significantly different from zero).
Frequency of Students Above 50th and 80th Percentile Ranks (University Indicators):
The above chart depicts monitoring of student performance at the fiftieth and eightieth percentiles from 1992 to present and has revealed three interesting trends. First, student performance (percentage of students achieving at the 50th and 80th percentiles) generally declined in the mid-1990s. Second, student performance has improved coincident with faculty intervention beginning in 2001 (arrow). Third, student performance at the 80th percentile has been consistent (values in oval) since inception of faculty intervention; however, performance at the 50th percentile is still variable.
Conclusions
1) Increased faculty involvement in the administering of the Biology MFAT has positively affected student performance.
2) With regard to student performance, the Biology MFAT is now a reliable indicator of students' capacity to perform, rather than their willingness to perform.
Evidence summarized below supports these two conclusions:
during the fall 2000 testing period ~65 % (corr. coeff. of 0.81) of the variability in student MFAT scores was attributable to student effort and with the beginning of faculty intervention in spring 2001 there has not been a significant correlation between student effort and student performance (MFAT scores)
during the fall 2000 testing period there was not a significant correlation between GPA and MFAT scores and with the beginning of faculty intervention in spring 2001 there has been a significant positive correlation between GPA and MFAT scores
inconsistencies in student performance (overall mean percentile and frequency at 80th percentile) have diminished appreciably; consistent student rankings have been achieved
Recommendations:
1) Continue to monitor GPA, student attitude and MFAT scores. A positive correlation between GPA and MFAT scores and no significant correlation between student attitude and MFAT scores can serve as a standard for acceptable reliability of MFAT scores as an indicator of student performance.
2) Biology faculty should remain committed to their expressed interest in student performance on the Biology MFAT.
Reliability and Validity of Senior Portfolio Project Assessment Data: The Biology faculty is seeking to demonstrate growth in the areas of oral/written communication and of understanding/application of scientific processes. Quality submissions from early and late in students' progression through the overall Biology curriculum are most desirable. The purpose of examining portfolio data gathered for the University Senior Portfolio Project was to determine whether a suitable late (best effort as called for by the Portfolio prompt) submission was currently being assessed in a manner that could partially meet our needs, understanding that an early submission would still need to be collected and assessed.
Action:
We examined cover letters and Scientific Reasoning submissions of Biology seniors submitted over the past five years (1999-2003). A sample of twenty submissions for each year were requested and received from Doug Davenport, director of the University Portfolio Project. Through instructions for the cover letter, students were instructed to address their process of assembling their portfolios (e.g., time spent on task) and to reflect on their submission and how it addressed the prompt (i.e., understanding of process). Cover letters were scored based on their expressed effort toward the exercise (e.g., time committed to the task), expressed understanding of the prompt, and expressed attitude toward the process. Attitude was assessed by comments specifically indicating a positive or negative experience. If no such comment was made, it was considered not addressed. Submission sources were identified as being in one of the following categories: freshman Biology course, sophomore Biology course, upper-level Biology course, independent research project, or not Biology (Agriculture, Chemistry, or Physics).
Findings:
Ninety-eight of one-hundred portfolios that were received did contain Scientific Reasoning submissions, two did not. The source of submissions and number of each type are given in the following table:
Submission Source Number of Submissions
Freshman 12
Sophomore 6
Upper-level 55*
Independent Research 13*
Not Biology 12
Total 98
* submissions deemed as valid for our assessment needs
Of the ninety-eight submissions reviewed, ten did not contain cover letters. These submissions were scored as low effort, but were not scored for understanding or attitude because no indication was received. Qualitative ranking of effort, understanding and attitude are given for each portfolio in the following table:
Indicator
Assessed Category High Low Not Addressed
Effort 44 42 12
Understanding 49 15 24
Attitude 50 20 18
The readers noticed a general trend in that students demonstrating very positive attitudes, high effort and good understanding of the prompt also indicated that their Senior Seminar instructor took time to explain the importance of the Portfolio Project and collected submissions one at a time throughout the semester.
Conclusions:
Twelve out of one-hundred randomly selected portfolios submitted by senior Biology majors were not complete, and one was submitted without a single entry.
A large number of portfolios (42%) received a low effort rating.
Approximately two-thirds of the portfolio submissions were appropriate while one-third were not appropriate for our intended purpose.
Recommendations:
1) Portfolio submissions should be used to assess the second and third pillars of our curriculum.
2) Biology faculty should prepare a NEW portfolio prompt that asks students for two
submissions, an early project submission and a late submission.
3) The submissions should be examined by committed senior seminar teachers in order to encourage students to submit exemplars of their best work.
4) A team of three or four faculty should assess the quality of these works during the summer
following the student’s graduation and bring appropriate curriculum recommendations to the biology faculty.
A recommendation of appropriate submissions and a prompt to guide students in selection of submissions and in formulation of a cover letter is given as Appendix II.
To determine whether the Biology MFAT and specific submissions to the Senior Portfolio Project are appropriate for assessment within the Biology discipline
The Appropriateness of the Biology MFAT for Assessment of the Biology Core:
Action: Two pairs of Biology faculty members independently evaluated the questions of the current Biology MFAT exam, rating each question as being answerable solely by pattern recognition (knowledge base) or requiring at least some problem-solving skill in addition to a knowledge base (knowledge base and process). Having gained permission from the providers of the Biology MFAT, both pairs also answered each question in an effort to produce an in-house key that can be used to score student exams on site so that multiple analyses can be performed by our Testing and Reporting Service that are not made available by the MFAT testing service.
Preliminary Findings: The MFAT is an objective instrument with 150 items. Each item has a single correct answer and four distractors.
The four faculty members were able to develop an in-house key for the current Biology MFAT. Future keys will need to be generated as the exam is revised.
Seventy-five percent (112 of 150) of the questions on the MFAT were of the knowledge-recall type, not requiring problem-solving skills. Twenty percent (30 of 150) were judged to require recall but to also require some process and 5.3 percent (8 of 150) were judged to be purely process with data being supplied in the test.
A series of analyses were performed on the fall 2003 MFAT exams (a sample of only 20 students) to determine:
1) student performance in each of the four subcategories of the exam,
2) performance on knowledge vs. knowledge/process questions,
3) question types and topics that are frequently missed by students, and
4) to calculate a discrimination index for each question as well as other test statistics.
1) Student Performance on MFAT test and subtests: The small sample scored a mean total score of 154.74 (s.d.=35.7) on the MFAT. This corresponds to a national percentile rank of 71.1% (Kangas’ interpolation; see table below). On the Molecular Biology and Genetics subtest (MBG) students scored a mean of 56.31 (s.d.=15.1) with a national percentile rank of 52.0. On the Cell Biology subtest (CB) students scored a mean of 59.6 (s.d.=16.0). This is equivalent to a national percentile rank of 68.2. On the Organismal Biology subtest (OB) students scored an average of 56.46 (s.d.=19.03) which is equivalent to a national percentile rank of 63.4. On the Population, Evolution, and Ecology subtest (PEE) students scored an average of 61.15 (s.d.=15.7). The national percentile rank that corresponds is 73.4.
Number Average Percentile Percent Percent
Test of students score rank above 50%tile above 80%tile
Overall 20 154.74 71.1 85 40
MBG 20 56.31 52.0 60 25
CB 20 59.6 68.2 70 30
OB 20 56.46 63.4 70 45
PEE 20 61.15 63.4 90 35
2) Student Performance on MFAT knowledge and knowledge / process questions: One hundred and twelve (112) of 150 questions were judged by us to be knowledge or recall level questions. Our students answered 39 to 77 of these question correctly (average = 65.2). Thirty (30) questions were judged by us to be process/recall type questions. The sample of twenty students scored form 8 to 20 correct (average = 14.95). Only eight (8) questions were judged to be purely process questions. Students ranged from 2 to 7 correct (average = 4.90). It is not possible at this time to estimate national norms. This may be possible at some future time.
Question types and topics that are frequently missed by students: We calculated Kronbach’s alpha for the overall raw score of the exam (using our key and this small sample of 20 students. This coefficient was 0.83 where 0.0 indicates no reliability and 1.0 is perfect reliability. Reliability here refers to internal consistency of the test. It is greatly affected by the number of test items and the average inter-correlation among the items. Alpha scores greater than 0.8 are considered acceptable. As mentioned in the previous paragraph there were 112 questions that were considered recall questions. When we examine percentage correct for knowledge or recall level questions the average was 58.2%. For the questions judged to be process / recall type questions the average percent correct was 49.8. And for purely process questions the average was 61.2%. We also see from the table above that the poorest subtest scores were in the molecular biology and genetics test section. The best subtest scores were in the population, evolution and ecology subscores.
Recommendations:
Biology faculty who teach core courses (and perhaps others) should review the MFAT
exam in order to determine where (and if) the curriculum covers the tested materials.
This might be accomplished (as Business faculty have done) by the faculty gathering for
a few hours to look at the test together, developing a kind of matrix so that visually they can see that they do or do not cover particular details.
2. Biology faculty may wish to write some additional questions that relate to the processes
of analysis and “higher thinking” to be given when the MFAT is administered but scored
and analyzed locally.
The Appropriateness of the Senior Portfolio Project for Assessment of the Biology Core:
Action: Described above.
Finding: Not appropriate due to current submission selection and the absence of the value-added component.
Recommendations: Biology collect its own portfolio submissions and evaluate them within the discipline, receiving financial compensation equivalent to that given to faculty participants in the University-wide Portfolio Project. A recommended portfolio prompt and submission selection guide are given in the Biology MFAT Exam.
The Biology faculty attempt to use all available data to examine the effectiveness of our teaching. The MFAT exam that you have recently taken constitutes one of those data sets. A small committee of the Biology faculty has been asked to examine the way you (and your classmates) feel about the exam and the effort you have taken with the exam. Although we ask you to identify yourself, we (the committee members: Dr. Hite, Dr. Kangas and Dr. Rutter) promise that your identity and your answers will be held in the strictest confidence and NO other faculty will see anything except the summaries of data from our work WITHOUT NAMES or IDENTIFICATION NUMBERS.
Most of the questions below are to be scored on a Likert scale. For example, answer 1 if you strongly agree and 9, if you strongly disagree. Answer 5 if you are ambivalent or uncertain. Other choices reflect a range of opinion between these points. Please answer each question carefully and honestly.
1. The Biology Core Curriculum (Introductory Biology 107 and 108, Cell Biology, Genetics, Ecology, and Physiology or Plant Physiology) prepared me for the MFAT exam.
1 2 3 4 5 6 7 8 9
strongly agree unsure disagree strongly
agree disagree
2. Through the Biology Core Curriculum, I have been exposed to the concepts and knowledge addressed by the questions on the MFAT exam. For each sub-discipline of biology listed below, write the number from the Likert scale that corresponds to your response.
Cell biology_____ Molecular biology & Genetics _____
Physiology / Anatomy _____ Population biology / Evolution / Ecology _____
Likert Scale: 1 2 3 4 5 6 7 8 9
strongly agree unsure disagree strongly
agree disagree
3. When I compare my effort while taking the MFAT exam, relative to efforts given on exams I have taken in most of my biology courses, I gave the MFAT
1 2 3 4 5 6 7 8 9
much about much
less the same more
effort effort effort
4. My preparation for the Biology MFAT exam consisted of the following (circle all that apply):
a) Studying d) Reading
b) Reflecting on material covered through biology core e) Discussing the exam in Senior Seminar
c) Nothing at all
5. I believe that my MFAT score will reflect the quality of the education, in my major, that I have received at Truman State University.
1 2 3 4 5 6 7 8 9
strongly agree unsure disagree strongly
agree disagree
6. I believe that my MFAT score can be used to meaningfully assess the quality of instruction my professors sought to give to me at Truman State University.
1 2 3 4 5 6 7 8 9
strongly agree unsure disagree strongly
agree disagree
7. Of the choices given below, rank potential times to take the MFAT exam in order from most desirable (1) to least desirable (5).
_____ Saturday morning _____ Saturday afternoon
_____ weekday evening _____weekday morning
_____other ____________________________________________________________________
8. When you went in to take the exam, did you realize the biology discipline seeks to use MFAT scores to assess the biology core curriculum?
_____ Yes _____ No
9. When you went in to take the exam, did you realize the Missouri State government uses student performance on MFAT exams (as well as other assessment instruments) to monitor the success of educational programs at Truman State University?
_____ Yes _____ No
10. I have studied for the following standardized exam(s) in the past four semesters. Please check all that apply.
DAT____ MCAT____ MVAT____ GRE____ Biology GRE____ Other________________
(Please specify)
11. If the University listed my MFAT score on my transcript I would have worked harder while taking the MFAT exam. (Currently these scores are not included on transcripts.)
1 2 3 4 5 6 7 8 9
strongly agree unsure disagree strongly
agree disagree
12. Comments:_______________________________________________________________________
____________________________________________________________________________________
Appendix II: Biology Portfolio Prompt
Prompt for portfolio submissions from Biology majors
Scientific process and communication are two of the three ‘pillars’ that the biology discipline considers central to our educational efforts. The third pillar, Knowledge, will be assessed by separate methods (notably, analysis of senior test results).
Our goal with this prompt was to elicit submissions that will allow us to assess our students’ development in two key areas of biological education: communication and scientific process. The communication assessment will come from evaluating the later submission and the reflective statement. By asking for works involving original research, methodology, and scientific reasoning, we will have a submission with which to assess our students’ understanding of scientific process. The final goal, that of assessing student development in these areas, will be achieved by a comparison of the early and later works.
The prompt:
Include one piece of scientific writing from early in your tenure here at Truman. This work should be from Introductory Biology I or II, Cell Biology, or Genetics. Choose a work that you were proud of at the time, and that demonstrates scientific writing style, scientific reasoning, and an understanding of methodology.
Also choose a recent work of your scientific writing. This submission should come from independent research or a research proposal, a class in the Biology core curriculum, a Biology elective, or an internship. Choose a strong effort that shows scientific writing style, scientific reasoning, and originality of experimental design.
Compare and contrast the two works. Consider things such as clarity of expression, strength of reasoning, consideration of alternative hypotheses, integration of literature with research data, and use of methodologies (including quantitative analyses, development of approaches, etc.). Are there areas where you see significant progress between your two works? Are there areas that still need development? Reflect on how you’ve developed as a scientific writer and thinker.
For a printable version, click here.