What You Measure Is What You Get
John H. Hummel and William G. Huitt
Valdosta State University

Reference: Hummel, J., & Huitt, W. (1994, February). What you measure is what you get. GaASCD Newsletter: The Reporter, 10-11. Retrieved [date], from http://www.edpsycinteractive.org/papers/wymiwyg.html

Return to: | Readings in Educational Psychology | Educational Psychology Interactive |

The country's current focus on promoting critical thinking skills is our collective reaction to a problem that has been developing for some time. Until recently we assumed that critical thinking would automatically develop as students acquired knowledge and primary importance was given to the discussion of specific disciplines to be studied. We were able to ignore issues such as: What is critical thinking? How can it be measured? How can it be promoted? until it became obvious that the level of critical thinking of too many high school and college graduates was insufficient to the demands of modern society.

In order to effectively and efficiently accomplish the objective of improving students' critical thinking abilities, we need to address the issues of defining and measuring critical thinking. These must be done well before we can develop and test empirical intervention strategies that will promote students' critical thinking skills in all grades.

Critical thinking is probably the most current label for what many call analytical reasoning, synthesis, problem-solving, or higher mental processes (Scriven & Paul, 1992). Common threads that tie the various definitions of critical thinking together are the terms used to describe the processes and outcomes associated with thinking critically, the development of concepts and principles, the application of facts, concepts and principles to solve problems and make decisions, and the evaluation of these solutions for effectiveness (Chance, 1986; Ennis, 1987). Almost four decades ago, Bloom, Engelhart, Furst, Hill, and Krathwohl (1956) published their now widely-accepted taxonomy for classifying objectives and assessment items for the cognitive domain. Their system specifies six levels of understanding and mastery, and each higher level subsumes the properties of the lower levels. The levels of the taxonomy are, from lowest to highest, knowledge, comprehension, application, analysis, synthesis, and evaluation. Subsequent research has lead to the conclusion that the taxonomy is indeed a hierarchy with the exception that perhaps evaluation and synthesis are misplaced (Seddon, 1978).

Typically, students' achievement and critical thinking skills are assessed using a forced-choice format. Unfortunately, most items used in these assessments address levels of knowing and thinking not typically associated with critical thinking. Many researchers (e.g., Carter, 1984; Gage and Berliner, 1992; Woolfolk, 1993) agree that the objective test items used at all levels of education overwhelmingly tap the lower (i.e., knowledge and comprehension) levels of the Bloom et al. (1956) taxonomy. Other researchers who developed alternative taxonomies have drawn a similar conclusion (e.g., Stiggins, Rubel & Quellmalz, 1988).

These problems are crucial in that the types of assessments used in education affects how students learn and how teachers teach (Fredericksen, 1984). This conclusion is so central to teaching and assessment practices at all levels of education that in our preservice and inservice teacher education classes we use the acronym WYMIWYG to emphasize its importance. WYMIWYG specifies a concept we believe ought to be a guiding principle for all educators: What You Measure Is What You Get. If educators develop assessments aimed at higher-levels thinking skills, (a) they will be more likely to teach content at those levels, and (b) students, according to Redfield and Rousseau (1982), will master-and-perform at those levels. Students not only need to know an enormous amount of facts, concepts, and principles, they also must be able to effectively think about this knowledge in a variety of increasingly complex ways.

Getting Started

We believe all educators can immediately begin improving students' abilities to think critically by implementing a few basic strategies. First, teachers must insure that there are instructional/behavioral objectives that cover the lesson's content. Care should be taken to match objectives with an outside assessment, a next level of learning, or a stated requirement for success in a given field or career (Wiggins, 1991). Objectives that are too broad or general should be rewritten to specify what students should be able to do after mastering the objective, and if teachers identify expected outcomes not covered in existing objectives, new ones should be developed. After validating the congruence or overlap between the objectives and content taught, educators should then analyze each objective to determine its level vis-a-vis a validated taxonomy of the cognitive domain (e.g., Bloom et al., 1956; Ebel, 1965; Gagne, 1985; Stiggins et al., 1988). If appropriate, objectives should be rewritten to reflect a higher level of the taxonomy. (As one rewrites objectives to require students to use critical thinking skills, one may also have to revise the instructional techniques used to teach the course content.) Next, one should evaluate the assessment instruments used to establish whether students have mastered the content at the stated level of the taxonomy. If test items are used that only require lower-level thinking skills such as knowledge and comprehension, students will not develop and use their higher- order skills even if instructional methods that employ these skills are implemented. This follows the maxim that individuals do not do what is expected, only what is inspected.

Conclusions

Convincing educators, including college teachers, to demand precise, operational definitions of critical thinking is going to be no easy matter. In addition, getting teachers and standardized test developers to assess students at the higher levels of the taxonomy will not be an easy task. It takes time to prepare good assessments (e.g., tests, demonstrations, exercises) that require students to think critically; it takes even longer to prepare the necessary keys and to grade assessments such as essay exams and term papers. Everyone is willing to say that good teaching and assessment (especially as they relate to critical thinking) are important, but not enough educational resources are committed to support and promote these activities.

Of course, simply having teachers give more essay-type or activity-oriented assignments (even good ones that tap into the higher cognitive domains) will not necessarily improve students' critical thinking skills. Likewise, outcomes assessment efforts, including standardized tests, for high school or college graduates will not, by themselves, produce improvements in students' critical thinking skills (though such assessments, if valid, may help to emphasize and document the extent of the problem!).

Many teachers at all levels will likely need to be provided with inservice instruction so they can (a) discover/rediscover the value of instructional techniques that include well developed objectives and task analyses, and (b) incorporate new teaching strategies (e.g., Gray, 1993; Hummel & Rittenhouse, 1990) into their pedagogy that research shows help to promote/develop the skills associated with critical thinking. Fortunately, a variety of alternatives are becoming available (e.g., Georgia's Critical Thinking Skills Program, 1993; Oxman, 1992). However, unless measurement of students' critical thinking skills are completed regularly and given prominent attention by educators, the sustained efforts required to make changes in our educational system are not likely to occur.

References

• Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., Krathwohl, D. R. (1956). Taxonomy of education objectives, handbook I: Cognitive domain. New York: David McKay.
• Carter, K. (1984). Do teachers understand principles of writing tests? Journal of Teacher Education, 35, 57-60.
• Chance, P. (1986). Thinking in the classroom: A survey of programs. New York: Teachers College, Columbia University.
• Ebel, R. L. (1965). Measuring educational achievement. Englewood Cliffs, NJ: Prentice-Hall.
• Ennis, R. H. (l987). A taxonomy of critical thinking dispositions and abilities. In J. B. Barron, & R. J. Sternberg's (Eds.) Teaching thinking skills: Theory and practice (l-26). New York: W. H. Freeman.
• Fredericksen, N. (1984). The real test bias: Influences on teaching and learning. American Psychologist, 39, 193-202.
• Gage, N. L., & Berliner, D. C. (1992). Educational psychology (5th ed.). Boston: Houghton Mifflin.
• Gagne, R. M. (1985). Conditions of learning (4th ed.). New York: Holt, Rinehart, & Winston.
• Georgia's Critical Thinking Skills Program. (1992). Atlanta: Georgia Department of Education.
• Gray, P. (1993). Engaging students' intellects: The immersion approach to critical thinking in psychology instruction. Teaching of Psychology, 20, 68-74.
• Hummel, J. H., & Rittenhouse, R. D. (l990, May). Revising Woods' taxonomy of instrumental conditioning. A paper presented at the annual meeting of the Association for Behavior Analysis, Nashville, TN.
• Oxman, W. (Ed.). (1992). Critical thinking: Implications for teaching and teachers. Upper Montclair: Montclair State College.
• Redfield, D. L., & Rousseau, E. W. (1981). A meta-analysis of experimental research on teacher questioning behavior. Review of Educational Research, 51, 181-193.
• Scriven, M., & Paul, R. (1992, November). Critical thinking defined. Handout given at the Critical Thinking Conference, Atlanta, GA.
• Seddon, G. M. (1978). The properties of Bloom's taxonomy of educational objectives for the cognitive domain. Review of Educational Research, 48(2), 303-323.
• Stiggins, R. J., Rubel, E., & Quellmalz, E. (1988). Measuring thinking skills in the classroom (Revised edition). Washington, DC: National Education Association.
• Wiggins, G. (1991). Teaching to the authentic test. Educational Leadership, 46(7), 41-47.
• Woolfolk, A. E. (1993). Educational psychology (5th ed.). Boston: Allyn and Bacon.

Authors' Notes

An earlier version of this paper was presented to the National Panel of Education 2000. Requests for reprints should be addressed to John H. Hummel, Department of Psychology, Counseling and Guidance, Valdosta State University, Valdosta, GA 31698. At the time this article was written, John Hummel and William Huitt were co-directors of the Georgia/Alabama Center For Critical Thinking, established in 1992. Its purpose is to promote improved thinking and problem solving among students at all levels of education (including K-12 and higher education.) It is affiliated with the National Council for Excellence in Critical Thinking Instruction and is one of 12 Regional Centers established throughout the United States. The Center serves as training resource as well as a clearinghouse for books, articles, films and videotapes, audiotapes, computer software, and other materials related to critical thinking. Interested educators should contact Dr. Linda Schwartz, current director of the Center at (912) 333-5930 for additional information.