Journal of Staff Development, Winter 2000 (Vol. 21, No. 1)

"That’s simply not true!" Beverly, a 3rd grade teacher fumed as she looked at the newspaper editorial. The headline read: "Area Elementary Schools Shortchanging Students." She quickly scanned the article and found that the newspaper had based its claim on the latest state assessment scores for the community’s three elementary schools.

In math and science, students at all three schools, including Beverly’s, ranked below the state average. "We are, too, doing a good job! I know our kids are doing better than these state tests show," she muttered to herself, vowing to raise the issue at the next meeting of the school improvement committee.

"Look, this block scheduling we’ve been using for the past two years appears to be the answer to our problems," explained Murray, a middle school teacher, to his fellow teachers during a school improvement committee meeting. "But I’m getting lots of grumbling from our high school colleagues. They say we just jumped on the fad bandwagon and our kids are suffering. The high school teachers believe our kids simply won’t be ready for the rigors of high school courses. How can we prove to them that block scheduling is working and our kids will indeed be ready?"

"I don’t want to have to do that again," Joan complained to the principal’s secretary as she walked out of the vice-principal’s office, where she left four very angry 9th grade girls. "That’s the second time today I’ve had to break up a shouting match between girls during passing period. Is it me or does it seem that this kind of thing is happening all the time? We need to do something to stop this!"

Beverly, Murray, and Joan need answers. In the first scenario, Beverly and her colleagues need to take stock of their school, answering the question: Where are we NOW? How "healthy" is our school? In the second scenario, Murray and his fellow educators need to determine the effectiveness of block scheduling. They need to ask and answer tough questions: Is it working and should we continue using it? In the third scenario, Joan needs to ask questions and get answers, but her questions have a different purpose: To find a solution to a problem.

Getting answers to these questions requires that educators know how to collect, analyze, and interpret data. In other words, they need to know how to conduct a credible program evaluation so that they decide how best to meet the learning needs of all youngsters. Conducting a credible program requires that educators follow six steps:

These six steps can serve as a suitable guide no matter how narrow or broad the scope of an evaluation. They work equally well for both individual teachers who want to take stock of their own classroom practices and teams of educators who want to determine the effectiveness of schoolwide interventions. Finally, the process can guide both formal and informal evaluations, as well as formative and summative evaluations.

If there’s one thing we’ve learned in the last decade of school reform it is this: Good schools don’t just happen. They happen because the adults in them decide to collaborate on systematically collecting and analyzing data, and then take action based on their findings. The six-step program evaluation process outlined here can serve as a beginning guide that we, as staff developers, can use to support teachers and principals as they seize the data and make decisions.

In Step 1, educators pose the questions that will guide their evaluation. At the outset, their questions will most likely fall into three categories – questions that take stock, determine effectiveness, or seek solutions.

Some taking-stock questions that can get an evaluation off to a good start include:

• Where are we now?
• How "healthy" is our school?
• How do we stack up against the standards?

If teachers want to determine the effectiveness of a strategy, they can simply ask:

• Did it work?
• Did we achieve what we set out to accomplish?
• Did it make a difference?

Finally, if their purpose is to find a solution to a problem, they can begin by simply asking:

• What’s going on here?
• What’s causing the problem?

As they generate questions, educators should spread their net wide. They should seek input from decision makers who decide the fate of the program as well as people who are affected, either directly or indirectly, by the intervention. Finally, they should review the program’s intentions.

Step 2 requires that educators determine up front the criteria and standards they will use to make their judgments. Why do this up front? A primary reason has to do with the difficulty we have in separating feelings from facts. Often stakeholders already have in mind some sort of criteria and standards they will use to judge the worthiness of the program. However, these criteria are often implicit and rarely well-defined. Furthermore, if stakeholders actually do define their judgment criteria, they tend to do so at the end of an evaluation, when their emotions are so mixed with the facts that they can barely tell one from the other. Under such emotional situations, we tend to make snap judgments based on gut feelings rather than data. To avoid snap, emotion-laden judgments, program evaluators explicitly state the judgment criteria and standards before they gather any data.

For example, Murray and his middle school colleagues in the example above might use these criteria to evaluate the effectiveness of block scheduling:

• Students on the block schedule will master a higher percentage of items on the state criterion-referenced assessment in reading comprehension, math problem solving, science knowledge, and expository writing than similar students who aren’t on a block schedule.
• Students on the block schedule will show more engagement during academic classes than similar students who aren’t on a block schedule.
• Students on the block schedule will demonstrate greater perseverance when faced with a difficult learning task than students who aren’t on a block schedule.

Another reason for defining judgment criteria up front is that frequently these yardsticks will dictate the kinds of data evaluators need. For example, if the criteria for judging the effectiveness of the Drop Everything and Read (DEAR) program is "Our students will read significantly more fiction books during the semester," the teachers know they will need to collect data that show the number of fiction books read by students the semester before implementation of the DEAR program, as well as the number of books they read during the semester of implementation.

Educators can glean ideas for the judgment criteria and standards by asking stakeholders; reviewing checklists, standards, and guidelines; consulting with experts; reviewing the purposes of the strategy; and examining research.

Once they have their evaluation questions and judgment criteria in hand, the educators-turned-program-evaluators are ready to make a plan for getting their questions answered. Essentially, they sketch out who will collect what data, by when, and using what methods.

They also need to specify how to analyze the data they gather.

Building a matrix like the example above is the easiest way to develop an evaluation plan. In column A, the evaluator simply lists each of the evaluation questions posed in Step 1. Then, using common sense mixed with knowledge of good program evaluation techniques, the evaluator builds the rest of the matrix. Even modest, informal evaluations benefit from planning.

Program evaluators use three basic methods to gather information:

• Reviewing documents or artifacts;
• Asking people for facts or opinions; and
• Observing situations and behaviors.

No matter which data collection methods they use, evaluators should keep them simple and reliable. "Simple" means relying on existing data when possible and making data collection easy for both those who provide the data and those who collect the data. "Reliable" means using systematic and impartial methods to gather data.

Step 5 consists of three tasks: organizing, describing, and analyzing the data. Organizing data involves putting the data into some sort of frequency table. One way to organize test scores is shown on the following page in the chart, "I feel most in control when I am..." The accompanying chart, "I can make a difference in my life," displays a frequency table summarizing data from a survey.

When describing data, both pictures and numbers work well. Some common graphs used to summarize and display data include bar charts, line charts, pie charts, pictograms, scatter plots, and box and whisker plots. Evaluators can also describe a set of data by using numbers. The mean, median, mode, range, and standard deviation can indicate both the central tendency and the variability within a data set. Spreadsheet computer programs can generate elegant graphs, and also calculate various measures of descriptive statistics in a flash.

In analyzing data, program evaluators seek to answer a range of questions. Some answers to these questions rely solely on logic and critical thinking, while others also require analysis using inferential statistics. This is the point in an evaluation where technical knowledge about statistics and research comes in handy.

In Step 6, the educators-turned-evaluators can finally answer the questions posed in Step 1. No matter what type of questions they posed, the evaluators should follow a similar process to interpret the results of their analyses.

First, look for patterns. Do most of the student opinions point to a positive learning environment within the school? When do most of the fights in the hall occur? Do students on the block schedule achieve as well as or better than those not on that schedule? Using some sort of summary matrix can often help evaluators see the patterns emerge.

Next, using these identified patterns, draw conclusions that will withstand the scrutiny of supporters and critics alike. Finally, apply the pre-established criteria and standards set in Step 2 and make judgments about the effectiveness and worthiness of the program.

As they interpret their findings, teachers and principals should be aware of various pitfalls that can render an evaluation useless. Some common missteps include:

• Seeing what they want to see, rather than the facts.
• Looking at the data only through a "microscope," thus failing to see the big picture.
• Performing only an "eyeball" test of significance and concluding that any change is significant.

The six-step evaluation process outlined above puts the power of data in the hands of those who most need it – the teachers and principals who are on the "front lines" with kids. The process prepares these educators for the implied seventh step in the process: Taking action to improve student learning, and continually re-evaluating the situation and the actions taken to see if modifications should be made. By helping teachers and principals acquire the skills of program evaluation, staff developers can cultivate educational professionals who use data, and not just instinct, to guide their classroom and school practices.

• Getting answers to your questions: an elementary educators’ guide to program evaluation, by Brenda G. LeTendre and Richard P. Lipka. Norwood, MA: Christopher-Gordon. In press.
• Getting answers to your questions: a middle-level educators’ guide to program evaluation, by Brenda G. LeTendre and Richard P. Lipka. Norwood, MA: Christopher-Gordon, 1999.

For information, contact: Christopher-Gordon Publishers, 1502 Providence Highway, Suite 12, Norwood, MA 02062, (800) 934-8322.

Here is a list of recommended print and Internet resources on using data to guide instructional decisions. Many of these resources can

• Fundamentals of descriptive statistics, by Z.C. Holcomb. Los Angeles: Pyrczak Publishing, 1997.
• Statistics: A spectator sport (2nd ed.), by R.M. Jaeger. Thousand Oaks, CA: Sage, 1990.
• Making sense of statistics: A conceptual overview, by F. Pyrczak. Los Angeles: Pyrczak Publishing, 1995.

Connects to the home page of the Clearinghouse on Assessment and Evaluation, sponsored by the Educational Resources Information Center (ERIC) funded by the U.S. Department of Education. A great jumping-off site to a myriad of topics all related to assessment and program evaluation. Simply click on the category of your choice and immediately you see links to quality Internet resources on the topic. A sampling of categories includes: action research, achievement data, alternative assessment, fairness in testing, goals and standards, instructional and program evaluation, organizations, pedagogy in educational measurement (much how-to stuff here), qualitative research, statistics, test descriptions, test reviews, and tests online.

Connects to Professor Glass’ web site supporting his course "Introduction to Quantitative Methods" taught at the College of Education at Arizona State University. Provides clear explanations with graphics illustrating concepts dealing with data analysis. Periodically asks questions to check your understanding and then gives you answers. Includes links to other statistics resources for "another treatment" of the topics. Not heavy on the statistical/mathematical symbols. Concentrates on explaining the concepts, but does show how to calculate various descriptive and inferential statistics using a computer program.

Takes you to Professor Mettetal’s web page at Indiana University-South Bend, where you’ll find links to action research web sites. Also includes a handy piece called "How to report statistics, a quick guide."

Connects to the Test Review Locator, a searchable database giving descriptions and reviews for various testing instruments. Sponsored by the federal government’s Educational Resources Information Center (ERIC). Also allows you to search the 10,000 tests included in the Educational Testing Service (ETS) Test Collection. The Test Collection encompasses virtually all fields within education and the social sciences. Also includes a helpful document giving tips on selecting tests appropriate to your purpose.

Connects directly to the "data analysis" section of Professor Trochim’s "The Knowledge Base, An Online Research Methods Textbook" at Cornell University. The best site I’ve found dealing with data analysis. Covers the concepts and procedures of organizing data, describing data, and using inferential statistics. Great graphics, clear explanations, enough detail yet doesn’t overwhelm the reader. Very user friendly.

A dy-no-mite book!! Very practical. Written in easily understood language with a collegial tone. Thoroughly walks through how two schools (one elementary and one high school) used data to make decisions. Excellent illustrations. Gives suggestions on gathering, displaying, and analyzing data from the following dimensions: (a) demographics, (b) perceptions, (c) student learning, and (d) school processes. Illustrates 10 levels of analysis from "snapshots" to the interaction of multiple measures over time.

One of the best explanations of standardized tests I’ve seen. Gives an excellent overview of the purposes and uses of standardized tests and then clearly explains how test makers construct norm-referenced and criterion-referenced tests. Also covers performance tests. Provides a jargon-free explanation of how to interpret tests, complete with sample printouts. Also provides a brief but clear discussion of basic descriptive statistics.

Packs into 70 pages a good overview of program evaluation. Quickly walks through a 10-step process for planning and conducting an evaluation. A good "starter" book for teachers and administrators alike.

Very user friendly and makes the assumption that the reader is an intelligent person who happens to be naive about conducting surveys. Covers the process from start to finish, from planning to designing to conducting to analyzing to presenting findings. Includes many real world examples. Gives an excellent overview of how to do various statistical analyses of survey data. Can act as good reference on doing survey research and evaluation, since you can easily turn to the section you want and get the information you need.

Grew out of the author’s experience in teaching a graduate course on educational research. Deals with both research and program evaluation, with the purpose of creating reflective practitioners. Ably guides the educator through the research process, from identifying what to study, selecting a research design, collecting data and finally analyzing and interpreting data. Hits that happy medium between too much and too little information. Includes periodic exercises (with answers provided) so that you can check your understanding of concepts.

Written for practitioners (both teachers and administrators) interested in whole school improvement. Focuses on providing educators with the guidance and tools to answers the following questions: How are we doing? How can we improve? How can we share our successes? Includes an annotated list of suggested readings at the end of each chapter.

Targets practicing teachers (Pre-K-12) who wish to conduct research that will inform their work with children. Written in a collegial tone with many detailed samples of data collection and data analysis techniques. Examples come from the work of practicing teacher-researchers (listed in the appendices of the book). Chapter 2 covers note taking while "kid watching," surveys, student work and classroom artifacts, sociograms (one of the best explanations and examples of how to do this and what it tells you as a teacher), and the use of audio and video tape transcriptions. Chapter 3 includes a good discussion of how to find and refine a research question.

Provides a comprehensive guide for taking stock of your school or district. Includes judgment criteria as well as suggested indicators dealing with the following areas: mission, goals, organizational culture, strategic planning, management, leadership, instructional services, decision making, policies, procedures, and school-community relations at both the district and school levels. Also provides sample surveys that you could use to collect the data you need to make your judgments.

Chapter 6 focuses on "Responsive Action Research" where an educator gathers data to help solve a problem.

Gives a step-by-step explanation of the action research process, along with two case studies showing how two teachers used the process to solve the problems of unresponsive students and poor relationships among students of different racial groups.

A must for middle-level educators wanting to know if their programs fit the needs of their students. Provides more than 25 survey and observation instruments based on the characteristics of exemplary middle schools.