Taking Measure: Pause to look at the intersections in the data

JSD, Spring 2005 (Vol. 26, No. 2)

Copyright, National Staff Development Council, 2005. All rights reserved.

The reality of most program evaluations being done today is that staff developers do not make thorough use of the data we collect. While measurement experts push holistic examination of all available information, we often are guilty of failing to pause to question what one piece of data is saying about another. To become leaders in the push for accountability, we need to get in the habit of looking for the meaning of intersections of whatever data we have collected and asking hard questions that will lead to better decisions. To cultivate this habit, we must begin to gain a deeper understanding of the concept of data intersections.

Our evaluations can be strengthened by using multiple measures. Whether our data include questionnaires distributed at the end of training events, samples of rubrics instructional teams have designed after rubrics training, artifacts such as minutes of team meetings, weekly logs kept by mentors, notes from interviews with participants in internship programs, observational data from walk-through visits to classrooms of teachers we are coaching, or tracking questions that participants ask throughout a seminar series, we need to do much more with our available data. In particular, we need to consider the meaning of intersections among the data. While it may seem that examining all data together is comparing apples to oranges, each kind of data contributes to a more comprehensive understanding of the whole story. Dialogue about the implications of various data leads to better explanations of root causes of problems, to uncovering subtle data trends, and to deeper understanding of dramatic turnarounds or stalled implementation.

One of the most articulate experts advocating the use of multiple measures and examining data intersections is Victoria Bernhardt, California State University's executive director of the Education for the Future Initiative and author of numerous books on using school data, including Data Analysis for Comprehensive Schoolwide Improvement (Eye on Education, 1998). Bernhardt answered several questions about using data and data intersections.

Q: What different categories of measures do you consider essential to making good improvement decisions?

Bernhardt: I think that to make good improvement decisions, educators should look at four categories of data: demographic data, student learning data, perceptual data, and school process data.

Demographic data provide descriptive information on items such as enrollment, attendance, grade level, ethnicity, gender, home background, and language proficiency. Student learning data describe an educational system in terms of standardized test results, grade point averages, standard assessments, and other formal assessments.

Perceptual data help us understand what students, parents, teachers, and others think about the learning environment. Perceptions are important since people act based on what they believe. School process data define programs, instructional strategies, and classroom practices. This measure seems to be hardest to describe, yet it is the one type of data that's most readily available to document.

School processes, meaning the instructional and assessment strategies we use and programs we offer, are critical for understanding how we get the results we are getting. Schools have these data, but they often are not coded in their data analyses that way. Or sometimes it is hard to describe how reading is taught, for instance, because there is no systematic approach.

Q: You emphasize the importance of looking holistically at school data, especially moving on to ask hard questions about where the data intersect. How would you explain the concept of intersection?

Bernhardt: When we "intersect" data, we are looking at two or more types of data at the same time. This may or may not be the same as disaggregating data, where we break a whole into parts. When we intersect data, we look for the overlapping, the commonalities, or the relationship of two or more types of data.

Q: Can you give us a specific example of how examining the intersections can help us make better use of data for professional learning?

Bernhardt: If I were going to evaluate a professional development program, I would compare teachers in the population with those enrolled in the professional development program (intersecting program information with demographics). I would be able to learn if my program reached those I wanted to reach.

Next, I would ask questions about the impact of the professional development in the teachers' classrooms and study the results by what they teach (perceptions by school processes). This intersection would help me know if the program had more of an impact on lower grade-level teachers versus upper grade-level teachers, for example.

If I had the wherewithal also to study the school's student learning results, I could understand which processes lead to the best results at each grade level and the degree to which the professional development contributed to the results (demographics by school processes by student achievement).

If I wanted to know how to improve the program, I would add perceptions to the last intersection, and my understanding of the program's impact and of how to improve it would be close to complete.

Q: Often when educators examine data closely, they have "aha" moments as they make new inferences from the data. Frequently they realize how their results likely have been negatively impacted by decisions made by policy makers. What is a useful way to handle their frustration with problems rooted in decisions made upstream?

Bernhardt: I find that if teachers and principals can pull together their data comprehensively and show the results objectively, local policies often can be changed. If the policies are larger than local, they would have to use as much positive information as they can glean from the data to keep going in the right direction. It is important to share data with the public so everyone can understand where schools really are. Perhaps this would help policy makers in making smarter decisions for schools.

REFLECTION: MAKE BETTER USE OF OUR OWN DATA

As professional developers learn to move beyond the era of "happiness sheets" (customer satisfaction surveys) to gauge program effectiveness, we must learn the new habit of collecting a full range of data to drive decisions.

Along with collecting a better variety of data, we must learn to make fuller use of all the information we already have at our fingertips. Holistic examination of all available information will help us avoid relying on anecdotal comments, making wrong assumptions, or missing subtle cause-and-effect relationships among various factors. Pausing to examine where our different kinds of data intersect, we can become much more deliberate in figuring out the root causes of recurring problems in professional development programs and results.

Bernhardt uses slightly different terminology when identifying the four categories of important data from the terminology we use in staff development. First, staff developers usually refer to "processes" as "models." Just as schools often do not document the processes they are using, we often simply mention the models to be used in professional development plans. We may note that peer coaching or collaborative analysis of student work groups will be used, but we do not describe the processes fully. Without full descriptions, we sometimes have to guess about how the processes used impacted our results.

In addition, Bernhardt's student learning category fits our world. It is a major focus for most staff development efforts. However, the category could be broadened to include other kinds of impact indicators, such as the important preliminary step, evidence of staff learning. In staff development, we sometimes launch programs without fully describing the intended short-term impact or how we will measure long-term results.

Bernhardt's third data category is demographics. Staff developers usually are fully aware of the impact of serving a particular faculty demographic in a program. We often consider the demographics as a way to estimate the potential for the impact on students of the staff's learning. After a program is implemented, we often reflect on the demographic to better understand the program's results. For example, when a district decides to include only two teachers per school in a professional development program, it is understandable when a school with 125 faculty members never moves beyond awareness-level learning for the faculty.

Bernhardt's fourth and final data category, perceptions data, wins the prize as the data source most frequently collected and used - some would say grossly overused - to make decisions in staff development. We sometimes refer to it as anecdotal, and we often collect it in a range of ways. Bernhardt's advice is to use perceptions data by trying to figure out how people's perceptions affect what is going on since people's behavior usually reflects their real beliefs.

New habits in dealing with our own data are a step toward achieving better results. Becoming more data-savvy increases staff developers' credibility, which is very important if we are to be successful in encouraging other educators to use data regularly.

About the Author

Robby Champion works as an independent consultant. You can contact her at Champion Consulting, 322 Mt. Tabor Road, Staunton, VA 24401, (540) 886-0655, e-mail: Robbychampion@aol.com.