By Victoria L. Bernhardt

Journal of Staff Development, Winter 2000 (Vol. 21, No. 1)

Too often, schools in this country conduct their education programs with little formal analysis of how well those programs work. Teachers and administrators rely instead on "gut feelings" about what’s working and what isn’t.

They try to be optimistic, hoping that they are doing the right things, but they never get a clear sense of whether their program is working particularly well. Neither do they analyze their goals and challenges systematically, which robs them of the chance to ask better questions and get answers that can lead to meaningful change in classroom practice.

There are many reasons schools don’t use data well. In many schools and districts, data analysis has never been viewed as a high priority. Many state education departments put little emphasis on schools gathering data, and thus provide little incentive for districts and schools to devote time, money, and staff resources to using data in new ways. School-based educators often lack the training, equipment, and time to develop and carry out complex analyses. And many educators fear data analysis, instead of embracing it as a way to make their jobs easier and more rewarding, because they fear data will be used to scrutinize and even attack them.

These are serious issues which educators must address in coming years. But meanwhile, schools can still use data to provide meaningful, ground-level guidance on teaching practice. You don’t need an advanced degree in statistics and a roomful of computers to start asking data-based questions about your school, and using what you learn to guide reform.

Here we present a model that lets a school quickly begin posing and answering data-based questions about teaching and learning. As educators become more familiar with collecting and interpreting school data, they can begin "running data at each other," framing questions that require analysis of multiple types of information. Educators can cross two, three, and four categories of data in ways that can provide new insight into student learning and how to improve it. This process can help:

• Replace hunches and hypotheses with facts;
• Identify root causes of problems, not just the symptoms;
• Assess needs, and target resources to address them;
• Set goals and keep track of whether they are being accomplished; and
• Focus staff development efforts and track their impact.

An effective data analysis of a school or program can include four different types of data:

1. Student learning data describe an educational system in terms of standardized test results, grade point averages, standard assessments, and other formal assessments. Analyzing one year of student learning data, schools can answer questions like, "How did students at the school score on a particular standardized test?" Over time, schools can answer questions such as, "Are there differences in student scores on standardized tests over the years?"

2. Demographic data provide descriptive information on items such as enrollment, attendance, grade level, ethnicity, gender, home background, and language proficiency. Demographics are very important because they describe the part of our educational system over which we have least control. Demo-graphics help us understand past trends and predict future trends. One year of demographic data can answer questions like, "How many students are enrolled in the school this year?" Over time, that same question can be rephrased as, "How has enrollment in the school changed?"

3. Perceptions data help us understand what students, parents, teachers, and others think about the learning environment. Perceptions are important since people act based on what they believe. It’s important to know how students, teachers, and parents think about school, what relations have been like in the past, and what expectations they have for the future. Perceptions data can be gathered in a variety of ways, such as questionnaires, interviews, and observations. One year of perception data could answer the question, "What are current parent, student, or teacher perceptions of the learning environment?" Over time, the question we might want to answer is, "How have perceptions of the learning environment changed?"

4. School process data define programs, instructional strategies, and classroom practices. This is the measure that seems to be hardest for teachers to describe, yet it is the one type of data that’s most readily available to document. To collect school process data, educators must systematically examine their practice and student achievement, making sure both are aligned with specifically defined, desired student outcomes. One year of school process data can answer the question, "What are we doing to teach reading?" Looking over time, we can answer questions like, "How have we been teaching reading for the past five years?"

One category of data by itself gives useful information especially over time. "Crossing" different measures, however, can provide much deeper insight. Cross-ing two measures, we begin to see a much more vivid picture of the school, allowing us to pose and answer questions such as:

• Do students who attend school every day get better grades? (Looking at demographics and student learning.)
• Do students with positive attitudes about school do better academically, as measured by teacher-assigned grades? (Perceptions and student learning.)
• Did students who were enrolled in interactive math programs this year perform better on standardized achievement tests than those who took traditional math courses? (Student learning and school processes.)
• What strategies do 3rd grade teachers use with students who are not native English speakers? (Demographics and school processes.)
• Is there a difference in how students enrolled in different programs perceive the learning environment? (Perceptions and school processes.)
• Is there a gender difference in students’ perceptions of the learning environment? (Perceptions and demographics.)

• Do students of different ethnicities perceive the learning environment differently, and do they score differently on standardized achievement tests in patterns consistent with these perceptions? (Demographics and perceptions and student learning.)
• Which program is making the biggest difference this year with respect to student achievement for at-risk students, and is one group of students responding ‘’better" to the processes? (School processes and student learning and demographics.)
• Is there a difference in students’ reports of what they like most about the school, and is this different if they participate in extracurricular activities? Do these students have higher grade point averages than students who do not participate in extracurricular activities? (Perceptions and student learning and school processes.)
• What instructional process did non-English speaking students respond to best in their all-English classrooms this year? (Perceptions and demographics and school processes.)

Are there differences in achievement scores for 8th grade girls and boys who report that they like school, and do these differences correlate with the type of program in which they are enrolled?

Here is an example of how a school has crossed different data to define and address a problem:

The author has worked as a consultant with a large elementary school in northern California, where 42% of stu-dents are members of minority groups, 40% arrive speaking languages other than English, and 36% receive free- or reduced-price lunch.

Three years ago, the district superintendent announced that all students should be reading on grade level by 3rd grade.

The school began by looking at student scores on standardized tests, and the number and percentage of students not meeting the grade-level standards. Next, they created a two-measure analysis and broke down the test scores by demographic qualities — including gender, ethnicity, and language fluency — to identify low-performing students.

The school then added a third measure to identify academic programs for these students. At this point they were able to see a simple but powerful truth: Students who were not fluent in English were having the greatest trouble reading on grade level, and they weren’t responding well to the school’s reading intervention program. Based on these findings, teachers spent extra time reading with these low-achieving students, to identify their specific reading problems.

Teachers soon realized that these students did not adequately understand the meaning of many words, even though they were completing their assigned work in reading. The teachers decided newly arriving students should go first to classrooms where they would receive more work on word concept skills, and then into the reading intervention program. The teachers also sought professional development training on how to provide individualized reading instruction within groups of students.

This change in instruction has yielded results: Reading scores for non-English speakers improved immediately and have continued to improve. By the second year of the change, 2nd graders with limited English skills were scoring better on standardized reading tests than 3rd graders in this category had scored in the past. Non-English speaking students still don’t score as well on the tests as native English speakers, but the gap has narrowed.

What’s more, teachers at the school are far more familiar with using data — many have become adept at using the district’s new database software which lets them cross-analyze data more easily — and will continue to examine student achievement systematically as they seek further improvement.

While the deepest insight into schools and students can be gained by crossing different measures, to gain a more well-rounded picture of the school and its challenges, even a relatively simple analysis of school data can help teachers shape their practice more effectively.

A short, related article follows here.

LACK OF CULTURAL EMPHASIS

Within the school and district, data analysis is not deemed particularly important. District personnel are still working within antiquated job definitions that do not make helping schools with data a priority. Likewise, many state education departments put little emphasis on data analysis, providing little incentive for districts and schools to change. Teachers and administrators don’t see data collection as part of their jobs, perceiving it instead as a waste of time: "After all," they might say, "we’re here every day, we already know what the problems are." There is no person or group of people specifically assigned to this task in a school.

LACK OF TRAINING

Too few people at the school level are adequately trained to gather and analyze data, or to establish and maintain databases. There are not enough good examples of schools gathering, maintaining, and benefiting from the use of data. Teachers don’t have access to adequate computer resources, including hardware and specialized data software, nor would they know how to use them.

FEAR

Many educators are afraid that data analysis will turn up something they do not want to see, such as evidence of their incompetence. They’ve seen test scores and other data analyses used to "beat up" other educators — a district may, for example, use student test scores as evidence that a particular school should be reconstituted.

The examples and concepts in this article are presented in greater detail in Data Analysis for Comprehensive Schoolwide Improvement by Victoria L. Bernhardt (Larchmont, NY: Eye On Education, 1998). For more information, contact Eye on Education, 6 Depot Way West, Larchmont, NY 10538, (914) 833-0551, fax (914) 833-0761.

Additional resources include:

• Designing and using databases for school improvement by Victoria Bernhardt. Larchmont, NY: Eye on Education, 1999.
• The school portfolio: A comprehensive framework for school improvement (2nd edition) by Victoria Bernhardt. Larchmont, NY: Eye on Education, 1998.
• The example school portfolio by Victoria Bernhardt, et al. Larchmont, NY: Eye on Education, 1999.
• Toward quality in education: The leader’s odyssey by National LEADership Network Study Group on Restructuring Schools, U.S. Department of Education. Washington, DC: U.S. Department of Education, 1993.