Facilitator 10, Refreshments 8

Facilitator: 10
Refreshments: 8
Evaluation: 0

Workshop satisfaction misses the point. Evaluation means understanding what participants learn, when and how they apply the learning, and when and how it benefits students.

By Hayes Mizell

Journal of Staff Development, Fall 2003 (Vol. 24, No. 4)

Evaluation is the burr under the saddle of professional development. To date, many educators have only gone through the motions of evaluating efforts to increase what their colleagues know and can do. They have focused more on the delivery of staff development than on its results, often using the most rudimentary techniques for gathering data. Simplistic surveys and the use of the Likert scale are favorite techniques. What did participants think of the workshop leader? Was the setting comfortable? Did participants enjoy the refreshments? Did they find the experience helpful? Educators answer these questions in different ways. Many provide answers they believe their supervisors want to hear. Others affirm the person responsible for organizing the professional development. Some people use the opportunity to channel their anger about even having to participate in staff development, or about unrelated grievances.

Educators responsible for professional development continue to take this approach to evaluation for four reasons. First, the evaluations do not take much time to develop, administer, or analyze. Surveys and the Likert scale lend themselves to evaluations that are distributed almost as an afterthought to session participants just before they rush out the door. Predictably, participants respond hastily, drawing more on their gut reactions than a thoughtful and honest assessment of their experience. Second, these evaluations almost always generate positive responses, thereby allowing the person in charge to feel good and cite the encouraging "results" of their efforts. Even if there is a significant minority of less favorable reactions, it is possible to explain them away because many more are positive. Third, school board and central office leaders rarely ask for evidence about the effects of staff development, and if they do ask, almost any evidence is satisfactory. What they value is data they can use to communicate positive results. They are less concerned with the validity or nuances of the data. Fourth, staff developers have invested little in increasing their own learning about effective evaluation and how to use it to improve professional development. It is far easier to rely on evaluation techniques that are not only familiar but also readily acceptable to school system leaders.

In light of these powerful and longstanding disincentives for effective evaluation, why is it now gaining greater attention? Two factors are converging to impinge on professional development. On the one hand, current funding constraints are causing school systems to make difficult budget decisions that reduce the frequency of staff development. This is not new. In tough economic times, school boards often have targeted professional development as a soft activity they can cut with virtually no political risk. On the other hand, school systems are under increasing pressure to demonstrate that they can educate all students to higher levels. Unfortunately, in past decades most school systems invested little in learning how to educate low-performing students more effectively. The result is that many teachers do not have the knowledge and skills to bring below-grade level students to higher levels of performance. Evaluation is important because if educators are going to get the staff development they need to help their students perform proficiently, they will have to demonstrate that adult learning can significantly increase student achievement. The field of staff development needs better evaluation both to improve the effectiveness of teachers' learning experiences and to produce credible evidence that will garner more support for professional development.

Staff developers also need to devote more attention to evaluation because professional development is changing rapidly. Workshops and speakers will probably always be with us, but questions about their effectiveness are prompting more educators to consider other approaches. It was tempting to take the low road to evaluation when the tactics of staff development were rudimentary. Now, however, such sophisticated methods as coaching, study groups, small learning communities, and distance learning are increasing. Although these innovations are promising, even exciting, educators should not once again confuse process with results. In and of themselves, the newer types of staff development mean little. What matters is the degree to which they cause educators to develop and apply practical knowledge and skills that increase student achievement. It will require more intentional, consistent, and robust evaluations to determine whether and how neo-professional development is more powerful than its predecessors.

But how should staff developers begin? The first challenge is to be thoughtful about what to evaluate. This can be confusing because potentially there are multiple points for evaluation. Staff developers can:

Evaluate the delivery of professional development.
Evaluate what educators learn as a result of staff development.
Evaluate whether and how educators apply what they learn.
Evaluate whether and how students benefit as a result of the educators applying what they learn through professional development.

These opportunities for evaluation are not a menu. Staff developers do not have the luxury of choosing to focus on only one or two points of inquiry. Instead, it is necessary to construct a sequential evaluation that not only assesses the strength of each link in the chain of professional development, but increases staff developers' understanding of the connections between each link. Staff development is a process, not an event. It begins with creating learning opportunities for educators and ends with students benefiting from the educators' learning. The higher the quality of the complete process, the more effective professional development will be. This requires staff developers to take a new, holistic view of professional development as a system designed to raise the performance levels of both educators and students.

Creating this alpha and omega approach to staff development will require hard thinking about tangible benefits students should derive from the whole process of professional development. In the past, educators have often said that of course the ultimate goal of staff development should be to benefit students, but they have also said it is not realistic to expect professional development to increase student achievement. Why not? While it is certainly true that not just any activity conducted under the label of staff development will enhance student performance, it should be possible to achieve this objective if educators thoughtfully design and carefully implement professional development for this specific purpose. It is even more possible when staff developers collaborate with school leadership teams to understand student performance data and identify learning gaps teachers need to address. If teachers do not have the knowledge, skills, or curricula to meet students' needs, staff developers should be able to help schools create learning opportunities that will enable the educators to tackle students' learning problems successfully.

The seminal question driving the design of professional development should be, 'What are students' specific learning needs and what does that tell us about educators' specific learning needs?' It is what students need rather than what adults want that should shape educators' learning. If staff developers begin with this question, it is more likely they will succeed in conceiving and implementing adult learning experiences that have a positive, demonstrable effect on student achievement. This is not easy, because it is challenging to forge links in the chain of professional development with "meeting students' learning needs" at one end and "staff development experiences" at the other. It requires a high degree of focus and analysis, and a sophisticated understanding of causal relationships. Nevertheless, this is the challenge educators must meet to demonstrate the power of professional development.

If staff developers start thinking of professional development as a sequential process that begins with engaging educators in learning experiences and culminates with specific benefits to students, they will discover it has profound implications for evaluation. Assessing the effects of staff development will call for a more comprehensive approach than merely gauging participants' immediate reactions to a short-term training. Adult learning only begins with participation in a structured professional development experience. The learning process continues as educators struggle to make meaning of what they have learned and apply it in their daily practice. They only learn the power of the knowledge and skills they have developed when they analyze its effects on their students.

It will be necessary, therefore, to develop evaluation strategies that are integral to the entire process of professional development. While it will continue to be important to evaluate the delivery of staff development, it will be essential to understand what participants actually learn, when and how they begin to apply their learning, and when and how it benefits students. This will require a rollout timeline that projects when each component of the professional development process will occur or when there is likely to be demonstrable evidence of the desired learning or application of learning. If one assumes that staff development, no matter how powerful, may not immediately result in more effective teaching, staff developers must consider what interim indicators, and within what time frame, they should look for as valid measures of progress.

Considering such difficult questions and agreeing on acceptable answers can help staff developers in several ways. The intended, ultimate result of professional development is clearer to more people, creating the basis for educators in diverse roles to collaborate to achieve the result. Staff developers gain allies in their efforts to organize more productive professional development and raise expectations about its outcomes. Engaging others also makes the evaluation process easier because others understand and accept the measures staff developers use to determine professional development's effectiveness. Data collection is less of a chore because educators see that it improves staff development and documents its benefits. Evaluation ceases to be a simplistic, self-justifying exercise and becomes a process that is integral to a school system's or school's overall effort to improve student performance. If staff development is less marginal, and if it is valued more by both front-line educators and central office leaders, they will make a greater effort to protect and then expand it. In short, getting serious about evaluating professional development demonstrates that a school system or school is truly serious about improving teacher and student performance.

As this issue of JSD illustrates, NSDC is taking the lead in raising the awareness of staff developers, and educators generally, about the importance of evaluating professional development, and it is providing examples of how they can develop and execute evaluations more effectively. One can hope that month by month, year by year, more and more educators responsible for professional development will learn how to examine and reflect on the results of their efforts.

But another step is necessary. They must then use the information and insights effective evaluations provide. This can be painful because it often means acknowledging that the staff development one planned and implemented did not produce the results one intended.

No matter how rigorous evaluations may be, if educators do not use their findings to strengthen professional development so it produces better results, the evaluations will be useless. On the other hand, when educators use evaluations to full effect, they can provide an infectious vision for a whole school system or school. They demonstrate they are strong enough to accept and use unfavorable data to improve their efforts. They show that learning is for everyone, no matter what their title or status, and that true learning results in new, more effective behaviors. They breathe new life into the concept of "continuous improvement."

The potential of evaluation, then, is all about positive leadership, and it is only through that leadership, in its many forms, that professional development will become all it must be. That is our destination, even if we have miles to go before we reach it.

About the Author:

Hayes Mizell is NSDC's distinguished senior fellow. You can contact him at 6408 Goldbranch Road, Columbia, SC 29206, (803) 787-0760, fax (803) 787-0769, e-mail: hmizell@msn.com.