Educators are constantly bombarded with new proposals for improving education. The majority of these reforms, however, have come and gone without a lasting impact. This revolving door of new initiatives in education is a product of implementation without critical thought.

How do educators evaluate whether a new initiative is well-conceived and better than another? How do they judge proposed solutions to educational challenges made by a conference speaker or an author? What questions should be considered before deciding whether to change or modify current practice?

The purpose of this guide is to provide questions to assist educators in making important decisions about the validity of an idea and the effectiveness of the initiative linked to that idea. Some questions may involve issues that are more important to you than others. As you go through this evaluation process, consider which issues are most important to your decision. If you ultimately decide to make a change, this exercise should help to defend your decision and to implement that change so that it has a good chance of success.

This guide was composed by Dr. Carol J. Mills, Director of Research, and Dr. Karen Ablard Mickenberg, Senior Researcher, at the Center for Talented Youth at Johns Hopkins University. The authors strongly promote an objective, critical, and empirically-based approach for evaluating educational initiatives.

In addition to research activities, Carol Mills served as a member of the Advisory Committee on Exceptional Children and Youth, U.S. Dept. of State and has coauthored (with Russell Beaton) Renewing our Commitment to the Education of Gifted and Talented Students: An Essential Component of Educational Reform. Karen Ablard Mickenberg reviews for the Journal of Educational Psychology and writes research papers pertaining to education, cognitive development, and achievement motivation.

There are two phases of the decision-making process, each having its own list of questions. Phase One guides you through an evaluation of the merit of any new approach you may be considering. Based on your answers to Phase One, you may decide that the approach in question is not worthy of implementation. On the other hand, if the approach appears to be valid and potentially useful, you should proceed to Phase Two to determine if implementation is feasible and beneficial.

PHASE ONE
CONCEPTION AND RESEARCH EVIDENCE

Is the approach well-conceived?
Is there enough research evidence supporting the effectiveness of the approach?

PHASE TWO
IMPLEMENTATION AND EVALUATION

Is there a need for the new approach?
Are there materials developed to implement this approach?
Is the approach effective; is it better than other approaches or can it enhance current approaches?

PHASE ONE

Determining the validity of a new idea and critically examining the evidence for the effectiveness of the proposed approach linked to that idea.

To make an informed judgment about whether or not a new approach should be implemented, it is important to evaluate the idea behind it in terms of its validity (i.e., well-founded, contains premises that lead to logical conclusions, produces the desired results). Even more important, it is critical to study the research to determine whether or not the approach yields the outcomes that it promises.

Examine the approach and relevant research for yourself; don't rely on unsupported summaries of "what the research says." Not all research is created equal and should be evaluated carefully. The questions posed here provide some guidelines for making such an evaluation. But, you may still be unsure as to whether or not a researcher has reached sound conclusions. If so, don't hesitate to seek advice from someone who does research -- someone who doesn't have a vested interest in the approach being studied.

CONCEPTION

• Is the purpose of the new approach clearly stated and easily understood by those who will have to implement it?
• Does the approach make sense from what you know about children and how they learn?
• Do the proposed outcomes of the approach follow logically from an accepted psychological or educational theory?
• Is the new approach modest and realistic? (Caution: be wary of approaches that claim to be the only or best way. "New" does not always mean best and very few approaches can meet the needs of all students. There are benefits of a balanced approach that combines or chooses different methods depending on a child's needs.)
• Is the approach free of blatant or implied ideology? (Caution: an ideology can bias interpretations and compromise the validity of conclusions supporting an approach.)

RESEARCH EVIDENCE

You can get a general idea of the research evidence by getting an overview of the body of research on an approach, but it is wise to evaluate critically each study.

Body of research evidence for claims

• Are there enough studies examining a new approach to judge its effectiveness? (Caution: one or two small studies are usually not sufficient to judge the effectiveness of an approach that is going to be mandated or replace an existing practice.)
• Have positive results been found repeatedly in separate studies and for a variety of settings? (Caution: although it is preferable if an approach works well across settings and with a variety of students, its effectiveness may vary depending upon the setting and the students with which it is used. This may mean that the approach is valuable only in particular settings or with specific students.)
• Do most of the studies conclude that the approach yields its proposed outcomes?

Individual Studies

• Has the research study been published in a professional, peer-reviewed journal? (Caution: unpublished research reports have not been evaluated by impartial experts and should not be accepted at face value.)
• Is the number of participants in this study large enough to make conclusions? (Caution: the smaller the group studied, the less reliable the findings. Samples over 100, however, tend to inflate the importance of very small effects that have little practical significance.)
• Are conclusions directly related to the sample of students in the study and to the population for which the method is recommended? (Caution: be wary of conclusions that go beyond the sample being studied.)
• Does the researcher account for family background by comparing only students from similar backgrounds or, at least, adjusting study results for the effects of student achievement attributable to family background? (Caution: failure to acknowledge the effects of family background on student achievement violates one of the basic rules of educational research.)
• Are students exposed to the new approach compared to a similar group of students who are not receiving the new approach?
• If the claim is made that one approach is better than another, are both included for comparison in the same study? (Caution: evidence that current approaches are not working cannot be claimed as evidence for the effectiveness of a new approach -- the new must be tested and compared to the old.)
• Is there an established and appropriate measure of student outcomes (e.g., grades, an increase in pre- to post-test achievement test scores, attitude improvement, less absenteeism, etc.)? (Caution: increased student learning should be a top priority. Be suspicious of studies looking only at correlates of learning such as self-esteem or empowerment. These correlates may not directly produce increased learning, but are often by-products.)
• Does the study present statistical evidence to back up its conclusions? Does it demonstrate meaningful differences between two approaches and show "statistically significant" increases in student learning? (Caution: these are rather technical judgments to make; you may need some guidance from an impartial researcher.)

This concludes the questions in Phase One. Based on your answers to the above questions, you may decide that the premise underlying the approach in question or the research evidence supporting its effectiveness is not sufficient to justify changing your existing program. If so, this guide can be put aside until another new approach arises.

If there are no research studies examining the effectiveness of the proposed approach, you may want to wait until some are conducted before considering implementation in your school, especially if your current program is adequate.

If you decide that the approach has merit, continue to Phase Two. This section will guide you in determining if the new approach is needed, if implementation is feasible, and how to evaluate outcomes.

PHASE TWO

Deciding whether to implement the new approach and, if so, how the implementation process should be planned, conducted, and then evaluated.

Since you have moved on to Phase Two questions, it is assumed that you have examined the premise and research behind the approach in question and found them compelling enough to explore the possibility of implementation. At this point, you need to decide if a new approach is needed in your school. If you decide you have a need to try a new approach, you should then determine how and on what level implementation will be accomplished. Finally, you will want to evaluate the effect of your implementation process to decide on appropriate modifications or possible abandonment of the approach.

NEED

• Are current approaches insufficient or not working at all? (Caution: this is a subjective judgment about your existing educational program. If your present program is working well for your students, you may not need a new approach.)
• In your judgment, can this new approach supplement, that is extend or enrich, existing approaches? (Caution: you may not need to substitute the new approach for an existing one. Try to avoid single-strategy instruction and "either-or" thinking.)
• Is the new approach designed to serve a group of students that is currently not being served adequately?
• After considering the research evidence and the claims made by advocates, do the benefits outweigh the costs of this new approach?

IMPLEMENTATION

• Is the new approach developed sufficiently, with materials and technology available for effective implementation? (Caution: without appropriate materials, techniques, and technology, it is often unreasonable to expect teachers to implement effectively many new approaches.)
• Will preparation and staff training be in place before implementation? (Caution: it is of vital importance that persons who will implement the idea understand and support it, and are sufficiently trained to implement it as intended.)
• Has the population that would be best served by implementation been clearly identified? Are we prepared to implement this school-wide or system-wide? Would it be better to do a pilot project and evaluate the results? (Caution: be careful about eliminating an existing approach that is working to replace it with a new approach. It is more prudent to test the new approach by offering it as a supplement to existing practices or piloting it with some of your students and teachers.)

EVALUATION

• Was the new approach appropriately implemented; did staff consistently use the new approach as it was designed? (Caution: the best idea poorly implemented can be worse than doing nothing!)
• Have we given the new approach sufficient time to adequately judge its effectiveness? (Caution: it takes time to tell whether a new approach is working. Some fine tuning is expected when implementing a new approach.)
• Are students learning more now than they were before implementation of the new approach? (Caution: given sufficient time and necessary adjustments, if students are not learning more now than before, don't be afraid to acknowledge that the new approach is not working.)
• Do the benefits of this new approach outweigh its costs? If the new approach was implemented as a pilot project, do the results justify eliminating the old approach? Or, can the new approach enhance an existing program, in combination with current practices or as another option for helping specific students?

CHECKLIST OF MAJOR QUESTIONS

PHASE ONE

CONCEPTION

• Is the purpose of the new approach clear, easily understood, based on an accepted theory, and realistic?

RESEARCH EVIDENCE

• Is there a sufficient body of research demonstrating repeated effectiveness of the approach? Do studies include a representative sample of students, compare approaches, measure success in an appropriate way, and report statistics?

PHASE TWO

NEED

• Are current approaches not working? Can a new approach replace or supplement existing approaches?

IMPLEMENTATION

• Are there materials, technology, and training to effectively implement the approach? Has the population for implementation been clearly identified?

EVALUATION

• Was implementation appropriate and given enough time to adequately judge its effectiveness? Is the new approach better than current approaches and worth the costs?

This document is part of a series of guides published by the Center for Talented Youth of The Johns Hopkins University, Baltimore, Maryland.

Last updated on 1/25/2007