Skip to Content

Notice

Scientifically Based Evaluation Methods

Document Details

Information about this document as published in the Federal Register.

Published Document

This document has been published in the Federal Register. Use the PDF linked in the document sidebar for the official electronic format.

Start Preamble

AGENCY:

Department of Education.

ACTION:

Notice of proposed priority.

SUMMARY:

The Secretary of Education proposes a priority that may be used for any appropriate programs in the Department of Education (Department) in FY 2004 and in later years. We take this action to focus Federal financial assistance on expanding the number of programs and projects Department wide that are evaluated under rigorous scientifically based research methods in accordance with the Elementary and Secondary Education Act (ESEA) as reauthorized by the No Child Left Behind Act of 2001 (NCLB). Establishing the priority on a Department-wide basis would permit any office to use the priority for a program for which it is appropriate.

DATES:

We must receive your comments on or before December 4, 2003.

ADDRESSES:

Address all comments about this proposed priority to Margo K. Anderson, U.S. Department of Education, 400 Maryland Avenue, SW., room 4W333, Washington, DC 20202-5910. If you prefer to send your comments through the Internet, use the following address: comments@ed.gov.

You must include the term “Evaluation” in the subject line of your electronic message.

Start Further Info

FOR FURTHER INFORMATION CONTACT:

Margo Anderson. Telephone: (202) 205-3010 or via Internet at Margo.Anderson@ed.gov.

If you use a telecommunications device for the deaf (TDD), you may call the Federal Information Relay Service (FIRS) at 1-800-877-8339.

Individuals with disabilities may obtain this document in an alternative format (e.g., Braille, large print, audiotape, or computer diskette) on request to the contact person listed under FOR FURTHER INFORMATION CONTACT.

End Further Info End Preamble Start Supplemental Information

SUPPLEMENTARY INFORMATION:

Invitation To Comment

We invite you to submit comments regarding this proposed priority.

We invite you to assist us in complying with the specific requirements of Executive Order 12866 and its overall requirement of reducing regulatory burden that might result from this proposed priority. Please let us know of any further opportunities we should take to reduce potential costs or Start Printed Page 62446increase potential benefits while preserving the effective and efficient administration of the Department's programs.

During and after the comment period, you may inspect all public comments about this proposed priority in room 4W333, 400 Maryland Avenue, SW., Washington, DC between the hours of 8:30 a.m. and 4 p.m., Eastern time, Monday through Friday of each week except Federal holidays.

Assistance to Individuals With Disabilities in Reviewing the Rulemaking Record

On request, we will supply an appropriate aid, such as a reader or print magnifier, to an individual with a disability who needs assistance to review the comments or other documents in the public rulemaking record for this proposed priority. If you want to schedule an appointment for this type of aid, please contact the person listed under FOR FURTHER INFORMATION CONTACT.

General

The ESEA as reauthorized by the NCLB uses the term scientifically based research more than 100 times in the context of evaluating programs to determine what works in education or ensuring that Federal funds are used to support activities and services that work. This proposed priority is intended to ensure that Federal funds are used to support projects and activities that are consistent with a statutory purpose of Department programs, and evaluated using scientifically based research. Establishing this priority makes it possible for any office in the Department to encourage or to require appropriate projects to use scientifically based evaluation strategies to determine the effectiveness of a project intervention.

Discussion of Proposed Priority

We will announce the final priority in a notice in the Federal Register. We will determine the final priority after considering public comments on this proposed priority and other information available to the Department. This notice does not preclude the Secretary from proposing or funding additional priorities, subject to meeting applicable rulemaking requirements.

Note:

This notice does not solicit applications. In any year in which we choose to use this proposed priority, we invite applications for new awards under the applicable program through a notice in the Federal Register. When inviting applications we designate the priority as absolute, competitive preference, or invitational. The effect of each type of priority follows:

Absolute priority: Under an absolute priority we consider only applications that meet the priority (34 CFR 75.105(c)(3)).

Competitive preference priority: Under a competitive preference priority we give competitive preference to an application by either (1) awarding additional points, depending on how well or the extent to which the application meets the competitive preference priority (34 CFR 75.105(c)(2)(i)); or (2) selecting an application that meets the competitive priority over an application of comparable merit that does not meet the priority (34 CFR 75.105(c)(2)(ii)).

Invitational priority: Under an invitational priority we are particularly interested in applications that meet the invitational priority. However, we do not give an application that meets the invitational priority a competitive or absolute preference over other applications (34 CFR 75.105(c)(1)).

Proposed Priority

The Secretary proposes a priority for program projects proposing an evaluation plan that is based on rigorous scientifically based research methods to assess the effectiveness of a particular intervention. The Secretary intends that this priority will allow program participants and the Department to determine whether the project produces meaningful effects on student achievement or teacher performance.

Evaluation methods using an experimental design are best for determining project effectiveness. Thus, the project should use an experimental design under which participants—e.g., students, teachers, classrooms, or schools—are randomly assigned to participate in the project activities being evaluated or to a control group that does not participate in the project activities being evaluated.

If random assignment is not feasible, the project may use a quasi-experimental design with carefully matched comparison conditions. This alternative design attempts to approximate a randomly assigned control group by matching participants—e.g., students, teachers, classrooms, or schools—with non-participants having similar pre-program characteristics.

In cases where random assignment is not possible and an extended series of observations of the outcome of interest precedes and follows the introduction of a new program or practice, regression discontinuity designs may be employed.

For projects that are focused on special populations in which sufficient numbers of participants are not available to support random assignment or matched comparison group designs, single-subject designs such as multiple baseline or treatment-reversal or interrupted time series that are capable of demonstrating causal relationships can be employed.

Proposed evaluation strategies that use neither experimental designs with random assignment nor quasi-experimental designs using a matched comparison group nor regression discontinuity designs will not be considered responsive to the priority when sufficient numbers of participants are available to support these designs. Evaluation strategies that involve too small a number of participants to support group designs must be capable of demonstrating the causal effects of an intervention or program on those participants.

The proposed evaluation plan must describe how the project evaluator will collect—before the project intervention commences and after it ends—valid and reliable data that measure the impact of participation in the program or in the comparison group.

If the priority is used as a competitive preference priority, points awarded under this priority will be determined by the quality of the proposed evaluation method. In determining the quality of the evaluation method, we will consider the extent to which the applicant presents a feasible, credible plan that includes the following:

(1) The type of design to be used (that is, random assignment or matched comparison). If matched comparison, include in the plan a discussion of why random assignment is not feasible.

(2) Outcomes to be measured.

(3) A discussion of how the applicant plans to assign students, teachers, classrooms, or schools to the project and control group or match them for comparison with other students, teachers, classrooms, or schools.

(4) A proposed evaluator, preferably independent, with the necessary background and technical expertise to carry out the proposed evaluation. An independent evaluator does not have any authority over the project and is not involved in its implementation.

In general, depending on the implemented program or project, under a competitive preference priority, random assignment evaluation methods will receive more points than matched comparison evaluation methods.

Executive Order 12866

This notice of proposed priority has been reviewed in accordance with Executive Order 12866. Under the terms of the order, we have assessed the potential costs and benefits of this regulatory action.

The potential costs associated with the notice of proposed priority are those Start Printed Page 62447we have determined as necessary for administering applicable programs effectively and efficiently.

In assessing the potential costs and benefits—both quantitative and qualitative—of this notice of proposed priority, we have determined that the benefits of the proposed priority justify the costs.

We have also determined that this regulatory action does not unduly interfere with State, local, and tribal governments in the exercise of their governmental functions.

Intergovernmental Review

Some of the programs affected by this proposed priority are subject to Executive Order 12372 and the regulations in 34 CFR part 79. One of the objectives of the Executive order is to foster an intergovernmental partnership and a strengthened federalism. The Executive order relies on processes developed by State and local governments for coordination and review of proposed Federal financial assistance.

This document provides early notification of our specific plans and actions for these programs.

Electronic Access to This Document

You may view this document, as well as all other Department of Education documents published in the Federal Register, in text or Adobe Portable Document Format (PDF) on the Internet at the following site: http://www.ed.gov/​news/​fedregister.

To use PDF you must have Adobe Acrobat Reader, which is available free at this site. If you have questions about using PDF, call the U.S. Government Printing Office (GPO), toll free, at 1-888-293-6498; or in the Washington, DC, area at (202) 512-1530.

Note:

The official version of this document is the document published in the Federal Register. Free Internet access to the official edition of the Federal Register and the Code of Federal Regulations is available on GPO Access at: http://www.gpoaccess.gov/​nara/​index.html.

(Catalog of Federal Domestic Assistance Number does not apply.)

Start Authority

Program Authority: ESEA, as reauthorized by the No Child Left Behind Act of 2001, Pub. L. 107-110, January 8, 2002.

End Authority Start Signature

Dated: October 29, 2003.

Rod Paige,

Secretary of Education.

End Signature End Supplemental Information

[FR Doc. 03-27699 Filed 11-3-03; 8:45 am]

BILLING CODE 4000-01-P