Skip to Content

Notice

National Registry of Evidence-Based Programs and Practices

Document Details

Information about this document as published in the Federal Register.

Published Document

This document has been published in the Federal Register. Use the PDF linked in the document sidebar for the official electronic format.

Start Preamble

AGENCY:

Substance Abuse and Mental Health Services Administration (SAMHSA), HHS.

ACTION:

Notice regarding SAMHSA's NREPP: Redesign of NREPP.

SUMMARY:

The mission of SAMHSA is to reduce the impact of substance abuse and mental illness on America's communities. Established in 1992, the agency was directed by Congress to target effective substance abuse and mental health services to the people most in need, and to translate research in these areas more effectively and more rapidly into the general health care system. NREPP is a key public resource SAMHSA has developed to help meet this directive. This notice announces the redesign of NREPP to better align the registry with the standards and processes of other evidence-based repositories. A re-launch of the Web site with revised content is anticipated in late fall/early winter 2015.

The notice explains the changes in how programs and practices will be identified for NREPP, how submissions will be screened and reviewed, and provides guidance on accessing updated information on the NREPP site. Potential applicants should be aware that this notice includes updated information relating to the eligibility of interventions for inclusion in NREPP and changes in the program and practice review process that supersedes guidance provided in earlier Federal Register notices.

Start Further Info

FOR FURTHER INFORMATION CONTACT:

Carter Roeber, Ph.D., Social Science Analyst, Center for Behavioral Health Statistics and Quality, SAMHSA, 1 Choke Cherry Road, Room 2-1050, Rockville, MD 20857, telephone 240-276-1488.

End Further Info End Preamble Start Supplemental Information

SUPPLEMENTARY INFORMATION:

Advancing Evidence-Based Programs and Practices Through Improved Decision Support Tools: Reconceptualizing NREPP

Introduction

SAMHSA's NREPP is an evidence-based repository and review system designed to provide the public with reliable information about behavioral health interventions in the areas of mental health and substance abuse. Programs and practices that are accepted for inclusion in NREPP undergo a review process that provides information on the quality of research and the magnitude and direction of program or practice impact on individual outcomes. Materials for dissemination are reviewed to determine the type and extent of information available to support implementation. The results of these reviews are published on the NREPP Web site (http://nrepp.samhsa.gov).

It should be noted that inclusion in NREPP indicates that some, but not necessarily all, of the evidence for a program or practice has been reviewed. In some cases, the quality of the research supporting the program or practice may have been determined to be poor or insufficient to earn a rating. Inclusion in NREPP does not constitute endorsement of an intervention as effective by SAMHSA. Moreover, since NREPP has not reviewed all behavioral health interventions, the use of NREPP as an exclusive or exhaustive list of interventions is not appropriate. Policymakers and funders in particular are discouraged from limiting providers and/or potential grantees to selecting exclusively from among NREPP interventions and from funding NREPP interventions regardless of the ratings the interventions receive.

This notice announces changes to (1) the process for identifying new programs and practices for NREPP review, (2) the process for announcing open submission periods, (3) the minimum requirements to be considered for NREPP review, and (4) the review process. This notice also announces the intent to re-review currently posted NREPP programs and practices to comport with new review criteria and ratings. The re-review of programs and practices currently posted will take place over the course of the next several years, depending on available resources. A re-launch of the NREPP Web site will take place in phases and the first phase is planned for late fall/early winter of 2015.

Identifying New Programs and Practices for NREPP

Open submissions periods, during which applicants may submit materials for review, will continue to be used to identify new programs and practices for review (see below). Programs and practices addressing specific SAMHSA priorities may also be identified by SAMHSA or through environmental scans (including literature reviews, focus groups, public input, and interviews), as time and resources permit. Programs and practices related to priority areas may be reviewed before programs and practices identified through the open submission period. SAMHSA will be consulting with subject matter experts and leadership in underserved groups and populations, including American Indian/Alaska Native Tribes, regarding ways to incorporate traditional and culturally-specific interventions into NREPP, in order to better meet the needs of groups whose efforts to promote behavioral health may not have not been routinely evaluated. Innovative, but perhaps less rigorously tested, programs and Start Printed Page 38717practices will be considered if specific to SAMHSA priority areas, as a parallel contribution to the standard review procedures.

Stand-alone pharmacologic treatments are not eligible for review and should not be submitted to NREPP. The evidence base for pharmacologic treatments is reviewed and approved through the U.S. Food and Drug Administration (FDA). FDA approved pharmacotherapy interventions (on-label use) are considered for NREPP review only when combined with one or more behavioral or psychosocial treatments.

SAMHSA reserves the right to reject for review programs and practices whose goals or activities are determined to be inconsistent with SAMHSA's mission, which is “to reduce the impact of substance abuse and mental illness on American communities.”

Open Submission Periods

SAMHSA accepts new applications for review during open submission periods. SAMHSA generally holds one open submission period a year but, depending on the number of reviews in progress and resources available may hold more or fewer within a calendar year. All future open submission periods will be announced on the NREPP Web site. Emails will also be sent announcing the open submission period to those on the NREPP listserv. Anyone wishing to be notified of future open submission periods can join the NREPP listserv by sending a request to nrepp@samhsa.hhs.gov.

Applications can be submitted at any point during an open period. Program and practice developers, researchers, and others interested in submitting an intervention should read below for information about the new minimum requirements to be considered for an NREPP review. Additional future changes to the review process and criteria will be posted on the NREPP Web site as they are implemented. Therefore, before submitting a program or practice for NREPP review, applicants should examine the most recent information posted on the NREPP Web site about the review process and criteria and the most recent guidance for preparing an intervention for submission (see http://www.nrepp.samhsa.gov/​ReviewSubmission.aspx). This guidance will be periodically updated to reflect the redesign of NREPP during the re-launch.

The selection of interventions will take place after the closing of the open submission period, and applicants will be notified whether they have been accepted after an initial screening to ensure that the application meets minimum requirements. The number of reviews conducted will depend on the availability of funds, with the timing of reviews to be determined by SAMHSA. In submitting an intervention, applicants should understand that if interventions are selected for review, the results of NREPP reviews are considered public information and will be posted on the NREPP Web site. Once a review is completed, the applicant will be provided with a summary document (“the program or practice profile”) that presents results of the review, ratings of program effectiveness, and descriptive information about the intervention. The applicant will have the opportunity to comment on the profile before it is posted but they will not have the option to refuse posting.

Minimum Requirements

To be considered for review, interventions must meet three minimum requirements:

1. Research or evaluation of the intervention has assessed mental health or substance use outcomes among individuals, communities, or populations OR other behavioral health-related outcomes on individuals, communities, or populations with or at risk of mental health issues or substance use problems.

2. Evidence of these outcomes has been demonstrated in at least one study using an experimental or quasi-experimental design. Experimental designs require random assignment, a control or comparison group, and pre- and post-intervention outcome assessments. Quasi-experimental designs do not require random assignment, but do require a comparison or control group and pre- and post-intervention outcome assessments. Comparison/control groups must be a no-treatment control group, a wait-list control group, a treatment-as-usual comparison group, or an intervention that is presumed to be ineffective or substantially less effective than the intervention (e.g., a “placebo” control or, in cases in which providing no treatment might be considered unethical, less effective treatments, even if not treatment-as-usual, such as “supportive therapy”). Studies with single-group, pretest-posttest designs or single-group, longitudinal/multiple time series do not meet this requirement, but will be considered to identify emerging programs and practices for consideration in the Learning Center. Comparative effectiveness trials, in which two interventions, both presumed to be equally effective, are compared, and studies in which the effects of the same intervention on various subpopulations are compared or in which various doses or components of the same intervention are compared will be reviewed, but only for purposes of providing readers in the Learning Center the opportunity to determine whether a particular program or practice may have use or show promise in different settings or circumstances.

3. The results of these studies have been published in a peer-reviewed journal or other professional publication, or documented in a comprehensive evaluation report, published within the previous 25 years. Comprehensive evaluation reports must include a review of the literature, theoretical framework, purpose, methodology, findings/results with statistical analysis and p values for significant outcomes, discussion, and conclusions.

Applicants are required to provide full-text documents at the time of submission that demonstrate the intervention meets these minimum requirements. Other research articles, published or unpublished evaluation reports, grant final reports, and replication studies may be submitted as additional supporting documentation. Note: Abstracts or links to partial articles are regarded as incomplete and will not be considered.

NREPP will no longer require programs and practices to have developed implementation materials, training and support resources, and quality assurance procedures. However, programs and practices with such dissemination and/or implementation resources will be considered for prioritized review if within the priorities established by the SAMHSA review process, and the materials, along with the location of their availability, will be listed in the program or practice profile.

Applicants submitting dissemination and implementation resources should include a brief narrative description of the materials that are being submitted. These materials may include, but are not limited to, treatment manuals, information for administrators, information for direct service staff, tested training curricula, mechanisms for ongoing supervision and consultation, protocols for gathering process and outcome data, ongoing monitoring of intervention fidelity, and processes for gathering feedback. Applicants should also provide the location of where the materials can be obtained.Start Printed Page 38718

Selection of Interventions for Review

SAMHSA will select interventions for review from among submissions meeting the minimum requirements. In selecting interventions for review, SAMHSA may give special consideration to interventions that meet one or more of the following conditions:

  • More than one research study or evaluation has been conducted on the same or a similar target population that meets the minimum requirements.
  • The intervention targets underserved populations (e.g., minority populations, tribal communities or American Indian/Alaska Native populations, elderly individuals, young adults, individuals who are incarcerated, etc.).
  • Dissemination and implementation materials (e.g., program or practice manuals, training guides, measurement instruments, implementation fidelity tools) are available. Lower costs and no-cost materials may be prioritized.
  • The intervention contributes to a content area in which few evidence-based interventions have been previously identified.

Interventions that are not selected for review may be resubmitted by the applicant in a future open submission period.

The Review Process

The review process has been revised to improve the quality of the reviews and utility of information that NREPP can provide its users. In addition to articles and reports submitted by NREPP applicants, additional studies, articles, and evaluation reports regarding the interventions will be identified through literature searches. Studies and outcomes to be reviewed will be determined through the systematic application of standardized screening criteria, and the number of studies and outcomes to be reviewed will be expanded to more comprehensively represent the evidence base for the program or practice. Inclusion of studies and outcomes will no longer be limited to positive significant outcomes; all studies and outcomes that meet the standardized screening criteria will be reviewed, including those with negative and non-significant effects. Programs and practices will be assessed on the basis of evaluation studies of program or practice impact, information related to conceptual framework (that is, program or practice goals, theory of change, and program or practice components), and information about implementation fidelity (that is, whether a study employs quality assurance measures to declare that the program or practice is delivered as intended to the program's or practice's target population).

The methodological rigor (that is, internal validity, statistical validity, and measurement validity) of the research for each program or practice will be reviewed, as it pertains to each outcome examined, along with the magnitude and direction of the program's or practice's effect on each outcome. Based on this information, the program's or practice's effectiveness for each outcome will be rated, along with the rigor of the research examining the program or practice, and the ratings will be displayed on the NREPP Web site.

In general, each NREPP evidence review will be conducted by two trained and certified reviewers. However, based on funding and available resources, SAMHSA use one reviewer for programs and practices being re-reviewed. When necessary, NREPP may conduct author queries to confirm or gather additional information needed for the review. Program and practice profiles will be developed on the basis of the information gathered. Applicants will have the opportunity to review the program or practice profile before it is posted on the NREPP site, but they will not have the option to refuse posting.

Dissemination and implementation materials will no longer be rated as they were historically. Instead, descriptions of available materials for each program or practice, highlighting information that may be of most interest to NREPP users, will be included in the program or practice profile, along with information documenting the extent to which materials are available.

Programs and practices currently posted on NREPP will be re-reviewed as time and resources permit but the re-reviews of currently posted programs and practices will take place over the next few years.

Detailed information about the revised review process will be available at http://www.nrepp.samhsa.gov after the re-launch of the new NREPP Web site.

Enhancing the Learning Center

NREPP's Learning Center is a developing and underutilized component of the NREPP Web site. With the evolution and enhancement of the registry, SAMHSA seeks to bring greater recognition to both rigorously evaluated behavioral health interventions and those interventions that have been implemented, demonstrate promise, but have not necessarily been evaluated in a rigorous manner. To that end, the Learning Center is being significantly revamped to support stakeholder engagement and to become a shared learning environment for all stakeholders. SAMHSA recognizes that the successful promotion and dissemination of evidence-based programs and practices requires an environment that promotes community assessment, program and practice planning and evaluation, as well as guidance on the selection and implementation of programs and practices listed on NREPP. There are useful types of evaluation research, often conducted among underserved populations, which provide valuable insights for practitioners, but do not meet the minimum criteria required for experimental or quasi-experimental design. SAMHSA intends the Learning Center to be a forum for presenting research on emerging programs and practices, and exploring ways that pre-experimental and qualitative research can complement and enrich findings from experimental and quasi-experimental research designs. An inventory of such programs and practices will be compiled and maintained within the Learning Center and will operate in parallel to the listing of reviewed programs and practices with experimental and quasi-experimental designs. In this way, SAMHSA intends to support programs and practices researched with the most rigorous approaches while also supporting the development of practice-based evidence, especially for certain populations and emerging practices that are critical to learning and improving behavioral health outcomes for persons with or at risk of developing behavioral health issues.

Start Signature

Summer King,

Statistician.

End Signature End Supplemental Information

[FR Doc. 2015-16573 Filed 7-6-15; 8:45 am]

BILLING CODE 4162-20-P