Skip to Content

Notice

Notice: Request for Comments; National Registry of Evidence-Based Programs and Practices (NREPP)

Document Details

Information about this document as published in the Federal Register.

Published Document

This document has been published in the Federal Register. Use the PDF linked in the document sidebar for the official electronic format.

Start Preamble Start Authority

Authority: Sec. 501, Pub. L. 106-310

End Authority

SUMMARY:

The Substance Abuse and Mental Health Services Administration (SAMHSA) is committed to preventing the onset and reducing the progression of mental illness, substance abuse and substance related problems among all individuals, including youth. As part of this effort, SAMHSA is expanding and refining the agency's National Registry of Evidence-based Programs and Practices (NREPP) so that the system serves as a leading national resource for contemporary and reliable information on the scientific basis and practicality of interventions to prevent and/or treat mental illness and substance use and abuse.

NREPP represents a major agency activity within SAMHSA's Science to Service initiative. The initiative seeks to accelerate the translation of research into practice by promoting the implementation of effective, evidence-based interventions for preventing and/or treating mental disorders and substance use and abuse. Of equal measure, the initiative emphasizes the essential role of the services community in providing input and feedback to influence and better frame the research questions and activities pursued by researchers in these areas.

Through SAMHSA's Science to Service initiative, the agency ultimately seeks to develop a range of tools that will facilitate evidence-based decision-making in substance abuse prevention, mental health promotion, and the treatment of mental and substance use disorders. In addition to NREPP, SAMHSA is developing an informational guide of web-based Start Printed Page 50382resources on evidence-based interventions that will be available in 2006. SAMHSA also is exploring the feasibility of supporting a searchable web database of evidence-based information (e.g., systematic reviews, meta-analyses, clinical guidelines) for mental health and substance abuse prevention and treatment providers. Such a system could reduce the lag time between the initial development and broader application of research knowledge by serving as a real-time resource to providers for “keeping current” in ways that will enhance their delivery of high quality, effective services. In combination, these three tools—NREPP, guide to web-based resources, and database of evidence-based information—would provide valuable information that can be used in a variety of ways by a range of interested stakeholders.

With regard to NREPP, during the past two years, SAMHSA convened a series of scientific/stakeholder panels to inform the agency's expansion of the system to include interventions in all substance abuse and mental health treatment and prevention domains. These panels thoroughly assessed the existing NREPP review process and review criteria and provided comments and suggestions for refining and enhancing NREPP. As part of this expansion effort, SAMHSA also engaged a contractor to assess the NREPP process and review criteria, including how the system and criteria compare to other, similar evidence review and rating systems in the behavioral and social sciences. The cumulative results of these activities have guided efforts to refine the NREPP review process and review criteria, as well as inform the agency's plans for how such a system may be used to promote greater adoption of evidence-based interventions within typical community-based settings.

This Federal Register Notice (FRN) provides an opportunity for interested parties to become familiar with and comment on SAMHSA's plans for expansion and use of NREPP.

DATES:

Submit comments on or before October 25, 2005.

ADDRESSES:

Address all comments concerning this notice to: SAMHSA c/o NREPP Notice, 1 Choke Cherry Road, Rockville, MD 20857. See SUPPLEMENTARY INFORMATION for information about electronic filing.

Start Further Info

FOR FURTHER INFORMATION CONTACT:

Kevin D. Hennessy, PhD, Science to Service Coordinator/SAMHSA, 1 Choke Cherry Road, Room 8-1017, Rockville, Maryland 20857. Dr. Hennessy may be reached at (240) 276-2234.

End Further Info End Preamble Start Supplemental Information

SUPPLEMENTARY INFORMATION:

Electronic Access and Filing Addresses

You may submit comments by sending electronic mail (e-mail) to nrepp.comments@samhsa.hhs.gov.

Start Signature

Dated: August 18, 2005.

Charles G. Curie,

Administrator.

End Signature

Overview

Increasingly, individuals and organizations responsible for purchasing, providing and receiving services to prevent substance abuse and/or treat mental and substance use disorders are considering the extent to which these services are “evidence-based”—that there exists some degree of documented scientific support for the outcomes obtained by these services. As the Federal agency responsible for promoting the delivery of substance abuse and mental health services, SAMHSA is particularly interested in supporting and advancing activities that encourage greater adoption of effective, evidence-based interventions to prevent and/or treat mental and substance use disorders. With this in mind, SAMHSA proposes to refine and expand its National Registry of Evidence-based Programs and Practices (NREPP). SAMHSA believes that the growth and evolution of NREPP can serve as an important mechanism for promoting greater adoption of evidence-based substance abuse and mental health services,—one that can do so in conjunction with an ever-growing array of scientific knowledge, clinical expertise and judgment, and patient/recipient values and perspectives. By clearly identifying and assessing the scientific basis and disseminability of a range of behavioral interventions, NREPP is likely to prove an important resource to both individuals and systems seeking information on the effectiveness of various services to prevent and/or treat mental and substance use disorders.

Background and Need

As SAMHSA promotes the identification and greater use of effective, evidence-based interventions for individual-, population-, policy-, and system-level changes, the agency seeks to build upon the strong foundation provided by the precursor to the National Registry of Evidence-based Programs and Practices—namely, the National Registry of Effective Prevention Programs. The previous system provides an important building block in the agency's efforts to develop a SAMHSA-wide registry.

The National Registry of Effective Prevention Programs developed in SAMHSA's Center for Substance Abuse Prevention (CSAP) beginning in 1997 as a way to help professionals in the field become better consumers of prevention programs. Between 1997 and 2004, NREPP reviewed and rated more than 1,100 prevention programs, with more than 150 obtaining designation as a Model, Effective, or Promising Program.

Information on all current NREPP programs is available through the Model Programs Web site at http://www.modelprograms.samhsa.gov. Additional details about the review process, review criteria, and rating system for the National Registry of Effective Prevention Programs are available in the SAMHSA publication “Science-Based Prevention programs and Principles 2002,” which can be downloaded from SAMHSA's Model Programs Web site http://www.modelprograms.samhsa.gov) by clicking on “Publications” on the tool bar on the left side of the page; or by requesting the publication through SAMHSA's National Clearinghouse for Alcohol and Drug Information (NCADI) at 1-800-729-6686 (or by visiting the NCADI Web site at http://www.health.org).

As SAMHSA expands the NREPP system, one area of potential improvement is in the efficient screening and triage of applications. Given the historical applications trends among substance abuse prevention programs, combined with the increased demands on the system through expansion to other SAMHSA domains, it is essential that the agency develop a transparent and scientifically defensible process for screening and triaging applications.

Moreover, as SAMHSA engaged NREPP scientific/stakeholder panels over the past 2 years, concerns about the existing review process and review criteria emerged. In particular, a range of scientific experts voiced concerns regarding specific review criteria and other elements of the review process.

In addition, systematic efforts to examine and compare the current NREPP review criteria with other evidence-grading systems in the social and behavioral sciences has revealed both areas of relative strength and relative weakness. At a minimum, this comparison has affirmed for SAMHSA the importance and value of reexamining and refining the NREPP review process and review criteria in ways that reflect to the public SAMHSA's commitment to identifying Start Printed Page 50383and promoting interventions that have shown to be effective through prevailing scientific standards. One important element of this process is providing support for the re-review of existing NREPP programs against these prevailing scientific standards (see below), while another component is identifying both SAMHSA and other mechanisms and resources for supporting efforts to evaluate and document the evidence-base of innovative interventions in ways that will maximize their opportunity for entry into NREPP.

Further, SAMHSA's experience with NREPP to date suggests that the system is limited in its ability to identify and rate interventions designed to promote population-, policy-, and system-level outcomes, such as those promoted by community prevention coalitions. SAMHSA's plans for NREPP include an expansion of the system in this area. As part of this expansion, SAMHSA proposes a second set of review criteria for these interventions, with the recognition that some interventions may be designed to affect a community over time, and that the prevailing scientific standards for assessing the effectiveness of these interventions may indeed be different than those for interventions seeking to change individual-level outcomes. Finally, input into the NREPP process to date suggests the need for SAMHSA to provide greater policy guidance on how best to use the system to appropriately select specific interventions, as well as contextual guidance on how NREPP might be used in conjunction with other important information—such as clinical expertise, patient values, and administrative and policy perspectives and data—in making decisions regarding the financing and delivery of substance abuse and mental health services.

Proposal

After extensive consultation with both scientific experts and a range of stakeholders. SAMHSA is seeking your comments on a proposal to advance a voluntary rating and classification system for mental health and substance abuse prevention and treatment interventions—a system designed to categorize and disseminate information about programs and practices that meet established evidentiary criteria. This proposal presents in detail the new NREPP system, including refinements to the review process and review criteria for programs and practices in substance abuse and mental health, as well as an expansion of the system to include successful community coalition efforts to achieve population-, policy-, or system-level outcomes. The proposal also describes SAMHSA's plans for a new Web site that will highlight the scientific evidence-base of interventions, as well as provide a range of practical information needed by those who are considering implementation of programs or practices on the Registry.

SAMHSA further anticipates that additional revisions and refinements to the NREPP system may be needed on a periodic basis, and proposes the formation of an external advisory panel to regularly assist the agency in assessing proposed suggestions for improvements to the system (see Question # 10 below).

Initial Input From the Field

Upon determining that SAMHSA would expand NREPP to include interventions in all agency domains, three expert panel meetings of both scientist and nonscientist stakeholders were convened to provide feedback on the current review system and review criteria, as well as solicit suggestions about redesigning the system to promote the goals noted above. Each meeting was conducted over a 2-day period, and included invited participants representing a range of relevant organizations, expertise, and perspectives. All meetings took place in Washington, DC, in 2003, with mental health experts meeting in April, substance abuse prevention and mental health promotion experts meeting in September, and substance abuse treatment experts meeting in December. Transcripts of these meetings are available on-line at the NREPP Web page accessible through the “Quick Picks” section on SAMHSA's Home page (http://www.samhsa.gov).

SAMHSA also convened a meeting in May 2005 to solicit recommendations for integrating evidence-based findings from community coalitions into NREPP. The 2-day meeting brought together prominent researchers and practitioners who reaffirmed the importance of including prevention coalitions within NREPP, and offered suggestions as to the types of outcomes and evidence criteria appropriate to the assessment of community coalitions. A summary of this meeting is available on-line at the NREPP web page accessible through the “Quick Picks” section on SAMHSA's Home page (http://www.samhsa.gov).

Review Process for Determining Individual-Level Outcome Ratings for Interventions

A primary goal of the Registry is to provide the public with reliable information about the evidence quality and strength of scientific support for specific interventions. The strength of scientific support includes: the quality of evaluation design (e.g., experimental or quasi-experimental designs); fidelity to predetermined intervention components; confidence in the link between intervention components and specific outcome(s) achieved; freedom from internal and external sources of bias and error; and the statistical significance and practical magnitude (e.g., effect size) of outcomes achieved.

An additional goal is to provide key information about the transferability of these programs and practices to real-world prevention and treatment settings. NREPP utility descriptors provide information about the appropriate settings and populations for implementation of specific interventions, the availability of training and dissemination materials, and their practicality and costs.

This section describes the NREPP review process, including the evidence rating criteria and utility descriptors that will form the basis for Web-based information about programs and practices.

Based on important feedback from scientists and practitioners in the prevention and treatment fields, the NREPP review process has been enhanced in several important respects:

—Programs and practices will be rated on the strength of evidence for specific outcomes achieved, rather than on global assessments of the effectiveness of intervention(s). In addition, indicators of strength of association or magnitude of outcome effects, such as effect size statistics, will be utilized in NREPP to complement traditional, statistical significance (null-hypothesis) testing.

—There will be multiple, outcome-specific ratings of evidence quality strength. All programs and/or practices listed on the Registry will be considered “Effective” for demonstrating specific outcomes having varying levels of evidence quality and confidence resulting from independent (or applicant) replication(s).

—Evidence rating criteria have been refined and now emphasize intervention impacts, evaluation design and fidelity, quality of comparison conditions, and replications.

The section below is an overview of the NREPP process for obtaining expert reviewers' ratings of the evidence quality for outcome-specific program and practice interventions. The process includes an internal screening and triage process conducted by qualified Start Printed Page 50384NREPP contractor staff serving as review coordinators, as well as an independent, external scientific review process conducted by qualified and trained external scientists—working independently of one another—to assess the evidence quality of candidate interventions.

I. Submitted Materials

Applicants will submit a variety of documents that allow a panel of expert reviewers to rate objectively the evidence for an intervention's impact on substance abuse or mental health outcomes. Materials submitted may include:

  • Descriptive program summaries, including the specific outcomes targeted;
  • Research reports and published and unpublished articles to provide scientific evidence (experimental or quasi-experimental studies) about the effectiveness of the intervention for improving specific outcomes;
  • Documents to verify that participants were assured privacy and freedom from negative consequences regardless of participating (used to assess participant response bias, not human subjects protections per se); and
  • Documents to provide evidence that outcomes and analytic approaches were developed through a theory-driven or hypothesis-based approach.

These materials will provide SAMHSA and potential reviewers with objective evidence to support conclusions about the validity and impact of the program or practice. Reviewers must be assured that program investigators did not capitalize on chance findings or excessive postintervention data analyses to find effects that were not components of the intervention design or theory.

II. Internal Review and Triage

Upon receipt, each set of materials will be assigned to an NREPP review coordinator (contractor staff), who will inventory and document the contents of the submission. The review coordinator will contact the applicant by phone and/or e-mail confirming receipt and notifying the applicant if additional application materials are required.

When all materials for a program have been received by the review coordinator, an internal review is conducted to eliminate those programs or practices that do not meet NREPP minimum evidence-based standards. These minimum standards are (1) An intervention that is consistent with SAMHSA's matrix of program priority areas; (2) one or more evaluations of the intervention using an experimental or quasi-experimental design; and (3) statistically significant intervention outcome(s) related to either the prevention or treatment of mental or substance use disorders. Non-reviewed programs will receive written notification of this decision, including problematic or missing components of their application, and may be considered for re-review at a later date.

SAMHSA will maintain oversight over the entire NREPP application and selection process. Moreover, SAMHSA's Administrator and Center Directors may establish specific program and practice areas for priority review.

III. Ratings by Reviewers

For all NREPP applications determined appropriate for review, three (3) independent, external reviewers will evaluate and rate the intervention. Reviewers are doctoral-level researchers and practitioners who have been selected based on their training and expertise in the fields of mental health promotion or treatment and/or substance abuse prevention or treatment. Moreover, reviewers will be thoroughly trained in the NREPP review process, and will be monitored and provided feedback periodically on their performance. Of note, interventions targeting individuals or populations with co-occurring mental health and substance use disorders (or other cross-domain initiatives) will be assigned to reviewers across these categorical domains.

To maintain the objectivity and autonomy of the peer review process, SAMHSA will not disclose to the public or applicants the identities of individual reviewers assigned to specific reviews. On a periodic basis, SAMHSA may post listings of reviewer panels within specific SAMHSA domains as an informational resource to the public.

Reviewers will be selected based on their qualifications and expertise related to specific interventions and/or SAMHSA priority areas. In addition to reviewers identified by SAMHSA, NREPP will consider third-party and self-nominations to become part of the reviewer pool. All reviewers will provide written assurance, to be maintained on file with the NREPP contractor, that they do not have a current or previous conflict of interest (e.g., financial or business interest) that might impact a fair and impartial review of specific programs or practices applying for NREPP review.

Reviewers provide independent assessments of the evidence quality and provide numerical summary scores across the 16 outcome-specific evidence quality criteria. Each criterion is scored on a 0 to 4 scale. The 16 evidence quality criteria are presented below.

Individual-Level Outcome Evidence Rating Criteria

1. Theory-Driven Measure Selection

Outcome measures for a study should be selected before data are collected and should be based on a priori theories of hypotheses.

0 = The applicant selected the measure after data collection for the apparent purpose of obtaining more favorable results than would be expected from using the measures originally planned, OR there is no documentation of selection prior to data analysis.

4 = Documentation shows that the applicant selected the measure prior to study implementation, OR the measure was selected after study inception, but before data analysis, and is supported by a peer review panel or literature related to study theories or hypotheses.

2. Reliability

Outcome measures should have acceptable reliability to be interpretable. “Acceptable” here means reliability at a level that is conventionally accepted by experts in the field.

0 = No evidence of measure realibility.

1 = Reliability coefficients indicate that some but not all relevant types of reliability (e.g., test-retest, inter-rater, inter-item) are acceptable.

3 = All relevant types of realibility have been documented to be at acceptable levels in studies by the applicant.

4 = All relevant types of reliability have been documented to be acceptable levels in studies by independent investigators.

3. Validity

Outcome measures should have acceptable validity to be interpretable. “Acceptable” here means validity at a level that is conventionally accepted by experts in the field.

0 = No evidence of measure validity, or some evidence that the measure is not valid.

1 = Measure has face validity.

3 = Studies by applicant show that measure has one or more acceptable forms of criterion-related validity that are correlated with appropriate, validated measures or objective criteria; OR, as an objective measure of response, there are procedural checks to confirm data validity, but they have not been adequately documented.

4 = Studies by independent investigators show that measure has Start Printed Page 50385one or more acceptable forms of criterion-related validity that are correlated with appropriate, validated measures or objective criteria; OR, as an objective measure of response, there are adequately documented procedural checks that confirm data validity.

4. Intervention Fidelity

The “experimental” intervention implemented in a study should have fidelity to the intervention proposed by the applicant. Instruments that have tested acceptable psychometric properties (e.g., interrator reliability, validity as shown by positive association with outcomes) provides the highest level of evidence.

0 = There is evidence the intervention implemented was substantially different from the one proposed.

1 = There is only narrative evidence that the applicant or provider believes the intervention was implemented with acceptable fidelity.

2 = There is evidence of acceptable fidelity in the form of judgment(s) by experts, based on limited on-site evaluation and data collection.

3 = There is evidence of acceptable fidelity, based on the systematic collection of data on factors such as dosage, time spent in training, adherence to guidelines or a manual, or a fidelity measure with unspecified or unknown psychometric properties.

4 = There is evidence of acceptable fidelity from a tested fidelity instrument shown to have reliability and validity.

5. Comparison Fidelity

A study's comparison condition should be implemented with fidelity to the comparison condition proposed by the applicant. Instruments for measuring fidelity that have tested acceptable psychometric properties (e.g., interrater reliability, validity as shown by predicted association with outcomes) provide the highest level of evidence.

0 = There is evidence that the comparison condition implemented was substantially different from one proposed.

1 = There is only narrative evidence that the applicant or provider believes the comparison condition was implemented with fidelity.

2 = Researchers report observational or administrative data that the comparison condition was implemented with fidelity.

3 = Documentation confirms that comparison group participants did not receive interventions that were very similar or identical to intervention participants, AND there is documentation of degree of participation in any comparison conditions such as lectures or treatment.

4 = There is evidence from a tested instrument suggesting that the comparison condition was implemented with fidelity.

6. Nature of Comparison Condition

The quality of evidence for an intervention depends in part on the nature of the comparison condition(s), including assessments of their active components and overall effectiveness. Interventions have the potential to cause more harm than good; therefore, an active comparison intervention should be shown to be better than no treatment.

0 = There was no comparison condition.

1 = The comparison condition is an active intervention that has not been proven to better than no treatment.

2 = The comparison condition is no service or wait-list, or an active intervention shown to be as effective as or better than no treatment.

3 = The comparison condition is an attention control.

4 = The comparison condition was shown to be as safe or safer and more effective than an attention control.

7. Assurances to Participants

Study participants should always be assured that their responses will be kept confidential and not affect their care or services. When these procedures are in place, participants are more likely to disclose valid data.

0 = There was no effort to encourage and reassure subjects about privacy and that consent or participation would have no effect on services.

1 = Data collector was the service provider, AND there were documented assurances to participants about privacy and that consent or participation would have no effect on care or services.

2 = Data collector was not the service provider. There were indications, but no documentation, that participants were assured about their privacy and that consent or participation would have no effect on care or services.

4 = Data collector was not the service provider, AND there were documented assurances to participants about privacy and that consent or participation would have no effect on care or services; OR, data were not collected directly from participants.

8. Participant Expectations

Participants can be biased by how an intervention is introduced to them and by an awareness of their study condition. Information used to recruit and inform study participants should be carefully crafted to equalize expectations. Masking treatment conditions during implementation of the study provides the strongest control for participant expectancies.

0 = Investigators did not make adequate attempts to mask study conditions or equalize expectations among participants in the experimental and comparison conditions, or differential participant expectations were measured and found to be too great to control for statistically.

2 = Investigators attempted to mask study conditions or equalize expectations among participants in the experimental and comparison conditions. Some participants appeared likely to have known their study condition assignment (experimental or comparison).

3 = Investigators attempted to mask study conditions or equalize expectations among participants in the experimental and comparison conditions. Some participants appeared likely to have known their study condition assignment (experimental or comparison), but these differential participant expectations were measured and appropriately controlled for statistically.

4 = Investigators adequately masked study conditions. Participants did not appear likely to have known their study condition assignment.

9. Standardized Data Collection

All outcome data should be collected in a standardized manner. Data collectors trained and monitored for adherence to standardized protocols provide the highest quality evidence of standardized data collection.

0 = Applicant did not use standardized data collection protocols.

2 = Data was collected using standardized protocol and trained data collectors.

3 = Data was collected using standardized protocol and trained data collectors, with evidence of good initial adherence by data collectors to the standardized protocol.

4 = Data was collected using standardized protocol and trained data collectors, with evidence of good initial adherence to data collectors to the standardized protocol and evidence of data collector retraining Start Printed Page 50386when necessary to control for rater “drift.”

10. Data Collector Bias

Data collector bias is most strongly controlled when data collectors are not aware of the conditions to which study participants have been assigned. When data collectors are aware of specific study conditions, their expectations should be controlled for through training and/or statistical methods.

0 = Data collectors were not masked to participants' conditions, and nothing was done to control for possible bias, OR collector bias was measured and found to be too great to control for statistically.

2 = Data collectors were not masked to participants' conditions, but data collectors received training to reduce possible bias.

3 = Data collectors were not masked to participants' conditions; possible bias was appropriately controlled for statistically.

4 = Data collectors were masked to participants' conditions.

11. Selection Bias

Concealed random assignment of participants provides the strongest evidence of control for selection bias. When participants are not randomly assigned, covariates and confounding variables should be controlled as indicated by theory and research.

0 = There was no comparison condition, OR participants or providers selected conditions.

3 = Participants were not assigned randomly, but researchers controlled for theoretically relevant confounding variables, OR participants were assigned with non-concealed randomization.

4 = Selection bias was controlled with concealed random assignment.

12. Attrition

Study results can be biased by participant attrition. Statistical methods as supported by theory and research can be employed to control for attrition that would bias results, but studies with no attrition needing adjustment provide the strongest evidence that results are not biased.

0 = Attrition was taken into account inadequately, OR there was too much attrition to control for bias.

1 = No significant differences were found between participants lost to attrition and remaining participants.

2 = Attrition was taken into account by simpler methods that crudely estimate data for missing observations.

3 = Attrition was taken into account by more sophisticated methods that model missing data, observations, or participants.

4 = There was no attrition, OR there was no attrition needing adjustment.

13. Missing Data

Study results can be biased by missing data. Statistical methods as supported by theory and research can be employed to control for missing data that would bias results, but studies with no missing data needing adjustment provide the strongest evidence.

0 = Missing data were an issue and were taken into account inadequately, OR levels of missing data were too high to control for bias.

1= Missing data were an issue and were taken into account, but high quantity makes the control for bias suspect.

2= Missing data were an issue and were taken into account by simpler methods (mean replacement, last point carried forward) that simplistically estimate missing data; control for missing data is plausible.

3= Missing data were an issue and were taken into account by more sophisticated methods that model missing data; control for missing data very plausible.

4= Missing data were not an issue.

14. Analysis Meets Data Assumptions

The appropriateness of statistical analyses is a function of the properties of the data being analyzed and the degree to which meet statistical assumptions.

0= Analyses were clearly inappropriate to the data collected; severe violation(s) of assumptions make analysis uninterpretable.

1= Some data were analyzed appropriately, but for other analyses important violation(s) of assumptions cast doubt on interpretation.

2= There were minor violations of assumptions for most or all analyses, making interpretation of results arguable.

3= There were minor violations of assumptions for only a few analyses; results were generally interpretable.

4= There were no violations of assumptions for any analysis.

15. Theory-Driven Selection of Analytic Methods

Analytic methods should be selected for a study based on a priori theories or hypotheses underlying the intervention. Changes to analytic methods after initial data analysis (e.g., to “dredge” for significant results) decrease the confidence that can be placed in the findings.

0= Analysis selected appears inconsistent with the intervention theory or hypotheses; insufficient rational provided by investigator.

1= Analysis selected appears inconsistent with the intervention theory or hypotheses, but applicant provides a potentially viable rationale.

3= Analysis is widely accepted by the field as the most consistent with study theory or hypotheses; no documentation showing methods were selected prior to data analysis.

4= Analysis is widely accepted by the field as the most consistent with study theory or hypotheses; documentation shows that methods were selected prior to data analysis.

16. Anomalous Findings

Findings that contradict the theories and hypotheses underlying an intervention suggest the possibility of confounding causal variables and limit the validity of study findings.

0 = There were anomalous findings suggesting alternate explanations for outcomes reported.

4 = There were no anomalous findings, OR researchers explained anomalous findings in a way that preserves the validity of results reported.

Based upon the independent reviewer assessments, review coordinators will compute average evidence quality ratings for specific outcome measures (based on the 16 evidence quality criteria), and then ask reviewers to determine the overall intervention outcome evidence ratings according to two components: quality of evidence and intervention replications. Average evidence quality ratings scores below 2.0 will be considered “insufficient current evidence” for the effectiveness of a given outcome, and will not be included in the Registry. Evidence quality rating scores of 2.0 to 2.5 will be considered “emerging evidence” for effectiveness, and scores of 2.5 and higher (4.0 is the maximum) will be considered “strong evidence.”

Specific rating category labels for effective outcomes remain to be finalized, but might include categories such as: (1) Strong evidence with independent replication(s); (2) Strong evidence with developer replication(s); (3) Strong evidence without replication; (4) Emerging evidence with independent replication(s); (5) Emerging evidence with developer replication(s); and (6) Emerging evidence without replication.

IV. Applicant Notification

Applicants will be notified in writing of their final NREPP rating category(s) by the SAMHSA Administrator or his/Start Printed Page 50387her designee within 2 weeks following the completion of the review. This notification will include the summary comments of reviewers as well as the consensus ratings on each review criteria. Where relevant, the notification letter will provide applicants with the effective date of the program status designation.

Applicants will have the opportunity to appeal any review decision by submitting a written request to the NREPP contractor within 30 days of notification. Appeals will be resolved through the assignment of two (2) additional reviewers to conduct focused reviews of the evidence quality for specific, disputed outcome ratings. Reviews conducted as part of a formal appeal process will be independent and reviewers will be unaware of previous ratings and decisions. The numeric evidence ratings will be averaged across the five (i.e., 3 original; 2 appeal) reviews for a final determination of intervention outcome rating(s).

V. Utility Descriptors

The NREPP utility descriptors will provide information to program purchasers and planners, service providers, consumers and the general public about the transferability of intervention technologies to different (including non-research-based) settings. These descriptors complement NREPP's scientific evidence-based program and practice ratings with information pertaining to the following dimensions:

1. Implementation. What kinds of materials are available to support implementation and what audiences are they appropriate for? What kinds of training and training resources are available to support implementation?

2. Quality Monitoring. What tools, procedures, and documentation are available to support quality monitoring and quality improvement as the program or practice is implemented?

3. Unintended or Adverse Events. What procedures, systems and data have been identified to indicate whether the intervention has ever resulted in unintended or adverse events?

4. Population Coverage. Were the study samples recruited representative of the persons identified to receive the intervention in the theory/framework?

5. Cultural Relevance and Cultural Competence. Were the outcomes demonstrated to be effective and applicable to specific demographic and culturally defined groups? If so, are training and other implementation materials available to promote culturally competent implementation of the intervention?

6. Staffing. What is aggregate level of staffing (e.g., FTEs) required to implement the intervention effectively? What are the individual skills, expertise, and training required of staff to deliver the intervention?

7. Cost. What are the estimated start-up and annual costs per person served and unit of service for the intervention at full operation?

Interventions with outcomes achieving any one of the effective statuses will be asked to provide descriptive information about the intervention's readiness for implementation, appropriateness for different populations, freedom from unintended or adverse effects, and staffing and cost requirements. These utility descriptors will be featured on the NREPP Web site to help assure a proper match between specific prevention and treatment interventions and the settings and populations to which they are most appropriate.

In light of SAMHSA's commitment to consumer and family involvement, the agency is seeking ways to ensure that these groups provide input into the assessment of interventions that achieve NREPP status. SAMHSA seeks suggestions regarding the most useful and efficient way to conduct this process (see Question 7 below).

Review Process for Determining Population-, Policy-, and System-Level Outcome Ratings for Interventions

The NREPP Evidence Rating Criteria for Population-, Policy-, and System-Level Outcomes are proposed as the basis for reviewer ratings of outcomes generated by community prevention coalitions and other environmental interventions to promote resiliency and recovery at the community level. SAMHSA's rationale for use of these separate criteria comes through a recognition that some interventions may be designed to affect a community over time, and that the prevailing scientific standards for assessing the effectiveness of these interventions may indeed be different than those for interventions seeking to change individual-level outcomes.

An outcome of a prevention or treatment intervention qualifies for review under these 12 criteria only when it can be included in one of the following three categories:

1. Population-Level Outcome—measures the effect of an intervention of an existing, predefined population. Examples of such existing, predefined populations include “all youth residing in a neighborhood,” “all female employees of a manufacturing plant,” or “all Native Americans receiving social services from a tribal government.” “All patients receiving a specific treatment,” in contrast, cannot be defined as an existing, predefined population because that population would have come into existence as a direct response to the intervention.

2. Policy-Level Outcome—measures the effect of an intervention on enactment, maintenance, or enforcement of policies that are assumed to have a positive aggregate impact on resiliency or recovery. Examples of such outcomes include “the rate of passage of legislation restricting access to alcoholic beverages” or “the percentage of arrests for illicit drug manufacturing that result in convictions.”

3. System-Level Outcome—measures the effect of an intervention on prevention and treatment capacity, efficiency, or effectiveness in an existing system or community. Examples of such outcomes include “increased capacity of a State government to quantify alcohol- or drug-related problems” or “increased effectiveness of a community treatment system to respond to the comprehensive needs of individuals with Axis I mental health diagnoses.”

To support the transparency of the review process, SAMHSA wants stakeholders to understand clearly the NREPP procedures and decision-making processes. All community coalition interventions included in NREPP will have demonstrated evidence of effectiveness at the population, policy, or system level. The ratings will indicate the strength of the supporting evidence, and may be as follows:

(1) Strong evidence with replication;

(2) Strong evidence without replication;

(3) Emerging evidence with replication; and

(4) Emerging evidence without replication.

All NREPP evidence ratings are defined at the level of specific outcomes. The 12 evidence rating criteria used for population-, policy- and system-level outcomes, summarized as an average Evidence Quality Score (EQS) for each outcome, allow independent expert reviewers to score along dimensions of outcome measurement, intervention fidelity, comparison conditions, participant and data collector biases, design and analysis, and anomalous findings. Each of the 12 criteria is assessed by independent reviewers on a 0 to 2 scale, in which a “1” indicates that methodological rigor may have been acceptable and a “2” indicates that adequate methodological rigor was achieved for this type of outcome. Start Printed Page 50388Preliminary discussions of classifications have suggested that “Strong evidence” be defined as an average EQS of 1.75 or above (out of a possible 2.0), and that “Emerging evidence” be defined as an average EQS between 1.50 and 1.74 (out of a possible 2.0).

The 12 criteria applied to each population-, policy-, or system-level outcome measures are:

1. Logic-Driven Selection of Measures

2. Reliability

3. Validity

4. Intervention Fidelity

5. Nature of Comparison Condition

6. Standardized Data Collection

7. Data Collector Bias

8. Population Studied

9. Missing Data

10. Analysis Meets Data Assumptions

11. Theory-Driven Selection of Analytic Methods

12. Anomalous Findings

Outcome Measurement Criteria

1. Logic-Driven Selection of Measures

Outcome measures should be based on a theory or logic model that associates them with the intervention.

0 = The applicant appears to have selected outcome measures for the purpose of identifying favorable results rather than from a logic-based rationale.

1 = There is no explicit description of a guiding logic model or theory for measures, although a rationale for the inclusion of most measures can be inferred.

2 = Measures are supported by a theory or logic model that associates the intervention with the outcome.

2. Reliability

Outcome measures should have acceptable reliability to be interpretable. “Acceptable” here means reliability at a level that is conventionally accepted by experts in the field.

0 = No evidence of reliability of measures is presented.

1 = Relevant reliability measures are in the marginal range.

2 = Relevant reliability measures are in clearly acceptable ranges.

3. Validity

Outcome measures should have acceptable validity to be interpretable.

0 = No evidence of validity of measures is presented or evidence that is presented suggests measures are not valid.

1 = Measures has face validity.

2 = Relevant validity has been documented to be at acceptable levels in independent studies.

4. Intervention Fidelity

The intervention should be well defined and its implementation should be described in sufficient detail to assess whether implementation affected outcomes.

0 = The intervention and/or its implementation are not described in sufficient detail to verify that the intervention was implemented as intended.

1 = The intervention and its implementation are described in adequate detail, including justification for significant variation during implementation.

2 = The intervention and its implementation are described in adequate detail, reflecting variation during implementation with little or no plausible impact on outcomes.

5. Nature of Comparison Condition

The quality of evidence for an intervention depends in part on the nature of the comparison condition(s).

0 = Research design either lacks a comparison condition, or employs a before/after comparison.

1 = Comparison condition was no service or wait-list (including baseline comparison for a multipoint time series), or an active intervention that has not been shown to be safer or more effective than no service.

2 = Comparison condition was an active intervention shown to be as safe as, or safer and more effective than, no service.

6. Standardized Data Collection

All outcome data should be collected in a standardized manner. Data collectors trained and monitored for adherence to standardized protocols provide the highest quality evidence of standardized data collection.

0 = Data collection or archival sources used by the evaluation to assess outcome did not use standardized data collection protocol(s).

1 = All outcome data were collected using standardized protocol(s).

2 = All outcome data were collected using standardized protocol(s) and trained data collectors.

7. Data Collector Bias

Data collector bias is most strongly controlled when data collectors are not aware of the interventions to which populations have been exposed. When data collectors are aware of specific interventions, their expectations should be controlled for through training and/or statistical analysis methods on resultant data.

0 = Data collectors were not masked to the population's condition, and nothing was done to control for possible bias, OR collector bias was identified and not controlled for statistically.

1 = Data collectors were not masked to the population's condition; possible bias was appropriately controlled for statistically or through training.

2 = Data collectors were masked to the population's condition, or only archival data was employed.

8. Population Studied

0 = A single group pre/posttest design was applied without a comparison group, OR the alleged comparison group is significantly different from the population receiving the intervention.

1 = Population(s) were studied using time trend analysis, multiple baseline design, or a regression-discontinuity design that uses within-group differences as a substitute for comparison groups.

2 = Population matching or similar techniques were used to compare outcomes of population that received the intervention with the outcomes of a valid comparison group.

9. Missing Data

Study results can be biased by missing data. Statistical methods as supported by theory and research can be employed to control for missing data that would bias results, but studies with no missing data needing adjustment provide the strongest evidence.

0 = Missing data were an issue and were taken into account inadequately, OR levels of missing data were too high to control for bias.

1 = Missing data were an issue and were taken into account, but high quality makes the control for bias suspect.

2 = Missing data were not an issue or were taken into account by methods that estimate missing data.

10. Analysis Meets Data Assumptions

The appropriateness of statistical analysis is a function of the properties of the data being analyzed and the degree to which data meet statistical assumptions.

0 = Analyses were clearly inappropriate to the data collected; severe violation(s) of assumptions make analysis uninterpretable.

1 = There were minor violations of assumptions, making interpretation of results arguable.

2 = There were no or only very minor violations of assumptions; result were generally interpretable.Start Printed Page 50389

11. Theory-Driven Selection of Analytic Methods

In addition to the properties of the data, analytic methods should be based on a logic model or theory underlying the intervention. Changes to analytic methods after initial data analysis (e.g., to dredge for significant results) decrease the confidence that can be placed in the findings.

0 = Analysis selected appears inconsistent with the intervention theory or hypotheses; insufficient rationale was provided by the investigator.

1 = Analysis selected appears inconsistent with the intervention logic model or hypotheses, but the investigator provides a potentially viable rationale.

2 = Analysis is widely accepted by the field as consistent with the intervention logic model or hypotheses.

12. Anomalous Findings

Findings that contradict the theories and hypotheses underlying an intervention suggest the possibility of confounding causal variables and limit the validity of study findings.

0 = There were anomalous findings suggesting alternate explanations for outcomes reported that were not acknowledged by the applicant.

1 = There were a few anomalous findings, but additional analysis or previous literature cited by the applicant provide a reasonable explanation.

2 = There were no anomalous findings, OR researchers explained anomalous findings in a way that preserves the validity of results reported.

Re-Review of Existing NREPP Programs

As noted above, SAMHSA believes it is important to ensure that both current and future NREPP interventions meet consistent scientific standards so that the public and other interested stakeholders can be confident in the effectiveness of these interventions. With this in mind, SAMHSA is committed to expeditiously re-reviewing all existing NREPP programs under the new process. As part of this effort, SAMHSA plans to provide—directly or indirectly—sufficient resources to each existing NREPP program to cover costs associated with a re-review. In addition, programs already received by NREPP and pending review will be reviewed under the new process. If additional support is needed by these pending programs regarding the new review processes, these resources will also be provided.

In order to accomplish these re-reviews efficiently and expeditiously, NREPP review coordinators will work with each program to obtain any additional documentation that might be needed for re-review. These review coordinators will then conduct a re-review of each program against the new review criteria. Programs with favorable re-reviews will be included in the new NREPP system when it is launched in 2006. Programs not receiving favorable re-reviews will have the opportunity to appeal the re-review decision, and will be eligible for re-review by independent, external reviewers. However, the schedule for re-reviews of appealed programs will be subject to SAMHSA Administrator and SAMHSA Center Director review priorities.

New Web Site

The primary goal of the revised NREPP Web site—http://www.nationalregistry.samhsa.gov—will be to provide the public with contemporary and reliable information about the scientific basis and practicality of interventions to prevent and treat mental and substance use disorders. All interventions achieving NREPP status will be listed on the Web site. Average ratings and evaluate scores from scientific peer reviewers, as well as information on the utility and transferability of these interventions, will be posted on the site.

In addition, a searchable outcomes database of evidence-based interventions will be a key feature. The Web site will also contain a variety of learning and self-assessment tools for prospective and current NREPP interventions to continuously improve their scientific evidence base. Features of the new Web site will include:

  • Evidence rating criteria and utility descriptors
  • Detailed review guidelines
  • Self-assessment tool to assist interventions in determining if they are ready to submit an application to NREPP
  • Detailed information on how to apply
  • Links to technical assistance resources available to potential applicants
  • Relevant resources, including publications, presentations, links, and other supplemental materials
  • Frequently Asked Questions (FAQs)
  • Information on how to contact a representative from the NREPP team
  • Glossary of terms

Support for Innovative Interventions

SAMHSA recognizes that the long term utility and value of NREPP rests, in part, on the ability of SAMHSA and others to support efforts to evaluate and document the evidence-base of innovative interventions in ways that will maximize their opportunity for entry into NREPP. SAMHSA is considering potential options for both the direct and indirect provision of such support, and will seek to clarify its intentions in this area sometime in Fiscal Year 2006.

Questions To Consider in Making Your Comments

Responders should feel free to comment on any, or all, questions, as well as provide relevant suggestions not included in the specific questions. In order to facilitate the compilation and analysis of comments, responders are asked to be explicit about the questions to which they are responding.

1. SAMHSA is seeking to establish an objective, transparent, efficient, and scientifically defensible process for identifying effective, evidence-based interventions to prevent and/or treat mental and substance use disorders. Is the proposed NREPP system—including the suggested provisions for screening and triage of applications, as well as potential appeals by applicants—likely to accomplish these goals?

2. SAMHSA's NREPP priorities are reflected in the agency's matrix of program priority areas. How might SAMHSA engage interested stakeholders on a periodic basis in helping the agency determine intervention priority areas for review by NREPP?

3. There has been considerable discussion in the scientific literature on how to use statistical significance and various measures of effect size in assessing the effectiveness of interventions based upon both single and multiple studies (Schmidt & Hunter, 1995; Rosenthal, 1996; Mason, Schott, Chapman, & Tu, 2000: Rutledge & Loh, 2004). How should SAMHSA use statistical significance and measures of effect size in NREPP? Note that SAMHSA would appreciate receiving citations for published materials elaborating upon responders suggestions in this area.

4. SAMHSA's proposal for NREPP would recognize as effective several categories of interventions, ranging from those with high-quality evidence and more replication to those with lower quality evidence and fewer replications. This would allow for the recognition of emerging as well as fully evidence-based interventions. Some view this as a desirable feature that reflects the continuous nature of evidence; provides important options for interventions Start Printed Page 50390recipients, providers, and funders when no or few fully evidence-based interventions are available; and helps promote continued innovation in the development of evidence-based interventions. Others have argued that several distinct categories will confuse NREPP users. Please comment on SAMHSA's proposal in this area.

5. SAMHSA recognizes the importance of considering the extent to which interventions have been tested with diverse populations and in diverse settings. Therefore, the agency anticipates incorporating this information into the web site descriptions of interventions listed on NREPP. This may allow NREPP users to learn if interventions are applicable to their specific needs and situations, and may also help to identify areas where additional studies are needed to address the effectiveness of interventions with diverse populations and in diverse locations.

SAMHSA is aware that more evidence is needed on these topics. Please comment on SAMHSA's approach in this area.

6. To promote consistent, reliable, and transparent standards to the public, SAMHSA proposes that all existing programs on NREPP meet the prevailing scientific criteria described in this proposal, and that this be accomplished through required re-reviews of all programs currently on NREPP. SAMHSA has considered an alternative approach that would “grandfather” all existing NREPP programs under the new system, but would provide clear communication that these existing programs have not been assessed against the new NREPP scientific standards. Please comment on which approach you believe to be in the best interests of SAMHSA stakeholders.

7. What types of guidance, resources, and/or specific technical assistance activities are needed to promote greater adoption of NREPP interventions, and what direct and indirect methods should SAMHSA consider in advancing this goal?

8. SAMHSA is committed to consumer, family, and other nonscientist involvement in the NREPP process. The panels convened by SAMHSA and described earlier in this notice suggested that these stakeholders be included specifically to address issues of intervention utility and practicality. Please comment on how consumer, family, and other nonscientist stakeholders could be involved in NREPP.

9. SAMHSA has identified NREPP as one source of evidence-based interventions for selection by potential agency grantees in meeting the requirements related to some of SAMHSA's discretionary grants. What guidance, if any, should SAMHSA provide related to NREPP as a source of evidence-based interventions for use under the agency's substance abuse and mental health block grants?

10. SAMHSA believes that NREPP should serve as an important, but not exclusive source, of evidence-based interventions to prevent and/or treat mental and substance use disorders. What steps should SAMHSA take to promote consideration of other sources (e.g., clinical expertise, consumer or recipient values) in stakeholders' decisions regarding the selection, delivery and financing of mental health and substance abuse prevention and treatment services?

11. SAMHSA anticipates that once NREPP is in operation, various stakeholders will make suggestions for improving the system. To consider this input in a respectful, deliberate, and orderly manner, SAMHSA anticipates annually reviewing these suggestions. These reviews would be conducted by a group of scientist and nonscientist stakeholders knowledgeable about evidence in behavioral health and the social sciences. Please comment on SAMHSA's proposal in this area.

References

Mason, CA, Scott, KG, Chapman, DA, Tu, S. A review of some individual and community level effect size indices for the study of risk factors for child and adolescent development. Ed. Psychol. Measurement. 2000; 60(3): 385-410.

Rosenthal, JA. Qualitative descriptors of strength of association and effect size. J. of Soc. Serv. Research. 1996; 21(4): 37-59.

Rutledge, T, Loh, C. Effect size and statistical testing in the determination of clinical significance in behavioral medicine research. Annals of Beh. Med. 2004; 27(2): 138-145.

Schmidt, F, Hunter, JE. The impact of data-analysis methods on cumulative research knowledge: statistical significance testing, confidence intervals, and meta-analysis. Eval. Health Prof. 1995 Dec; 18(4): 408-27.

End Supplemental Information

[FR Doc. 05-17034 Filed 8-25-05; 8:45 am]

BILLING CODE 4160-01-M