Grading the States 2006: A Brief Overview of Methodology

Grading the States is a "report card" that assesses each state's mental healthcare system, measured relative to three landmark documents:

• U.S. Surgeon General. (1999). Report on Mental Health.  
• President's New Freedom Commission on Mental Health. (2003). Achieving the Promise: Transforming Mental Healthcare in America.
• Institute of Medicine of the National Academy of Sciences. (2005). Improving the Quality of Health Care for Mental and Substance Abuse Conditions.

The report also is based consciously on the perspective of people with serious mental illness and their families. We are the customers whom the state systems are designed to serve.

Disseminated through the federal government's "Science to Service Initiative," the five evidence-based practices promoted by the Center for Mental Health Services (CMHS) have been used as a standard throughout much of the assessment, along with other recovery-oriented service and treatment measures. Each state's progress toward a proven, cost-effective system is indicated by a letter grade of A, B, C, D, or F, based on information from four sources, and scored from 39 specific criteria representing four categories: Infrastructure, Information Access, Services, and Recovery Supports.

Sources

1. State Mental Health Authority (SMHA) Self-Reported Questionnaire

SMHAs were surveyed through a questionnaire submitted and returned during October-December 2005. Colorado and New York declined to respond. Although narrative discussions are included for these two states, they have been graded simply as "U" for "unresponsive."

2. Public Information

Information also was obtained from public sources, such as official documents, including SMHA Community Mental Health Services Block Grant Applications to SAMHSA, state agency reports, reports from the Department of Justice, newspaper articles, and other sources.

3. Consumer and Family Test Drive

Access to services depends on access to information. In order to capture the perspectives of consumers and family members about access to basic information, a Consumer and Family Test Drive (CFTD) was developed. For details regarding the methodology for this score source, see the appendix of this report. In addition to serving as a source of information, the CFTD results were included as part of the criteria and were weighted 10 points (10 percent) in the numerical scoring, in recognition of the important role that access to information plays in the mental health system.

4. Interviews

Consumer and family advocates were interviewed on several occasions concerning their state systems. Other knowledgeable sources, such as legal and mental health policy experts, also were consulted.

Criteria and Scoring

Four individuals were selected to serve as masked scorers based on their diverse professional and personal backgrounds:

• Anand Pandya, M.D., chair, NAMI National Board of Directors Policy Committee
• Marty Raaymakers, chair, NAMI National Consumer Council
• Elizabeth Edgar, NAMI senior policy analyst, family member, social worker
• Jack Gorman, M.D., chair, NAMI Scientific Advisory Council

The 39 individual criteria used in the scoring process were weighted by relative importance. Where a perfect score is 100, the distribution is as follows:

Infrastructure:        18

Info Access:            16

Services:                 44

Support/Recovery:  22

Each question received a 2, 3, or 4 point valuation. For details regarding methodology of the values and scoring, please refer to the appendix of this report.

Criteria were evaluated through a combination of masked and unmasked scoring (where "masked" indicates that the identity of the respondent was unknown, and "unmasked" indicates that the identity of the respondent was known), as follows:

Masked scoring          71%

Unmasked scoring      29%

Based on the information gathered and the masked and unmasked scoring, grades were determined for the four categories, with the final grade representing an overall assessment of the state system. We graded the states on a scale of 0 to 100.  More information about our grading scale may be found on NAMI's Web site.

back to Grading the States methodology overview

Back