Community HealthChoices Evaluation Plan

University of Pittsburgh Health Policy Institute


The Department of Human Services (Department) received over 200 public comments from stakeholders including consumers, the aging and disability network, advocates, providers and managed care organizations on the plan. An internal evaluation work group reviewed all the comments and worked closely with the University to incorporate recommendations to the Plan. A summary of key themes follows, and a link to the updated Plan is available below.

The Department appreciates the time taken by stakeholders to provide this valuable feedback.

Comment ThemeResponse
1. Several comments pointed out the methodological difficulties of clearly attributing results to the implementation of the CHC. Specifically, commenters noted that variation exists across zones, which raises complications for defining appropriate comparison groups.Best practice research techniques will be used to address these challenges, including “difference-in-differences” analysis, and selection of matched groups across zones to ensure that fair comparisons are being made. These are methods that adjust for differences in people’s needs, service availability and other local factors.
2. Several comments suggested additions to the Plan’s lists of potential Key Informants and interview subjects.The list of key informants will be adjusted to ensure that consumers, caregivers, LTSS providers, managed care organizations, county agencies and advocacy groups are included as key informants
3. Several comments suggested that the over 60 age group should be divided into two groups, the second being over 80 years of age.The Department and University agree that analysis of older people in CHC should be segmented in this way and are studying the feasibility of doing so. Multiple age break-outs is feasible in administrative data analysis (analyzing claims for all participants, for example), but sample sizes may be too small to do this segmentation of the participant surveys.
4. The suggestion was made to conduct a survey of LTSS providers regarding their experiences with CHC.An annual survey of LTSS providers has been added to the Evaluation Plan.
5. It was suggested that the evaluation capture changes to the eligibility and enrollment process from the perspective of new consumers.This is an important aspect of the implementation. The experience of consumers will be addressed through qualitative interviews with newly enrolled consumers during each phase of the implementation, starting in July 2017. However, the University will not conduct interviews with clients newly enrolled under the current fee-for-service system, since there are many confounding changes underway that would interfere with assessing the impact of CHC per se.
6. A number of comments indicated an interest in having evaluation results provided on a timely basis during the implementation phase.This evaluation will provide annual interim reports and, during the first two phases, implementation studies. DHS intends to publish interim and implementation reports on its website and present results in public meetings.
7. Several comments expressed an interest in having results broken out by MCO.The evaluation will analyze results by MCO, but MCOs will not be named in the analysis. Their identity will be “blinded” for public reporting (e.g., they will be listed as MCO 1, MCO 2, etc.) If there are problems with a given MCO, it will be the responsibility of DHS to identify and address the issues through its ongoing monitoring process.
8. There was a suggestion to capture details of participants’ physical environments.The University will include an objective assessment of the living environment as part of the in-person interview component of the study. In addition, a brief subjective assessment of neighborhood quality is being considered. The University will also be able to link geographic data at the street or neighborhood (zip code) level to measure social indicators that may affect quality of life and access to health care services.
9. Some commenters expressed concern that surveys, interviews and other evaluation activities are planned only in English and Spanish, and should include several other prevalent languages, including American Sign Language.The University is committed to effective participation of all persons who are selected for interviews and surveys and is exploring the feasibility of translation services for languages beyond Spanish.
10. A number of comments suggested specific evaluation measures and tools pertaining to administrative data and interview data.We appreciate the suggestions. The University is studying a wide variety of existing measures and tools and will give careful consideration to the suggestions made.
11. Some commenters requested that the public be given access to the raw data from the evaluation.To ensure privacy of all subjects, DHS will not make raw data available to the public. The data collected and analyzed by the University is subject to strict protections in data use agreements with DHS and CMS.

The revised Evaluation Plan is available by clicking here, and will be updated as needed during the course of the project.

The University will conduct a comprehensive multi-year evaluation of Pennsylvania’s new Medicaid Long-Term Services and Supports (LTSS) program, Community HealthChoices (CHC). The evaluation will provide an independent assessment of the implementation and outcomes of the program to complement other oversight and quality assurance activities conducted by the Department of Human Services (DHS), Office of Long-Term Living. The evaluation will address the following broad research questions:

  1. Does CHC result in greater access to home and community-based LTSS and shift the balance of care away       from institutionalized settings for people who prefer to live in the community?
  2. Does CHC improve coordination of LTSS, physical health care, and behavioral health care?
  3. Does CHC improve the quality of care and quality of life of participants and family caregivers?
  4. Does CHC lead to innovation in the delivery of physical health care and LTSS?
  5. Does CHC reduce unnecessary utilization of services and reduce the growth in aggregate costs?

The evaluation will analyze administrative data, enrollment, utilization and cost data from Pennsylvania Medicaid and from Medicare. The evaluation will also incorporate data from a variety of sources, including the CHC-MCOs, nursing facilities, person-centered service plans, and level of care assessments.

The complete Evaluation Plan provides a narrative overview of how the University of Pittsburgh proposes to structure and implement the CHC program evaluation. The narrative describes the timeline of the evaluation, primary research questions, data collection methods and analytic strategies. The Evaluation Plan is available by clicking here, and will be updated as needed during the course of the project.