Article

A combination of studies and medical research data is examine to review the positive outcomes remote patient monitoring has on clinical chronic conditions such as cardiovascular, respiratory disease and behavioral health management.

June 1, 2016 Prepared by Pacific Northwest Evidence-based Practice Center Portland, OR; AHRQ Publication No. 16-EHC034-EF

Preface

The Agency for Healthcare Research and Quality (AHRQ), through its Evidence-based Practice Centers (EPCs), sponsors the development of evidence reports and technology assessments to assist public- and private-sector organizations in their efforts to improve the quality of health care in the United States. The reports and assessments provide organizations with comprehensive, science-based information on common, costly medical conditions and new health care technologies and strategies. The EPCs systematically review the relevant scientific literature on topics assigned to them by AHRQ and conduct additional analyses when appropriate prior to developing their reports and assessments.

The goals of this Technical Brief are to provide an objective description of the state of the science in an evidence map of systematic reviews, that identifies key areas important for practice and policy decisionmaking in relation to the available evidence, creates a potential framework for assessing the applications and implications of telehealth interventions, generates a summary of ongoing research, and provides information on what future research is needed. In particular, through the Technical Brief, AHRQ hopes to gain insight on the appropriate conceptual framework and critical issues that will inform future research.

AHRQ expects that the EPC evidence reports and technology assessments will inform individual health plans, providers, and purchasers as well as the health care system as a whole by providing important information to help improve health care quality.

If you have comments on this Technical Brief, they may be sent by mail to the Task Order Officer named below at: Agency for Healthcare Research and Quality, 5600 Fishers Lane, Rockville, MD 20857, or by email to epc@ahrq.hhs.gov.

Andrew Bindman, M.D. Director: Agency for Healthcare Research and Quality

Arlene S. Bierman, M.D., M.S.   Director : Center for Evidence and Practice Improvement; Agency for Healthcare Research and Quality

Stephanie Chang, M.D., M.P.H, Director, EPC Program Center for Evidence and Practice Improvement; Agency for Healthcare Research and Quality

Elise Berliner, Ph.D., Task Order Officer Center for Evidence and Practice Improvement; Agency for Healthcare Research and Quality

Acknowledgments

The authors gratefully acknowledge the following individuals for their contributions to this project: Andrew Hamilton, M.S., for literature searches; Leah Williams, B.S., for editorial support; Christine Morelli, student worker; and Elaine Graham, M.L.S., for her support and guidance in developing this report.

Key Informants

In designing the study questions, the EPC consulted several Key Informants with diverse experiences and perspectives in implementing and evaluating telehealth. The EPC sought the Key Informant input to identify areas where telehealth is being used but has not been studied, issues raised by telehealth, the important questions about telehealth that different stakeholders want research evidence to answer, and to help identify the next steps in research that would be most likely to move the field forward. Key Informants were not involved in the analysis of the evidence or the writing of the report. Therefore, study questions, design, methodological approaches, and/or conclusions do not necessarily represent the views of individual Key Informants.

Key Informants must disclose any financial conflicts of interest greater than $10,000 and any other relevant business or professional conflicts of interest. Because of their role as end-users, individuals with potential conflicts may be retained. The TOO and the EPC work to balance, manage, or mitigate any conflicts of interest.

The list of Key Informants who provided input to this report follows:

Rashid L. Bashshur, Ph.D.* University of Michigan Ann Arbor, MI

Johnathan Linkous, M.P.A. American Telemedicine Association Washington, DC

Alan Morgan, M.P.A.

National Rural Health Association Leawood, KS

Jamie Pina, Ph.D., M.S.P.H *

Director, Public Health Informatics Program RTI International

Research Triangle Park, NC

Karen Rheuban, M.D.*

University of Virginia Center for Telehealth University of Virginia Telehealth System Charlottesville, VA

Mark R. Thomas, P.R., M.P.H.

Office for the Advancement of Telehealth

  • Department of Health and Human Services

Washington, DC

*Thank you to these Key Informants for their review of the draft technical brief.

Peer Reviewers

Prior to publication of the final evidence report, EPCs sought input from independent Peer Reviewers without financial conflicts of interest. However, the conclusions and synthesis of the scientific literature presented in this report does not necessarily represent the views of individual reviewers.

Peer Reviewers must disclose any financial conflicts of interest greater than $10,000 and any other relevant business or professional conflicts of interest. Because of their unique clinical or content expertise, individuals with potential nonfinancial conflicts may be retained. The TOO and the EPC work to balance, manage, or mitigate any potential nonfinancial conflicts of interest identified.

The list of Peer Reviewers follows:

Annette Brown, Ph.D.

Deputy Director, International Initiative for Impact Evaluation

Washington, DC

Ann McKibbon, Ph.D.

Department of Clinical Epidemiology & Biostatistics, McMaster University Hamilton, ON, Canada

Telehealth: Mapping the Evidence for Patient Outcomes from Systematic Reviews

Structured Abstract

Background. Telehealth includes a wide range of technologies used to fulfill many functions in in health care for patients with a variety of clinical conditions. For this evidence map, telehealth is defined as the use of information and telecommunications technology in health care delivery for a specific patient involving a provider across distance or time. Various types of telehealth interventions have been evaluated in thousands of research studies and hundreds of systematic reviews. The vast size of the literature and the variations in how the literature has been collected, evaluated, and synthesized make it challenging to determine what is known about the effectiveness of telehealth for specific purposes and what questions remain unanswered.

Purpose. The purpose of this brief is to provide an overview of the large and disparate body of evidence about telehealth for use by decision makers. The approach used was to create an evidence map of systematic reviews published to date that assess the impact of telehealth on clinical outcomes. This evidence map describes a limited number of key characteristics of the systematic reviews currently available in order to evaluate the bodies of evidence available to inform practice, policy, and research decisions about telehealth.

Methods. An evidence map is a specific type of rapid or abbreviated review. While the creation of the evidence map is based on systematic review methodology, its goal is to describe rather than synthesize available research and to use graphics when possible to represent selected characteristics of the evidence. We included systematic reviews that synthesized the impact of telehealth interventions on clinical outcomes, utilization, or cost. We created bubble plots to separately examine the distribution of the evidence from systematic reviews in terms of volume (number of reviews, number of patients in the included studies), conclusions about benefit by clinical focus area, and telehealth function. We also determined how much evidence is available about combinations of clinical areas and telehealth functions reported in existing systematic reviews. We supplemented this by summarizing the topics covered in excluded reviews and the results of exploratory searches for primary studies on selected topics in order to assess the need for future systematic reviews or primary studies in key telehealth domains.

Findings. We identified 1,494 citations about telehealth, from which 58 systematic reviews met our inclusion criteria. A large volume of research reported that telehealth interventions produce positive outcomes when used for remote patient monitoring, broadly defined, for several chronic conditions and for psychotherapy as part of behavioral health. The most consistent benefit has been reported when telehealth is used for communication and counseling or remote monitoring in chronic conditions such as cardiovascular and respiratory disease, with improvements in outcomes such as mortality, quality of life, and reductions in hospital admissions. Given sufficient evidence of effectiveness for these topics, the focus of future research should shift to implementation and practice-based research. Topics with an evidence base that could be the focus of future systematic reviews include telehealth for consultation, uses in intensive care units, and applications in maternal and child health. We also identified topics with a limited

evidence base such as telehealth for triage in urgent/primary care, management of serious pediatric conditions, patient outcomes for teledermatology, and the integration of behavioral and physical health that may be best addressed by additional primary research. Finally, telehealth research should be integrated into evaluation of new models of care and payment so that the potential of telehealth can be assessed across the continuum of care in organizations that are implementing these reforms.

Box 1 below summarizes the key messages of this report.

Box 1. Key messages

Box 1

 

 

 

 

 

 

 

 

 

 

 

Contents

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figures

 

 

 

 

 

 

 

 

 

Appendixes

 

 

 

 

 

 

 

 

 

 

 

 

 

Introduction

The purpose of this technical brief is to identify and describe the body of research evidence currently available in the form of systematic reviews to inform decisions related to contemporary practice and policy issues about telehealth. Beyond describing what is available, the brief also aims to identify key areas in which systematic reviews are insufficient for these purposes and suggest what future research (systematic reviews or primary studies) is needed.

The existence of research studies about a topic does not guarantee that the evidence is in a form that can be used to support practice and policy decisions. Ideally for research to support decisionmaking, studies need to be identified, evaluated, and synthesized into a body of evidence. Furthermore, each of these steps needs to be planned, operationalized, executed, and presented so that the evidence addresses the questions relevant to the important decisions.

Decisionmakers and other stakeholders may be able to do this ad hoc if the volume of literature is small and the issues are straightforward. However, if the topic is broad and the body of literature is expansive, the task quickly becomes daunting.

This is precisely the case with telehealth. Telehealth includes several different technologies that are not treatments or inventions in and of themselves, rather the technologies are used to expand access, exchange information, and deliver care in alternate formats. Technologies such as remote patient monitoring and videoconferencing can be used to expand specialty care to seriously ill patients in intensive care units (ICUs), to patients in critical assess hospitals, or to patients and providers in areas with shortages of health care providers. Similarly, technology can be used to extend primary care to remote areas and increase the frequency of patient and primary care provider interactions. Internet applications can be used to facilitate psychiatric treatment and other counseling. Devices can be used to evaluate status in patients with chronic conditions who need close monitoring.

Many combinations of technologies, functions, and conditions have been studied to date. The National Library of Medicine added the Medical Subject Headings (MeSH) term “Telemedicine,” as a synonym for Telehealth to its list of indexing terms in 1993. As of January 2016 there were over 15,000 articles and over 400 systematic reviews that have been assigned Telemedicine as a major subject heading, indicating that telehealth is the focus of the publication.

Our task was to characterize the systematic reviews available about the effectiveness of telehealth. In this brief we present the results in the form of an evidence map. An evidence map is a combination of a systematic approach to identifying the existing literature on a topic and a description of key characteristics of the existing evidence. This description includes graphic presentation of these key characteristics. It is called a “map” because of the use of graphics and because, like a map, it is a representation that emphasizes and presents some, but not all features, just as we use topographic, economic, road, and climate maps of the same area for different purposes. An evidence map should help clarify the current state of research and possible future directions. Evidence mapping is “emerging as a less exhaustive yet systematic and replicable methodology that allows an understanding of the extent and distribution of evidence in a broad clinical area, highlighting both what is known and where gaps in evidence exist.”1 As a form of rapid review, evidence maps have been used by several organizations and are likely to become more common as the evidence base across health topics grows in size and complexity.2-11

Background

The State of Telehealth

Telehealth encompasses multiple technologies that have been applied to health services for a wide range of conditions, populations, and settings. In fact, telehealth is one of the oldest uses of technology in health care. Telehealth interventions in place today interact with many different specialties across the continuum of care, and affect patients of all ages.12 Additionally, telehealth mirrors the rapidly changing technology environment, and the corresponding evidence base is expanding in both volume and scope. Many different definitions of telehealth are used in the scientific literature, among policy leaders, and by industry and other stakeholders. The Health Resources and Services Administration (HRSA) provides the following well-accepted definition of telehealth: “the use of telecommunications and information technologies to share information, and provide clinical care, education, public health, and administrative services at a distance.”13 There are also several related terms such as telemedicine, eHealth, and mHealth, which have been defined by the Office of the National Coordinator for Health Information Technology (ONC).14

The relationships among the many varied terms related to telehealth has been illustrated by van Dyk (Figure 1).15 The different forms of telehealth can be used in a variety of clinical areas and the technologies evaluated in the literature range from videoconferencing, image exchange, and streaming media to wireless communications and monitoring.14 These telecommunications technologies can provide long-distance health care, educate patients and providers, and support management of chronic conditions in patients’ homes. The wide-ranging capabilities and applications also create one of the major challenges when systematically reviewing the literature on telehealth¾the heterogeneity among existing studies. Studies of telehealth vary by setting (e.g., rural or urban; home, community, clinic, nursing home, or hospital; radiology department; pharmacy)16 by clinical indication, by health care delivery function, by type of technology, and by expected impact.

Figure 1

 

 

 

 

 

 

 

 

 

 

 

 

Current Practice and Policy Issues

The motivation for this technical brief originates from a request by United States Senators Bill Nelson and John Thune for a literature review on the value of telehealth and remote patient monitoring, particularly for the chronically ill, with a focus on expanding access to care and reducing costs.17 A multi-stakeholder letter to Senators Bill Nelson and Susan Collins from several medical, patient advocacy, and industry groups supported the request for such a review.18 Initial searches in response to this request confirmed that there is a large volume of literature consisting of both primary studies and systematic reviews about applications of telehealth. This literature covers a broad range of topics and is of varying quality. Given both the volume and variability of the literature, it was not feasible to provide a full, comprehensive report on the evidence for effectiveness of all aspects of telehealth in a single technical brief. As such, this evidence map is the first step toward identifying domains or topics where systematic reviews have already synthesized evidence of effectiveness and topics which remain to be synthesized as well as areas where there is little primary research. As an evidence map, this technical brief was not designed to be a comprehensive review of primary research.

The request for a systematic review of “the growing body of evidence demonstrating the value of telehealth technologies”17 is rooted in a belief that telehealth has the potential to produce positive benefits, a desire to promote the effective use of telehealth, and motivation to remove barriers to its use. Telehealth has been described as having great promise in the sense that it could leverage the $30 billion investment in electronic health records that ONC has made in the last half-decade through the Health Information Technology for Economic and Clinical Health (HITECH) Act.19 While the potential benefits and possible uses have been extensively enumerated and described (e.g., improving quality, promoting safety, and expanding access),20-22 there is also a body of literature that outlines barriers and challenges to implementation and widespread adoption of telehealth.23-25 The goal of this evidence map is to increase our understanding of what uses of telehealth are supported by existing bodies of evidence in the form of systematic reviews so that resources can be used judiciously to support both systematic reviews and primary studies in areas where either research has not been conducted or the evidence is not in a usable form.

In order to inform the methodology and the structure for this evidence map we started by identifying the key issues that stakeholders hoped the research evidence would help address. We based our assessment on the letter mentioned above, our discussions with Key Informants (see Methods below for details), background materials such as reports and testimony, and our

expertise derived from both an earlier review and our team’s collective experience in this field.21,22,26,27

Key issues involve identifying situations where telehealth use is supported by the evidence, obtaining sustainable funding for its use, and encouraging health care providers to apply it.

Although telehealth has great potential to improve health care delivery,16,28,29 challenges include problems in reimbursement, scalability, and licensure.23,24

A technical brief differs from a systematic review both in terms of scope and methodology.

Because the letter from the Senators emphasized a focus on the use of telehealth within the health care system, we narrowed the scope of our analysis to interventions that included some aspect of a patient interacting with the health care system or a health care providers interacting about a specific patient for the purposes of treatment, management, or prevention of disease.

These interactions could occur over distance or time, which is in real time or asynchronous and in different or the same locations. This excluded applications such as informational Web sites, mobile applications that did not facilitate interaction, and any purely educational activities. Also, because issues related to implementation are addressed after effectiveness is established, we focused on describing the available evidence related to effectiveness and did not include evaluations of telehealth implementation or spread. How these decisions were operationalized is described in the next sections on objectives and methods.

Objectives and Guiding Questions

The purpose of this technical brief is to provide a survey of the large amount of currently available research about the impact of telehealth on health outcomes and health care utilization that can be used to inform policy and practice decisions and guide future research. This differs from a common use of technical briefs to explore topics with scant evidence. This technical brief uses an evidence map format as a means of both presenting and analyzing the information. The map first focuses on describing the currently available systematic reviews that could potentially be used to guide decisions. This approach acknowledges that evidence-based decisions should be guided by a body of literature, and not usually by an individual study. The map format also provides an opportunity for two additional activities: 1) to identify areas not addressed or inadequately addressed in these reviews, for which primary literature may be robust enough for further systematic reviews, and 2) to allow enumeration of areas with gaps in evidence that will require additional primary research.

The questions below guided our work mapping the available research on telehealth interventions.

  1. What is the current research on the effectiveness of telehealth interventions?
    1. What telehealth interventions have been studied for effectiveness or harms?
      1. For which interventions are there systematic reviews available?
    2. What patient populations and conditions have been studied with telehealth interventions?
    3. What settings and situations have been studied with telehealth interventions?
    4. What primary outcomes have been studied with telehealth interventions?
    5. What study designs have been used in studies of the effectiveness of telehealth interventions?
  2. What gaps exist in the current research?
    1. Which telehealth interventions identified by experts as currently relevant have no research evidence, or inadequate evidence?
    2. For which telehealth interventions are additional primary research studies needed to answer questions important to policy and practice, g., additional patient populations or outcome measures?
    3. For which telehealth interventions are there sufficient primary research studies that a new systematic review would add to current knowledge?

The organizing principal for this specific evidence map, given the goals of the stakeholders, is that the evidence must be structured in terms of both format and content so that it can be used to inform current policy and practice decisions. For this reason, we include general consideration of the quality and the findings of systematic reviews in our map. Quality and results are not addressed as comprehensively as they would be in a full systematic review, and it should be noted that these topics are often not included in technical briefs. We used and adapted approaches that have been used in other literature maps to include selected elements of quality and results.5-8 We describe these approaches in the methods section of this report.

Methods

An evidence map combines a systematic approach to identifying the existing literature on a topic with a description of key characteristics of the evidence identified. While evidence maps use some elements of systematic review methodology, they are a type of rapid review, and by definition are not comprehensive and do not purport to provide a definitive synthesis of results across a body of evidence. Methodology and guidance for the creation of literature maps exist;30- 32 however, there are currently no accepted standards for this type of abbreviated, descriptive review and a systematic review of evidence mapping confirmed that the exact content and approach vary based on the goals of the project.33

In order to achieve the objectives listed above and create an evidence map of systematic reviews about telehealth, we developed a protocol based on adaptations of widely accepted systematic review methods34-36 and after consulting with Key Informants (KIs) and the funding agency, AHRQ. The KIs and AHRQ provided valuable perspectives, but they are not responsible for the resulting protocol or report. The protocol was posted on the AHRQ Web site on August 11, 2015 (http://www.effectivehealthcare.ahrq.gov/search-for-guides-reviews-and- reports/?pageaction=displayproduct&productid=2110).

Discussions with Key Informants

A group of six KIs representing diverse perspectives, including policy, research, telehealth use, and practice, supplemented with representation from U.S. Senate staff, participated in interviews during the initial phase of the project. KIs are recruited to provide insight and offer opinions, but they are not authors of the report, they are not responsible for the content, and the report does not necessarily reflect their views. In addition to the initial consultation, KIs are offered the opportunity to comment on the draft report as part of the peer review process. The purpose of KI interviews varies depending on the project requirements. In technical briefs about topics with limited research evidence they may offer expert opinion. It this case the KI role was to identify the current major practice and policy issues surrounding telehealth. This information informed our searches and development of the review inclusion and exclusion criteria, and also guided the collection and presentation of descriptive information in the evidence map.

Recognizing these issues was crucial to understanding how research evidence about telehealth could best be organized in order to support stakeholder decisionmaking.

During the interviews, KIs raised the following major points:

KIs discussed the state of the field and the fact that the scope of a technical brief may not be sufficient to address all stakeholder needs.

KIs were concerned that there is already a vast body of literature available, but much of it may be of low quality and therefore not useful for decisionmakers. At the same time, they noted that, in their opinion, there are a number of well-executed studies. They stressed that combining results across studies without considering the quality of the research of the individual studies may be the reason many systematic reviews fail to come to clear conclusions about the effectiveness of telehealth.

KIs emphasized that enough detail about how telehealth is used, including for which type of patients and what situations telehealth is studied, needs to be included in the research evidence in order for it to inform decisions about payment, licensing, credentialing, and investment.

KIs underscored the need for cost effectiveness and other health care utilization outcomes in addition to clinical effectiveness outcomes. They emphasized the need for data on effectiveness

and cost as higher priority than data on more process-related outcomes such as acceptability of telehealth by providers, patient satisfaction, or implementation facilitators and barriers.

Search Strategies

We searched Ovid MEDLINE®, the Cochrane Database of Systematic Reviews, and PROSPERO, looking for completed systematic reviews and for systematic reviews in process. We also identified reviews of reviews (sometimes referred to as umbrella reviews) and checked their reference lists against our list of identified reviews. Searches were limited to systematic reviews published in or after 2006 through the end of January 2016 with search date ranges ending in 2005 or later. This date (2006) was selected because it was deemed early enough to capture all relevant published systematic reviews and primary studies of current telehealth approaches and technologies, and it coincided with the publication date of a previous systematic review of telemedicine that our Evidence-based Practice Center performed.26 These dates were discussed with the KIs, who agreed that older evidence would have limited relevance for pending decisions.

After screening, assessing, and categorizing the reviews, we conducted additional searches in Ovid MEDLINE for primary studies using the same relevant inclusion criteria on topics not covered by included systematic reviews. Similarly, to identify grey literature reports that were essentially systematic reviews, we searched the New York Academy of Medicine Grey Literature database. We also searched the Websites of telehealth-related organizations and U.S. government agencies with involvement in telehealth. When reviewing reports by U.S. government agencies and telehealth organizations we searched for both reports that were similar in methodology and purpose to published systematic reviews as well as for products with objectives similar to this brief (i.e., describing the state of the evidence available to support decisions about telehealth). Our search strategies including a list of Web sites searched are included in Appendix A.

Study Selection

We developed criteria for inclusion and exclusion of studies (first systematic reviews and then primary studies) based on the Guiding Questions and consideration of the current issues and questions raised by stakeholders. The ability of the research evidence to inform decisionmaking guided study eligibility criteria, influenced what information we collected, and shaped how we presented our findings. We adapted a standard framework used in systematic reviews referred to as PICOTS (population, intervention, comparators, outcomes, timing, and setting) to outline our eligibility criteria. Inclusion and exclusion criteria are detailed in Appendix B and are described in this section. A list of the included systematic reviews can be found in Appendix C; excluded reviews are listed in Appendix D.

Definition of telehealth interventions for this brief. As described in the introduction, telehealth can refer to the use of several different technologies for many purposes related to health care. In order to define a scope that corresponded to pressing policy questions and to be sure that we were summarizing evidence on comparable interventions (i.e., not comparing apples and oranges), we established a core definition for this brief that includes the use of interactive technology to provide health care for specific patients. For our purposes, for an intervention to be considered telehealth it had to include the use of technology to facilitate an interaction between a patient and the health care system or interaction between two or more providers when the interaction was directly related to an individual patient’s care. The interaction could occur over

distance and/or over time (asynchronous as opposed to in real time). Using this definition, telehealth includes using video or mobile devices to offer counseling (over distance and in real time), remote patient monitoring which uses a variety of technology to transmit patient physiologic data to providers who monitor the patient condition and adjust treatment when needed (over distance and asynchronous), or using technology to obtain a consultation from another provider (either in real time or asynchronous). Following a precedent set in previous studies, telephone-only voice conversations were not considered telehealth. E-mail and Short Message Service (SMS) text were considered to be telehealth if they were interactive and replaced an in-person interaction (i.e., automated text messaging was not included), but they were not included if they were only in one direction (e.g., notifications) or if they were not personalized (e.g., generic messages sent to a group of patients). This definition was applied to both systematic reviews and our later consideration of individual studies.

Study design and quality. Our core search was for systematic reviews that focused on telehealth and synthesized clinical or utilization/cost outcomes across primary research studies. The rationale for basing the map on systematic reviews is that systematic reviews are the pinnacle of the evidence hierarchy for informing decisions. In most cases changes in policy and practice are not based on a single study; rather they are based on a body of evidence consisting of several studies. Systematic reviews are by definition a means of assembling a body of evidence and making it more accessible to users than the individual studies on their own. A systematic review should identify, evaluate, and synthesize evidence, including drawing conclusions across studies about the effectiveness of interventions or explaining why such a conclusion could not be made. This corresponds to the main objective of the report, which is to identify telehealth topics for which sufficient evidence exists and topics for which either additional systematic reviews or additional primary studies are needed.

It is also important that included reviews be of high quality. We incorporated key elements of the AMSTAR checklist,37 one of several tools that can be used to assess the methodological quality of systematic reviews, into our inclusion criteria and analysis. Specifically, for a review to be considered “systematic” and included in our map it had to have 1) included a comprehensive literature search of one or more citation databases, 2) based study selection on prespecified inclusion and exclusion criteria, and 3) assessed the quality (or risk of bias) of individual studies included in the review. These correspond to three of the eleven AMSTAR criteria. Reviews that did not meet these criteria were excluded. Two additional AMSTAR criteria are included in our descriptions and analyses. We documented whether each review included a strength of evidence (SOE) assessment, which entails incorporating the quality of the individual studies and other explicitly stated criteria into a rating of the body of evidence. We also documented if meta-analysis were used to combine quantitative results. Incorporating these five criteria in this way allowed us to give significant weight to these criteria that were essential to our purpose and to use them in different ways in our descriptions and analyses that would not be possible if they were combined into a single AMSTAR score for each review. This does not mean the other criteria are not important nor that the score is not useful, rather in our design of an evidence map we decided to focus on these critical criteria and create a flexible approach that allowed us to use different criteria at different points in the process.

Outcomes. Included systematic reviews had to report clinical, resource utilization, or cost outcomes, corresponding to our interest in research on the effectiveness of telehealth in terms of patient-level outcomes. We did not include other outcomes such as patient or provider satisfaction with or attitudes toward telehealth or assessments of diagnostic accuracy or

agreement when telehealth was used. We also excluded studies where the outcome was the extent or success of implementation

Population. We included reviews that included studies involving adults and/or children for whose care telehealth was used for prevention, diagnosis, or treatment for any health condition.

Timing. We did not restrict inclusion according to timing, length of the intervention, or length of followup. We included systematic reviews published in 2006 or later and that included a search with an end date in 2005 or later.

Setting. We did not restrict the location of either the provider or patient.

To identify potential studies, abstracts were reviewed by two investigators and full-text articles for all citations deemed appropriate for inclusion by at least one of the reviewers were retrieved. Full-text articles were reviewed for inclusion or exclusion by one investigator and confirmed by a second investigator. Discrepancies were resolved by discussion and consensus.

Included systematic reviews were grouped by clinical focus and telehealth function. These groupings were developed by the investigators based on the identified and included reviews. For this reason, they are listed and defined in the Results section of this brief.

We used these groupings to describe the literature and generate the tables and bubble plots that constitute our primary analysis. As a secondary analysis we then compared the topics covered by the included reviews to major topic areas in which telehealth interventions have been observed. For the topics not covered by our included systematic reviews, we first examined the topics covered by excluded reviews as a means of verifying the existence of studies that could be analyzed, and supplemented this with searches for primary research on the topics that were still not represented. We applied the same inclusion criteria related to the population, intervention, definition of telehealth, outcomes, time, and setting, but not study design, to identify potentially relevant primary research.

Data Extraction and Data Management

After identifying the subset of systematic reviews that met our inclusion criteria, we  extracted data from the reviews into tables. This included basic information (dates of search, number of included studies, number of included randomized controlled trials [RCTs]), information on the clinical focus area, study purpose, populations included, the function telehealth played in care, telehealth modality/technology, and two indicators of the rigor and type of analysis used in the review (i.e., was there a strength of evidence assessment and was a meta- analysis attempted?). To develop these tables, we started with a list of information of interest and tested it on selected included studies, and then refined and finalized the list to include what is reported in the identified systematic reviews on telehealth. See Appendixes E and F for data extraction tables. Additionally, we generated a list of the included studies and their sample sizes for each review (Appendix G). We used these lists to eliminate duplicate studies and avoid double counting when reporting the number of studies and patients in the included reviews within a clinical or functional area as indicators of the size of the evidence base.

Data Synthesis: Generating an Evidence Map

An evidence map combines graphics, tables, and accompanying text. While the methodology for evidence maps is not standardized, by nature they involve a reductive approach to summarizing and presenting information. Evidence maps are not designed to be comprehensive,rather they present selected characteristics and they rely on categorization and grouping of information.

For our evidence map, the core graphics are two bubble plots and a figure. One bubble plot is organized by clinical focus of the telehealth interventions, while the other is organized by health care function. We selected these two categorizations because decisions about telehealth are usually made about its use with particular types of patients (clinical focus, e.g., patients with diabetes or heart failure) or to deliver a type of health care service (e.g., health care functions such as remote patient monitoring or psychotherapy). Each of the two bubble plots then convey four additional dimensions about the clinical focus or health care function categories: 1) number of studies included in reviews, 2) total sample sizes (e.g., the number of patients), 3) the general direction and strength of any reported effect, and 4) the percentage of the reviews that include strength of evidence assessment. A third figure presents how the clinical focus and function categories intersect. This creates groupings of the evidence that are more specific, for example it shows what evidence is available about telehealth for remote patient monitoring (function) for people with chronic conditions (clinical area). This figure reports for each grouping: 1) the number of included systematic reviews, 2) the number of studies in the reviews, and 3) the overall conclusion of each review.

To develop the clinical and function categories, team members first individually extracted the information from all the included reviews. Then team members met, reviewed the topics of the included reviews, and developed the categories for clinical focus and function through discussion and consensus. It is important to note that the categories for this descriptive analysis were derived from the literature and do not constitute a list of all possibilities for telehealth. For clinical focus the reviews included those with specific indications (e.g., diabetes) or more general clinical areas (e.g., behavioral health). For function we grouped the reviews by the service the telehealth intervention provided (e.g., monitoring and counseling). Each included systematic review was assigned to the one clinical focus and the one function category that best matched its content. The categories and their definitions are included in the Results below.

While both of these bubble plots represent the same group of systematic reviews we chose these two different approaches to organization and representation of the information as these correspond to the key types of decisionmaking identified by the Guiding Questions and the KIs. Policy and clinical decisions can be made based on the functional category (e.g., decisions about programs for monitoring chronic disease in the rural elderly), by a clinical area (e.g., support for specific interventions for patients with diabetes), or by combinations. The organization of the evidence in these plots and the figure are designed to assist users in determining if evidence exists about specific telehealth interventions and if this evidence can be used to inform decisions about telehealth for specific clinical focus areas and/or functions in health care delivery.

The individual bubbles in these plots represent the specific clinical area or function for which we found systematic review evidence that met our criteria. The other dimensions are the number of individual studies (indicated by the size of the bubble), the number of patients studied (represented on the y-axis), and a weighted estimate of the reported effect (represented on the x- axis). The first two characteristics required looking across reviews and determining which studies were in multiple reviews. The lists of studies included in each review were used to create a list of primary studies without duplicates. The number of patients in these studies and the number of unique studies were then aggregated by both clinical focus and function. In the figure that combines clinical focus and function, the number of studies reported is the total number in each review and therefore there are duplicates within the cells.

Creating a weighted way to summarize the conclusions about the effectiveness of telehealth reported in the systematic reviews involved more interpretation and decision rules. First we reviewed the conclusion of each review. Many reviews include multiple outcomes (details are provided in Appendix E); however, the conclusions needed to be summarized. The systematic review results were coded as 0=no effect, 1=unclear, 2=possible positive effect, 3=positive effect based on an assessment of the primary outcomes, as well as all outcomes. If the results had a consistent direction of effect for the primary outcomes, the codes of 0=no effect, or 3=positive effect were used. If the primary results were mixed, any secondary results were also considered. If the results had an inconsistent direction of effect and the review authors stated that a conclusion was not possible it was coded as 1=unclear in order to indicate that it was unclear what the conclusion about telehealth should be, not that the evidence was unclear. If either some primary outcomes or the majority of all outcomes showed a positive effect, the conclusion was coded as 2=possible positive effect. These codes were assigned by one team member and were checked by another team member. Any differences were discussed by the entire team and a code was assigned based on that discussion.

The weighted estimate of reported effect used in the bubble plots was created by multiplying the overall conclusion code (0 to 3) by the number of studies in the review, and then averaging the scores for all reviews in given clinical area or function. While this did not create a value with absolute meaning, it allowed us to compare the relative strength of the conclusions by clinical area or function. Based on this, the farther to the right the position of the bubble is on the plot, the more the conclusions of the reviews are consistently positive, where farther to the left indicates no effect or unclear findings.

While bubble plots and the intersection figure provide an overall picture of the literature, they can only represent limited numbers of variables and estimates that are not extremely precise (more detail is provided in the Discussion section of this brief). For this reason, we have included other tables, charts, and narratives. We used Chi-square tests to explore relationships between the conclusions of the systematic reviews and the patient setting, type of outcome, whether the reviews used quantitative analysis, and whether the reviews incorporated as assessment of the strength of evidence in their conclusions. We consider the evidence map to consist of the sum of the information in this report.

Findings

Results of Literature Searches

The search and selection of systematic reviews are summarized in the literature flow diagram (Figure 2). Database searches of published literature resulted in 1,311 potentially relevant articles. We identified an additional 183 potentially relevant articles through the grey literature search and searches of Web sites for telehealth organizations and government agencies. After dual review of abstracts and titles, 617 articles and grey literature reports were selected for full- text dual review. Of these, we determined that 58 systematic reviews met the inclusion criteria and we included these in the literature map. A list of included reviews is in Appendix C. We did not identify additional eligible reviews or relevant reports via the search of government or organization Web sites. The two reasons we excluded reviews that were about telehealth were: 1) because they included mixed interventions, some of which did not meet our definition of telehealth, and they did not report results separately for the interventions and outcomes we included or 2) they did not meet our criteria for a systematic review (i.e., they did not state questions, search citation databases, and assess the quality of identified studies; see Methods above). The reviews excluded for these two reasons are listed in Appendix D.

Figure 2

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Description of Included Systematic Reviews

The 58 included reviews met the strict criteria we established to distinguish systematic reviews that provided content organized, analyzed, and presented in a way that could support contemporary policy and practice decisions about telehealth.38-95 Information abstracted from each included systematic review is provided in Appendixes E and F. Figure 3 presents the publication year of the included reviews. Almost 80 percent (46 of 58) were published since 2011, indicating a high level of interest in this topic and enough studies to support numerous reviews.

Figure 3

 

 

 

 

 

 

 

 

 

 

 

 

Table 1 includes descriptive information on basic characteristics of the included reviews, such as the final year of the searches performed in the review, which we used as a means of checking how current the information was that formed the basis for the reviews. Table 1 also reports the setting (i.e., where the patients were located). In the majority of reviews, the patients were in their home (60%), while a small number of reviews addressed telehealth when the patient was hospitalized (5%), and some included a mixture of settings including home, hospital and clinic (35%). In terms of the types of outcomes reported, 55 percent reported clinical outcomes, 12 percent reported cost outcomes, and 33 percent reported both clinical and cost outcomes.

Table 1 also includes three characteristics of the reviews that can be used to consider the rigor and utility of the reviews. The current standards for systematic reviews require more than simply listing and describing individual studies – they require that the evidence for a topic be synthesized across studies and that the body of evidence is evaluated. This evaluation is often referred to as strength of evidence (SOE) assessment.34-36 We defined SOE as an evaluation of a group of studies that incorporated prespecified criteria. Quality assessment of individual studies is always considered in determining strength of evidence but other criteria are often added. For example, the SOE used in AHRQ reviews frequently includes the following criteria in addition to the quality of studies: consistency (i.e., whether the results are consistent across studies), directness (i.e., whether the studies provide direct or indirect evidence given the questions asked in the review) and precision (i.e., how precise the effect estimates are). SOE may also consider other factors such as evidence of publication bias. Less than half (41%) of the included reviews reported some form of SOE assessment. We also reported on whether the reviews included a meta-analysis. While quantitative synthesis is not appropriate in all cases, it was used in 26 (45%) reviews. The final characteristic of the included reviews reported in Table 1 is whether the review did or did not present a conclusion about the effectiveness of telehealth. Just over one- fifth (22%) reported that the evidence was inconsistent or contradictory (we used the term unclear to include both) and did not draw a conclusion after reviewing the literature.

Table 1

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The modality or type of technology used for telehealth is also included in Table 1 and represented in Figure 4. Half of the reviews (50%) included mix modalities, meaning either they included primary studies that used multiple technologies or they included primary studies of several different technologies.41-45,47,50,51,54-56,58,65,70,71,73,76,79-82,85-87,90,91,94,95 Other reviews limited their inclusion to studies of asynchronous communication, employing various technologies, including special monitors or internet-based applications on standard computers(29%) to facilitate communication.38,46,49,59,60,64,66,67,69,72,77,83,84,88,89,92,93 Another common modality was videoconferencing, which was the focus of 17 percent of included reviews.39,40,48,57,61,62,68,74,75,78 A smaller number of reviews included only studies that used mobile phones for telehealth (4%).53,63

Figure 4

 

 

 

 

 

 

 

 

 

 

 

Because technology has changed over time, we also looked at the distribution of the modalities covered by the included reviews by publication date (Figure 5). The number of reviews published that reviewed several technologies (mixed) peaked in 2012 at six but was also high in 2014 and 2015. One or two reviews of studies of video have been published every year from 2008 to 2014. More recently, since 2010, reviews have been published that evaluate the use of asynchronous communication and mobile phones.

Figure 5

 

 

 

 

 

 

 

 

 

 

 

 

 

We classified the included systematic reviews according to the clinical focus and telehealth function. As explained in Methods, these were developed based on our review of the included studies. “Clinical focus” is the clinical condition, indication, or situation telehealth was used to address. Function is the role telehealth played in health care. These categories are defined below and the number of reviews in each category is provided in Table 2.

The categories identified and used to describe the clinical focus of each systematic review are:Cardiovascular disease: These reviews included studies of the use of telehealth for the management of heart failure, acute care and followup for myocardial infarction, management of patients with implantable defibrillators, and primary and secondary prevention of coronary disease.

Diabetes: Reviews in this group included management of type 1, type 2, and gestational diabetes and a target range of activities from regulating glucose levels to promoting physical activity.

Respiratory disease: This category included reviews of telehealth interventions for managing chronic obstructive pulmonary disease (COPD), cystic fibrosis, asthma, and lung transplantation. Mixed chronic conditions: These reviews considered that the uses of telehealth are similar across chronic conditions and included studies conditions such as asthma, hypertension, diabetes, COPD, and kidney failure in their reviews. Physical rehabilitation: These reviews included telehealth uses for rehabilitation for stroke, traumatic brain injury, or multiple reasons in children or adults.

Behavioral health: Telehealth was evaluated in these reviews for mental health treatment for unspecified conditions, treatment of depression and anxiety, and addiction treatment, including substance abuse, smoking cessation, alcohol abuse, and pathological gambling.

ICU or Surgery: This is a comparatively diverse category, including use of telehealth to allow physicians to advise on ICU patient management or surgery remotely and to facilitate communication between NICUs and parents of preterm babies.

Dermatological conditions: This category included a review of studies that focused on treating several skin conditions.

Preterm birth: This refers to telehealth interventions designed to monitor maternal and fetal health and prevent preterm birth.

Mixed: When a review included uses for a wide range of conditions it was assigned to this category.

Burn care: This refers to telehealth interventions designed to address the clinical needs of patients recovering from burns.

In creating the function categories we looked at the activities telehealth was used for and, when appropriate, what activities telehealth supplemented or replaced. We grouped the reviews in to the following categories:

Remote patient monitoring: This category included interventions that are often called by other names such as home telehealth, or home telemonitoring, but it is broadly defined to also include remote monitoring and management of patients in other settings such as patients in critical care units or patients during transport to the hospital. Home telehealth may require special devices or may use computer applications and networks while in hospital monitoring may include video as well as transmission of data from monitoring devices. The key characteristic is that it involves the collection of data about a patient, usually physiological data such as blood glucose, weight, and blood pressure over time, and this data is transmitted to a health care provider or care team which reviews the data and adjusts care (often medications) based on this data.

Communication and counseling: This category included the use of technology to facilitate the exchange of information between a patient and health care provider as well as the provision of advice. This could be synchronous, as is the case with videoconferencing and chat or asynchronous such as via Web sites or email. These interventions are often designed to increase access and can be used to replace or supplement face-to-face interactions with health care professionals.

Psychotherapy: This differs from general or limited counseling and includes the use of technology to provide a course of treatment for a mental health condition.

Consultation: This category was applied to interventions designed to facilitate involvement of another provider, often a specialist, across time and/or distance. While the patient may or may not be involved in the consultation, the consultation was required to be about a specific patient in order to differentiate this from training (which would not meet our definition of telehealth).

Telementoring: This category was similar to consultation but refers specifically to the use of technology to allow a remote provider to view and advise on a procedure being conducted in another location in real time.

Telerehabilitation: This included any type of rehabilitation services delivered via technology so the patient can be in a different location or can be engaged in rehabilitation activities at different times.

Multiple Functions: In some cases, the intervention included more than one function and in these cases we classified it as mixed.

The second largest group in the clinical focus classification is “mixed chronic conditions.” These nine systematic reviews all defined their inclusion criteria such that the reviews either combined studies of several individual conditions, included primary studies with patients with more than one condition, or both. For example, the review by de Jong that evaluated internet communication between health providers and patients with chronic conditions included studies in which all patients had the same condition (e.g., diabetes), studies in which patients had related conditions (e.g., chronic neurological conditions), and studies in which included patients had different conditions (e.g., chronically ill women with a variety of clinical conditions).46 The similarity in all these reviews was that their scopes were limited to chronic conditions.

Mixed conditions was the label given to reviews that included a wide range of conditions, all of which may not have typically been considered chronic. In many cases these reviews focused on a particular technology or health care function and included studies from varied patient populations. For example, a review of electronic patient portals included studies with populations undergoing in vitro fertilization, diabetes, and congestive heart failure, and patients without specific conditions.38 Another review of electronic symptom reporting included studies of patients with several conditions including cancer and diabetes.56

Telehealth function included a similar category: multiple functions. Ten reviews were coded this way when the included studies stated telehealth was used for more than one function. For example, several reviews had a focus on a specific technology: video conferencing74,78 or the internet,49 and the technology was used to communicate with, monitor, and treat patients.

Data from Table 2 as well as the conclusions of the included systematic reviews were used to generate the bubble plots presented later in this report.

Table 2

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figures 6 and 7 graphically present the distribution of included systematic reviews across the three major characteristics of the reviews (clinical focus, telehealth function, and telehealth modality).

Figure 6 depicts the distribution across clinical focus areas for the included reviews. Taking into account the number of reviews, primary studies, and patients, the most common clinical focus areas studied were cardiovascular disease (12 reviews). 41,42,45,47,67,73,81,82,84,87,89,95 The next largest group was mixed chronic conditions (9 reviews),46,48-50,54,60,68,74,76 followed by diabetes (8),43,53,63,69,80,88,91,92 behavioral health (7),39,51,65,70,72,90,93 and mixed conditions (6).38,56,59,64,78,86 Focus areas with five or fewer included systematic reviews were physical rehabilitation (5),54,57,62,71,85 respiratory disease (5),44,55,58,66,83 ICU or surgery support (3),40,61,75 burn care (1),79 dermatology conditions (1),94 and preterm birth (1).77 Over one-quarter of included systematic reviews (26%) focused on mixed chronic or mixed but not exclusively chronic conditions.

Figure 7 depicts the distribution of the function the telehealth interventions perform in health care delivery. The included reviews examined telehealth used to provide treatment, monitor patients’ signs and symptoms, or facilitate communication between provider and patient. These

functions could replace or supplement in person service delivery. Telehealth was most frequently used for remote patient monitoring (17 reviews)41,42,45,54,55,58,66,68,73,76,77,81,83,87-89,91 and communication and counseling (14 reviews).38,43,46,56,59,60,63,64,67,71,72,75,92,95 Ten reviews combined research on multiple functions,44,49,50,53,69,74,78,80,84,86 seven summarized studies in which telehealth was used for deliver psychotherapy,39,48,51,65,70,90,93 and five reviews focused on telerehabilition.54,57,62,82,85 Four reviews examined studies in which telehealth was used to provide consultations about patient care47,61,79,94 and one review focused on telementoring.40

Figure 6

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 7

 

 

 

 

 

 

 

 

 

 

 

 

Indicators of Rigor of the Reviews

In Table 1 we reported selected characteristics of the included reviews that can be interpreted as indicators of the rigor or utility of the review. Specifically assessing the results across studies using either a “strength of evidence approach” or quantitative synthesis (i.e., a meta-analysis) are of interest, as reviews that incorporate these approaches may be more in accordance with contemporary standards for high-quality systematic reviews. While meta-analyses can be done poorly it may produce results and conclusions that are more definitive and easier to interpret.

In Tables 3 and 4 we report the percentage of included systematic reviews that used these approaches (strength of evidence and meta-analysis) as well as the number of studies in the reviews that were RCTs, according to clinical focus and telehealth function. While it is possible for RCTs to be of poor quality, randomized studies are generally considered to be higher in the hierarchy of evidence than observational studies and a preponderance of RCTs is often an indication, albeit imperfect, both of interest in the topic and the quality of the evidence.

Table 3 reports these systematic review characteristics by clinical focus. From this table it is possible to see that some clinical areas, such as burn care and ICU/surgery support, had been the focus of at least one systematic review, but that these reviews contained no or few RCTs and have not included meta-analyses. However, of the three reviews about ICU/surgery support, two included a strength of evidence assessment. Reviews of other topic areas also had a large proportion of RCTs among the included studies such as cardiovascular disease (85%) and diabetes (82%), and about half of the systematic reviews for these clinical focus areas reported strength of evidence (58% and 50%, respectively).

Table 3

 

 

 

 

 

 

 

 

 

 

Table 4 includes the same information by telehealth functions. Reviews of telehealth for consultation and telementoring had fewer RCTs. Reviews of communication and counseling studies contained 88 percent RCTs.

Table 4

 

 

 

 

 

 

 

We also evaluated the included reviews (n=58) for relationships between the conclusion (i.e., whether the telehealth provided benefit) and several independent variables; use of quantitative analysis (meta-analysis vs. not); use of strength of evidence (or not reported); and type of outcome (clinical, cost or utilization, or combined). Conclusions (dependent variables) were defined with two approaches: 1) benefit vs. no benefit and 2) reported positive or negative conclusion vs. no clear conclusion. All included studies were code for these independent and dependent variables. Chi square tests were used to explore differences between the observed counts and the expected counts using SPSS® (IBM SPSS® Statistics for Windows, Version 23). We found no statistically significant relationships (p>0.05) across all the chi-square analyses; however, the cell sizes for some comparisons were less than 10, suggesting that this quantitative approach was not appropriate for the analysis of this number of reviews and variables. To better understand the relationships between clinical condition, telehealth function, and effectiveness, we used qualitative approaches and the graphical presentations included in the next section.

Evidence Map Core

The bubble and intersection plots in the sections below constitute the core of our evidence map and help to clarify the type of evidence that exists on telehealth and how useful it is for policymaking and clinical decisionmaking. We constructed the bubble plots for clinical focus and for telehealth function. After examining the results, we also created an intersection plot in order to examine how clinical focus and function overlap. Combined, we believe these provided the insights that most closely matched our stated objectives and questions and provided the best way to summarize and assess the state of the evidence about telehealth. In this section we presented a more detailed analysis after an overview of the three plots.

Clinical Focus

Figure 8 is the bubble plot by clinical focus. In this plot each bubble is a clinical focus area.

The y-axis is the number of patients in studies in the systematic reviews, so the higher up the bubble is on the grid, the more patients were studied. The lists of studies were deduplicated, so that each patient is counted only once within a bubble. The size of the bubble is the number of studies included in the reviews, again with each study counted only once when determining the size of the bubble. The color of the bubble represents the percent of the reviews that included strength of evidence assessment. The horizontal placement along the x-axis is determined by weighting the overall conclusion of each review (coded as 0=no benefit, 1=unclear, 2=potential benefit, and 3=positive benefit) by the number of studies in the review. As stated above in the Methods section, this weighted estimate of reported effect was created by multiplying the overall conclusion code by the number of studies in the review and then averaging across the reviews for the clinical area or function. Bubbles more to the right indicate more positive findings while bubbles to the left represent findings that include more unclear conclusions or more reviews reporting no benefit. While the weighting does not create a value with absolute meaning, it allows comparisons of the consistency and direction of the conclusions across clinical areas.

Figure 8

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

As noted above (Table 2), the research volume, as measured by both the number of studies and the number of patients, is largest for mixed and mixed chronic conditions, followed by cardiovascular disease and behavioral health. This finding is represented on the plot by the fact that the bubbles are large and higher up on the y-axis. The mixed and mixed chronic condition bubbles are also farther to the right, indicating the conclusions of the reviews were that telehealth consistently provides benefit. The bubble representing diabetes shows that it is the single condition with fewer studies (the bubble is smaller) than mixed chronic conditions but about the same number as cardiovascular disease. However, diabetes studies included with fewer patients (the bubble is lower) than the cardiovascular disease studies but the findings were more positive findings (the bubble is farther to the right).

Telehealth Function

In addition to the evidence map by clinical focus, we also looked at the evidence by telehealth function. Figure 9 represents the same included systematic reviews as shown in Figure 8 except the reviews are summarized by the function telehealth played instead of clinical focus. Each bubble is a function of telehealth. The other variables are the same as in Figure 8. The y-axis is the number of patients in a deduplicated list of studies in the systematic reviews for that function; the size of the bubble is the number of unique studies included in the reviews about that function; and the color of the bubble is the percentage of reviews that include strength of evidence assessment. The horizontal placement along the x-axis is determined by weighting the overall conclusion of each review by the number of studies in the review (bubbles more to the right indicate more positive findings while bubbles to the left represent finding that that are unclear or found no benefit).

In this bubble chart, communication and counseling is the function bubble highest and farthest to right, indicating the most reports of positive benefits of telehealth when used for these purpose and that the studies in these reviews contained the highest number of patients among the function categories. Remote patient monitoring is lower than communication, as these studies included fewer patients, but it is higher than other functions. Remote patient monitoring is also toward the right, indicating that most reviews about remote patient monitoring conclude that telehealth provides benefits in quality of care or in utilization.

Reviewing the bubble plot provides a means of both comparing the characteristics of available evidence across topics and identifying areas where systematic reviews are not available to support decisions. The next steps in our analyses and mapping were designed to explore where clinical focus and function overlap.

Figure 9

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The Intersection of Clinical Areas and Telehealth Intervention Function

Much can be learned and several conclusions drawn from examining the literature on telehealth by clinical area/population of patients or by the health care function telehealth serves. However, examining the intersection of clinical areas and functions provides additional, more finely grained and potentially more useful insight for determining which telehealth intervention(s) could or should be used in specific patient populations. Figure 10 displays how the evidence clusters by telehealth clinical area and function. Each of the systematic reviews that provide evidence for the intersection of the clinical area and function are represented in the corresponding cell by a circle that is shaded to represent the overall conclusion of the review.

The number of studies in each review is included to the right of the circles. This intersection plot demonstrates how the research evidence about telehealth clusters into a few clinical area/telehealth function pairs, the extent to which the conclusions are consistent within and across these pairs, and the volume of research for each pair. For example, the pair with the most reviews is cardiovascular disease and remote patient monitoring. There are seven circles in this cell representing seven systematic reviews. The shading indicates that the conclusions included five reviews finding telehealth provided benefit, one citing potential benefit, and one reporting no benefit. The reviews range in size from 4 to 30 studies.

Identifying subgroups of reviews allows more details to be examined in order to better understand patterns and diversity within the pairs. Empty cells include some potentially important topics for which we did not identify any reviews, and therefore could be topics for future reviews if primary literature is available. Some empty cells are intersections that are likely not applicable (e.g., psychotherapy and physical rehabilitation).

The diabetes and communication pair provides an example of the diversity of reviews even within a cell, the range of information available, and the challenges researchers conducting reviews and users of the reviews face. Three reviews concluded that telehealth resulted in benefit or potential benefit. One review summarized studies of social networking services in diabetes care and concluded that their use was feasible and effective.92 Another review focused on how mobile phones were used in several ways to provide support and encouragement for patient self- management activities such as monitoring glucose, exercise, and maintaining diets, and found strong evidence of improvement in glycemic control in all patients, but the strongest for type 2 diabetic patients.63 A third review reported potential benefits based on included studies that evaluated the use of a range of technologies to promote physical activity as part of type 2 diabetes management and concluded that telehealth is effective but that additional interventions were needed to sustain adherence, noting that the high dropout rate also raised concerns about potential bias in the results.43 Reviews of telehealth for diabetes that included multiple functions varied in that the conclusions were less strong, with two concluding the evidence was unclear and one reporting potential benefits from telehealth. One study in this clinical focus-function pair included studies of different technologies (e.g., electronic messaging, Web sites, and video conferencing) used to support glycemic control in patients with type 2 diabetes, but concluded that the evidence in their review was “unconvincing” due to concerns about publication bias and a small effects.80 Another review related to diabetes was in the multiple function group as it included studies of mobile phone use both for communication as well as remote patient monitoring. This review concluded that telehealth shows promise in this area but the evidence lacked rigorous study designs. Specifically, studies had insufficient sample sizes and short interventions and followup periods.53 A third review of Web-based interventions for type 2 diabetes concluded that the Web could be used for behavioral interventions and to support self-management, however, the favorable results were enhanced if these were supplemented by other interventions such as case managers or mobile phone support and followup.69 Examining this plot shows where synthesized bodies of evidence (i.e., systematic reviews) about telehealth are available and allows more in-depth examination of details such as those included above about telehealth for diabetes and communication. Considering the empty cells or those with few or limited reviews allows consideration of the importance of these areas and whether they are gaps that should be addressed in future reviews and/or primary research. In the next sections of the results we identify gaps and delve more into selected topics. Then in the Discussion section we use this information to create categories related to the sufficiency and need for research in selected areas.

Figure 10

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 10 part 2

 

 

 

 

 

 

 

 

 

Gaps and Priority Topics

Evidence Gaps

In order to identify which clinical and functional focus areas were not covered in the included systematic reviews, we assembled lists of telehealth practice domains generated by selected organizations and in reports on uses of telehealth, reviewed the notes from our KI interviews,  and drew on our team’s experience and expertise. Examples are included in Table 5.         Certain domains on these lists may not be relevant if they do meet the definition of telehealth used for this report. For example, remote health care data management and some ancillary telemedicine services may not involve or augment an interaction between a provider and patient or interactions among providers about a specific patient, and would not be included here. It is also important to note that there is no definitive or authoritative list of domains, that these lists do not exactly correlate with our clinical focus areas and our definition of telehealth, and that the domains across these lists may overlap. This could be problematic if mutually exclusive categories were needed. However, for our purpose, which is to identify areas where systematic reviews that could support decisions are not available, these are useful. An initial review of these lists led us to identify certain areas that were not represented in our included reviews. For example, one such area is urgent/primary care.

Table 5

 

 

 

 

 

 

 

 

 

 

In order to determine if systematic reviews were underway on additional topics, we searched PROSPERO, the international prospective register of systematic reviews maintained by the University of York, Centre for Reviews and Dissemination.98 We searched from August 1, 2013 through February 2016 for any ongoing reviews with the following words in any field: “telehealth” OR “telecare” OR “telemedicine” OR “eHealth” OR “mHealth.” We reviewed the titles and identified 82 registered reviews that were listed as ongoing and appeared to be relevant. A list of the topics covered and the number of reviews on each topic is included in Appendix H. The most frequent specific topics of these reviews in process that were not well-represented in our included completed reviews are weight loss, cancer, and maternal/child health. Other topics such as diabetes (6 reviews in progress) and mixed chronic conditions (3 reviews in progress) are represented in our included reviews, however, the PROSPERO entries means additional evidence syntheses will be available in the near future.

We also looked at the reviews we excluded (see Appendix D for a list and Appendix I for a table with the clinical focus area and the number of reviews). While these reviews did not meet our inclusion criteria, knowing there are substantial numbers of these reviews and which clinical areas they cover is useful because it could indicate that a more formal, rigorous, or differently structured systematic review could be conducted using the primary studies in these reviews. An assessment of the clinical focus areas covered in these excluded reviews revealed that they included additional areas not well-covered in our included reviews such as cancer, chronic pain, autism, and pregnancy (our map is limited to one review on uterine monitoring to prevent preterm birth). At the same time, this list also includes many of the areas that were covered in our included reviews, suggesting it is possible additional research exists that could be added to the body of evidence for these areas. It may also suggest that the utility of the evidence in these reviews could be increased if the included studies were summarized and analyzed in a different way. Specifically, reviews could 1) include quality assessments of the studies they include and examine whether their conclusions would differ if only high-quality studies were included and 2) include subgroup analysis by clinical focus or telehealth function or modality for reviews that have a wide range of telehealth interventions.

We identified one example of this type of supplemental analysis in our literature search. The original review99 was excluded by us because structured telephone calls were included interventions in the review along with videophone and telemonitoring and the results were not summarized by type of intervention. In the subgroup analysis completed and published later, Conway and colleagues81 reanalyzed the data by type of technology used in remote monitoring for heart failure. This allowed us to include the review for the telehealth interventions, and exclude the studies of telephone calls. It is likely that this analysis would also be more informative for practice and policy decisionmaking.

After reviewing the topics covered in the included systematic reviews, reviews in progress, and the reviews that were excluded, we identified three areas in which telehealth had been proposed as appropriate or studied that were not well-represented: 1) triage for urgent care/primary care, 2) maternal health, and 3) pediatric cancer and chronic pediatric health conditions, and one area, 4) dermatology, where telehealth has been widely used, but the focus of the research included in reviews had been on diagnostic accuracy/agreement, not patient outcomes. We conducted a targeted search for primary studies on each of these topics (from 2006) and reviewed the abstracts for clinical focus, telehealth function, and modality. (Search strategies appear in Appendix A.)

Triage for urgent care has historically been provided most often in the form of advice from a nurse by phone; however, recent studies suggest telehealth is playing an increasing role, particularly related to heart health. Telehealth interventions are being used help decide the level of care needed in different situations. These have been used to address the following indications: chronic heart failure, arrhythmias causing dizziness/presyncope, flu, and a variety of primary care indications. Telehealth interventions are being used for the following functions: communication, monitoring, and diagnosis of heart arrhythmia conditions. Several modalities are described in the literature: mobile phone images, patient portals, single-lead electrocardiogram, mobile applications, and continuous mobile cardiac outpatient telemetry. We reviewed 353 abstracts of primary studies but identified only five potentially relevant studies, two of which were about cardiac care and may overlap with our included reviews on cardiovascular disease. Much of this literature on triage was about telephone-only advice or nurse lines, which is not telehealth as defined for our evidence map. Another subtopic identified that did not fit our definition is use of telehealth by first responders forwarding data to the emergency room about myocardial function or other cardiovascular problems so the emergency department can be prepared when the patient arrives.

For maternal health, we reviewed 129 abstracts and identified 33 articles that evaluated several telehealth functions (remote fetal monitoring, antepartum cardiotocography monitoring, triage, consultation, counseling and health promotion, communication, screening, and diagnosis) in managing the following: gestational diabetes, perinatal depression, high-risk pregnancy, fetal and pediatric cardiology, pre-eclampsia, pregnancy termination, and fetal alcohol spectrum disorder. The studies used several modalities, including robotic ultrasound, videoconferencing, patient portals, text messaging/SMS, customized Web sites, mobile applications, and electronic logbooks.

We reviewed 61 abstracts from our search on telehealth for pediatric cancer and other chronic pediatric conditions. We found 12 potentially relevant studies¾articles that performed the following telehealth functions: counseling in the form of support to families, remote case management, monitoring, psychotherapy, and consultation. These studies addressed the following indications: five were about pediatric cancer and the others covered several conditions including asthma, tic disorders, and other complex illnesses.

For dermatology, we identified and included one systematic review94 and reviewed references from one narrative review100 which included studies of clinical outcomes in addition to diagnostic concordance. As this suggests that research in the field may be expanding, we searched for studies of teledermatology that included clinical outcomes. We identified 315 abstracts on telehealth and dermatology of which only 15 included indexing terms for clinical outcomes. Our review of both the subset and the larger set of results failed to identify a discrete group of primary studies of teledermatology with clinical outcomes. The results included the studies in the reviews mentioned above, as well as abstracts of descriptive articles, articles not in English, feasibility studies, studies of diagnostic concordance, and studies with outcomes that were mixed or not clearly stated in the abstract. While further analysis of the literature would be needed to definitively confirm this, it appeared there were still few studies of teledermatology that include clinical outcomes.

Priority Topics

In creating the plots and tables, we sorted the included reviews into 11 different clinical categories and 7 different functions. While it is not unusual for bubble plots in literature maps to have 30 to 50 categories,5,8 we also wanted to look across categories and summarize the results related to selected key policy and practice questions. In this section we describe in more detail the findings related to two subsets of the reviews that cut across categories and overlap, but that represent important approaches to considering telehealth.

Chronic Disease/Older Patients

A frequently cited target population for telehealth is patients with chronic disease, most of whom are older. The logic is straightforward, if over simplified here: patients with chronic disease are likely to require frequent visits for monitoring and management as well as support to self-manage their conditions. However, they may have barriers to access, or office visits may not be the best type of support and these challenges can be ameliorated by telehealth. Furthermore,by avoiding acute episodes (e.g., hospitalization for COPD) or adverse effects (e.g., amputations in diabetics), telehealth can reduce costs while increasing function and quality of life.

Thirty-one of the systematic reviews we identified for this report examined telehealth in either multiple chronic conditions or specific chronic conditions. This included the 9 reviews we classified as mixed chronic conditions,46,48-50,52,60,68,74,76 all 8 of the reviews that focused specifically on diabetes,43,53,63,69,80,88,91,92 1041,42,45,73,81,82,84,87,89,95 of the 12 reviews on cardiovascular disease (those not considered chronic included 1 about acute myocardial infarction47 and 1 about primary prevention),67 and 4 of 5 about respiratory illness (1 included telehealth use in transplant),55 including 3 about COPD58,66,83 and 1 about cystic fibrosis.44

Only two of these reviews, one about diabetes53 and one about cystic fibrosis,44 included studies with children as well as adults, and one other review included pregnant women with pre- existing diabetes.88 While most of the reviews did not specify elderly, the patients in these studies were adults with chronic conditions and most were older. The majority of the systematic reviews (16 of 31) included telehealth interventions used for remote patient monitoring : six reviews focused on telehealth used to provide counseling or facilitate other communication and seven reviews included multiple functions, while one review examined the use of telehealth for psychological therapies and one considered rehabilitation.

As a group, the conclusions of the systematic reviews of research on the effectiveness of telehealth for chronic conditions were generally positive. Of the 31 studies, 13 (42%) reported benefits in primary or most outcomes, 11 reported potential benefits, 4 found no benefit, and 3 stated that the impact was unclear. Details on the findings from the thirteen reviews reporting benefits are included in Table 6 below. These reviews have characteristics associated with rigorous systematic review methods: 8 of 13 included some approach to assessing the strength of evidence across studies and 10 included a quantitative meta-analysis.

Table 6

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 6 part 2

 

 

 

 

 

 

 

 

 

 

 

 

 

Remote Patient Monitoring

Remote patient monitoring is a frequently studied telehealth function. Seventeen of the included reviews assessed telehealth use for monitoring and managing illnesses41,42,45,52,55,58,66,68,73,76,77,81,83,87-89,91 and five assessed multiple functions that included remote patient monitoring.49,50,78,84,86 Remote monitoring is of particular interest in considering telehealth because it makes new or significantly different forms of information and treatment available that can supplement and extend office-based care rather than replace face-to-face interactions. Specifically, many remote monitoring applications of telehealth allow patients to provide more data to providers, in a timelier way than could be obtained in outpatient visits, or allow patients to be monitored in their homes rather than in hospitals. With this information, providers can then tailor their recommendations and treatment. In this usage, telehealth changes not just the mode of care delivery (from face-to-face and in real time to something distant and/or asynchronous); rather it transforms the form of care.

In some studies, remote monitoring uses specialized devices to record and transmit data, but some types of remote monitoring may be done using more standard devices with specialized applications (e.g., mobile phones and computers with internet connections). Most, but not all of the reviews we identified used remote patient monitoring in the context of single or multiple common chronic conditions (e.g., diabetes, COPD, and congestive heart failure). The exceptions were a review of uterine monitoring of pregnant women to prevent preterm births77 and monitoring that allowed parents to monitor babies in neonatal ICU and communicate with providers.75

Of the 22 reviews that synthesized studies of monitoring, 10 concluded telehealth lead to positive benefits, 6 concluded benefits were possible, 2 were inconclusive, and 4 reported no benefit from telehealth. The 10 that reported benefits overlapped with those discussed in relation to chronic disease and are in Table 6 and the 6 that concluded potential benefit for remote patient monitoring also addressed chronic conditions. Three of the four that reported no benefits explored very different applications of remote monitoring: a review of the use of home uterine monitors to prevent preterm birth that found no impact on maternal and perinatal outcomes,77 a review of the addition of real time video as part of home care,68 and a review that identified only one study of the use of monitoring for parents with babies in neonatal intensive care unit that did not find a significant difference in the primary outcome (i.e., length of hospital stay).75

While the overlap of telehealth for chronic conditions and monitoring is not unexpected, it reinforces the potential of telehealth as a positive, transformative force in the care of chronic illnesses; one that may require more attention, development, or more adaptation for other uses.

Impact on Costs and Utilization

Fewer of the systematic reviews included in the map focused on costs or economic impact of telehealth exclusively. Out of 58 included reviews, 32 contained some cost/utilization outcomes and of these 741,57,59,61,68,73,75 focused on these outcomes exclusively. In the remaining 25 reviews, clinical outcomes and cost or utilization were included. 38,42,44-46,49,50,52,54-56,58,65,66,77- 81,83,84,86,87,89,94

In general, the results reported for clinical outcomes were more extensive and the syntheses more sophisticated than those reported for costs or utilization. For example, there were fewer meta-analyses of utilization than of clinical outcomes, and there were very few true cost- effectiveness or cost-benefit analyses identified in the included reviews. It is also important to note that costs are perhaps more affected by the organization of health services, coverage policies, and health policy in general than clinical outcomes. For this reason, it is necessary to underscore that the studies included in currently available reviews were not conducted under new models of care. In addition, not all of these studies were conducted in the United States and costs may be very different in different health care systems (e.g., 2 studies of telehealth for gastrointestinal care were conducted in Sweden, and one study that focused on travel costs was conducted in Newfoundland).

Figure 11 is a variation of Figure 10. For this version we re-reviewed the 32 systematic reviews that included any information on cost, cost-effectiveness, or health services utilization. Each review is represented by a bubble placed in the grid consisting of rows for telehealth functions and columns for clinical focus areas. The shading of the bubble represents the overall conclusion: whether for these outcomes the research suggests that telehealth provides a benefit, a potential benefit, is inconclusive, or provides no benefit. For this figure, benefit is defined as cost savings or reduction in health services utilization. For reviews that included both clinical outcomes and cost/utilization, only the cost/utilizations outcomes are included here.

Telehealth functions and clinical areas represented in the literature overall (including clinical as well as cost outcomes) are not all represented when we limit our focus to costs and utilization. None of the reviews contained cost or utilization outcomes for telementoring, and behavioral health is less well represented (1 review included costs/utilization out of 6 reviews included). In general, the cost/utilization results were less positive, in that they reported less benefit. Four reviews (13%) concluded telehealth provides benefit in terms of reduced costs or utilization, 11 (34%) potential benefit, 10 (31%) were inconclusive, and 7 (22%) found no benefit or increases in cost or utilization.

Table 7 contains a row for each included systematic review that included reports on cost or utilization outcomes and provides the key findings cited in the reviews. Most of these finding are not the result of complex, sophisticated, or even comprehensive analyses. A few meta-analyses on utilization such as hospital admissions and emergency department visits were available.

Furthermore, very few studies considered the overall cost-impact or cost-effectiveness of an intervention; rather they documented individual costs or resource use measures taken in isolation. Comprehensive cost-analyses are needed to understand the full implications of telehealth in various situations. Several of the authors of the reviews underscored that cost information was incomplete or inconsistently reported.

We re-examined our search of PROSPERO, a database of systematic review protocols for reviews in progress. From this we identified four pending reviews. Two planned to review the evidence for specific uses; costs of telehealth in home care and cost effectiveness of teleconsultations for patients in rural areas. The other two reviews were more general, examining the impact of mHealth and cost-effectiveness of health information technologies; this will only be useful for telehealth policy decisions if the reviews create subsets of health information technologies or include costs in their assessment of impact.

This initial review of information complied to create an evidence map suggests that information on costs is limited and costs and utilization may be an appropriate topic for additional research. While there may be primary studies that could be synthesized in a new review, there will be applicability challenges because current pending policy decisions are likely concerned with newer, integrated models of care, and existing research is likely to be based predominately on experiences in fee-for-service and nonintegrated-care organizations. More primary research is needed about how telehealth impacts costs and utilization of health services, although restrictions on funding cost-effective research may be a barrier to research that could address these current policy and practice questions.

Figure 11

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 7

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 7 part 2

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 7 part 3

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 7 part 4

 

 

 

 

 

 

 

 

 

 

 

 

 

Telehealth and New Models of Payment and Service Delivery

A key policy consideration is how telehealth might figure into new service delivery and payment models. Initiatives such as value-based purchasing and Accountable Care Organizations have been designed to create incentives for care that is high quality, accessible, and lower cost.101 Indeed, the US Centers for Medicare and Medicaid Services has a stated goal of increasing Medicare and Medicaid reimbursement to value-based models that share risk with provider organizations and give incentives for more coordinated and efficient care.102 The incentives are to improve or meet targets for performance on multiple measures. For example in the Medicare hospital value-based purchasing program, payments are adjusted based how a hospital scores on several measures in four domains (clinical processes, patient experiences, outcomes, and efficiency).103

Many of the evaluations of telehealth we identified considered a variety of outcomes, including clinical outcomes and health services utilization (e.g., hospitalizations and urgent visits) and costs. While most studies did include more than a single outcome, none purposefully examined the impact of telehealth on all the domains or the groups of measures used in these new models. This is understandable, as the widespread use of these models of reimbursement is relatively new to health care in the United States, and studies of telehealth have not yet been able to assess the contribution of telehealth to value-based models.

While it may be possible to make some inferences with regards to value-based care across studies, doing so would require re-examining the literature and organizing a review around groups of measures similar to those used in these new models. However, this approach is unlikely to identify studies where the suite of measures is used within an organization, making it difficult to determine how telehealth and performance measures interact across these domains.

Understanding this would require additional primary research that evaluates telehealth on all the relevant domains, though reviews of existing evidence could be used to inform the development of a demonstration or evaluation by suggesting which combinations of telehealth technologies and functions combined with specific patient populations should be the focus of larger studies.

Carrying out telehealth research under models of value-based care presents an important opportunity for future work, as any intervention or innovation that delivers care in more coordinated and efficient ways could be of great benefit to organizations entering into shared- risk models. For example, the processes and outcomes related to managing chronic disease could potentially be enhanced by some of the beneficial aspects of telehealth identified in the systematic reviews described above. Research would need to go beyond clinical factors and focus on delivering benefits not only from an individual-patient perspective, but from a population health management perspective as well.

Discussion

Summary and Implications

This technical brief was designed to characterize the existing systematic reviews available to inform decisions about telehealth, using an evidence map approach. Various stakeholders, including

federal and state policymakers, health care provider organizations, insurers and payers, are faced with making decisions related to supporting, implementing, and paying for telehealth. The literature, specifically in the form of systematic reviews is helpful and important to the extent that research can be used as evidence to support these decisions. When the evidence either does not exist or exists in a form that is not useful, additional work is required. An evidence map is a preliminary step in a multi-step process that can be used to identify relevant evidence and to initiate the process of developing a research agenda that can address remaining gaps.

This broad overview focuses on 58 systematic reviews that evaluate studies of telehealth, defined as the use of technology in interactions, over time or space, between providers and patients or between two providers. We grouped these results by clinical focus areas (a hybrid of conditions, body systems, and type of health care), and developed an approach to assessing the volume of the literature in terms of number of unique studies and the number of patients in these studies. We also weighted the conclusion (i.e., whether the systematic review concluded that telehealth provided a benefit) in order to provide a relative estimate of the benefit across clinical areas.

Creating bubble and intersection plots allowed us to identify areas where significant evidence exists supporting the benefits of telehealth (e.g., chronic conditions, diabetes, cardiovascular disease and behavioral health), where the evidence is more moderate (e.g., physical rehabilitation), and areas with a smaller evidence base and less positive conclusions (e.g., ICU, surgery support, and preterm birth).

We categorized abstracts of reviews in progress, reviews that mixed different interventions or did not consider the quality of the included studies, and conducted searches for primary studies in order to determine if more evidence was available for either the identified clinical focus areas with few included reviews or for areas that telehealth is likely being used or considered that were not covered by the included reviews. We identified protocols for 82 reviews in progress that would likely both increase the evidence for clinical areas we identified and add to additional topics such as maternal health and pediatrics. We also considered the reviews we excluded as indicators of the existence of primary studies that could be reanalyzed. Clinical areas not covered by our included reviews but included in reviews in process or excluded reviews were cancer, chronic pain, autism, and pregnancy.

Based on a combination of these assessments including the first bubble plot, we identified three areas where telehealth is likely being studied that were no covered by our included reviews and we conducted a search and scan of research primary literature. We identified primary studies in maternal health, in complex pediatrics, and in triage that evaluated the effectiveness of telehealth.

In a second plot, we replicated the bubble plot and organized the reviews by the health care function for which telehealth was used. In this analysis the largest number of studies and largest number of patient participants involved telehealth for communication and counseling followed by remote patient monitoring and multiple functions. A smaller evidence base of fewer studies and patients was available for the use of telehealth in psychotherapy and consultation.

Our third plot depicts the evidence at the intersection of clinical focus and function, creating smaller subgroups that we refer to as topics. This plot (Figure 10) showed that the systematic reviews that met the inclusion criteria were concentrated in communication/counseling and monitoring/management functions across all clinical categories. Cardiovascular disease/remote patient monitoring had the most evidence followed by psychotherapy and behavioral health.

We described the interventions and results in the reviews about telehealth for diabetes and communication in the results section as an example and we looked more broadly at applications for chronic diseases and telehealth use for remote patient monitoring. We also examined the impact of telehealth on service utilization and costs. Next we drew on our assessment of included and excluded reviews and our primary literature searches as well as the lack of results in some areas to develop recommendations related to the next steps in developing a research agenda and evidence base for telehealth.

Considering all the plots and additional analysis, we have created three categories for our overall assessment. These are defined as follows:

Category A: Topics that have a body of evidence in the form of several systematic reviews that according to our assessment can be used to inform decisions. For these topics there is a sizable quantity of evidence and some consistency in the conclusions. While there is always more to be learned and this research may not answer every specific question, for these topics it seems unlikely that in the near future new studies would overturn the conclusions supporting the effectiveness of telehealth.

Category B: Topics that would benefit from new or additional systematic reviews. For these topics, there appears to be enough primary studies to constitute a body of evidence, based on our assessment of excluded reviews, reviews in progress, and primary studies. However, these primary studies have not been a) synthesized in systematic reviews to date, but were identified in exploratory searches of primary literature; b) were included in reviews that did not meet the inclusion criteria (e.g. mixed interventions without separating results or did not consider the quality of the included studied); or c) were included in reviews that were unable to draw conclusions.

Category C: Topics with few primary studies completed to date and are less likely to constitute a body of evidence that could support policy decisions. Systematic reviews in these areas would risk being small and inconclusive until more primary research is done. Because this category is defined by a lack of identified evidence or evidence that is very disparate it is possible that the searches were not comprehensive or that research is constrained in these areas for good reasons. Topics in this category need to be assessed in term of importance to the field and potential reasons for the difficulty in locating or conducting primary studies (e.g., methodological or practical barriers to study).

In Table 8 below we sorted several topics into the three categories described above and provided a brief rationale for the placement in the assigned category.

Table 8

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Limitations of Evidence Maps

Evidence maps are exercises in abstraction and require a reductionist approach to information. Their purpose is to provide a view that combines a few selected variables in a way that increases understanding, but does not provide comprehensive review of the topic. Maps of a geographic area usually provide more information on a selected type of variable, such as natural features (e.g., rivers, mountains, or elevation) or manmade variables (e.g., roads, city, county, or state boundaries). Similarly, evidence maps must focus on a limited number of characteristics. These characteristics then have to be standardized and simplified in a way that allows them to be presented simultaneously. By definition, detail is reduced. Ideally the ability to identify patterns and relationships is worth the amount of detail lost and helps focus attention on which details should be re-examined using a different methodology. However, which characteristics are chosen and how data are simplified can also potentially mislead or at least not respond to the questions the analyses are designed to answer. Other than clarifying the purpose, soliciting feedback, and refining the approach, there is no way to avoid this limitation. There is no such thing as a “correct” or “definitive” evidence map. The best that can be achieved is that a map serves a useful purpose.

Additionally, like a road map, these plots provide information on key variables that help plan a route, but they do not select the route for the user. Interpretations of where the evidence is adequate and where more is needed that can vary based on perspectives and priorities of the user. Stakeholders interested in different aspects of telehealth maybe more interested in some specific uses over others. While one map cannot address all possible goals and priorities, the plots and the data in this report and the appendixes are provided in order to facilitate other considerations and interpretations.

Limitations of the Literature

The key limitations of the literature are related to both the nature of telehealth and the current state of systematic reviews.

Telehealth is a term that has been broadly applied to a range of applications of technology in health and health care. Using one term to describe everything from generic reminders sent to a cell phone, to the use of video for psychotherapy, to a complex system that allows a physician in another location to participate in a robotic surgery remotely is problematic for many reasons. As we found, the inclusiveness of the term telehealth can make searching literature and identifying relevant studies challenging and time consuming. Perhaps more importantly, such broad application of the term increases the chance that a synthesis could make comparisons of or summaries across, very different interventions that perhaps should not be combined or compared.

Knowing the diversity that exists under the label “telehealth,” when assessing evidence we want details that help create meaningful subgroups or identify trends. However, as is often the case with complex interventions, details about the actual intervention and its implementation are often under described in primary research and/or not selected for reporting in systematic reviews. In the literature on telehealth several variables were often not reported (e.g., the studies we identified did not discuss the frequencies or the intensity of telehealth use) reported inconsistently (e.g., the particulars of use in different settings such as rural verses urban health systems) or ambiguous (e.g., the lack of clarity and readers left to assume whether telehealth was replacing or augmenting in-person care).

Systematic reviews should identify, describe, and synthesize individual studies so that the collective results are accessible and more useful. Another limitation is that not all reviews achieve this. Many reviews we examined did not go beyond cataloging the research and providing varying levels of descriptions. Some did not examine the quality of the included studies or discuss how the quality of studies should affect the interpretation of the findings.Others included very different interventions and did not separate the findings by type of intervention. The result is often either the lack of a conclusion or a tentative conclusion that is not at the confidence level required to support current practice or policy decisions about the use of specific telehealth interventions. While it is possible that these reviews were designed and executed for other reasons, there is a risk that not only will their use be limited, but that reviews become viewed by policymakers and decisionmakers as an academic exercise rather than useful tools. Another challenge with systematic reviews in this literature is the uneven quality of studies within the reviews. Often, lower-quality studies are less likely to find an effect even where one exists such that the results of high-quality studies may be muted by lower-quality studies, especially when the latter are more numerous. This is particularly problematic when results cannot be analyzed by meta-analysis, where heterogeneity may be identified, when the stability of findings cannot be tested through sensitivity analysis, or when reviewers do not include strength of evidence assessments that include consideration of the quality of individual studies.

Other Summaries of Telehealth Research

As part of our searching and triage, we looked for other efforts to summarize, describe, or analyze the evidence base for telehealth. In reviewing both the published literature as well as grey literature and related Web sites (e.g., U.S. government agency sites) we did not identify any other efforts to map the literature on telehealth despite the fact that there have been numerous systematic reviews. We have found no other efforts to summarize what is available to support decisions and to identify areas lacking evidence that parallels our mapping.

We did identify several “reviews of reviews”, that is syntheses of systematic reviews and meta-analyses. These are sometimes called umbrella reviews. We found seven of these umbrella reviews published since 2012.104-110 Some of these asked research questions that differed from the focus of this literature map. One addressed factors that promote or inhibit the implementation of any e-health system and included 37 papers published between 1995 and 2009 that were not only systematic reviews, but also narrative reviews and meta-ethnographies.108 Others defined telehealth differently than it has been defined in this literature map. A meta-review of the use of mobile phones and text messaging for self-management interventions for chronic conditions included 11 systematic reviews,107 but many of the interventions were unidirectional or not personalized (e.g., appointment reminders, general encouragement). Four of the reviews did not assess the quality of the included studies. Based on the four highest-quality reviews, the authors concluded that these technologies show promise, but that “more high-quality studies are needed to judge the long-term benefits.”

Other umbrella reviews focused on a particular technology or condition and can, in some cases, be compared to subsets of our map. A summary of 29 systematic reviews on digital self- management support for adults and children with asthma reported evidence of some beneficial effects on some outcomes but emphasized that the characteristics of the patient population and the interventions themselves were so poorly described, and so few studies included economic analysis, that understanding the potential reach and uptake was difficult.110 Another targeted study summarized the findings of 10 systematic reviews of telepsychiatry.104 While the emphasis for the review was on the feasibility of use in resource constrained environments (such as South Africa, the country of the authors), the studies in the reviews were conducted in United States, Canada, Europe, Australia, Japan, and Hong Kong. These researchers reported the reviews were of acceptable quality, but that common deficiencies were lack of grading the strength of evidence or linking the quality of the included studies to the conclusions. Nevertheless, this review of reviews concluded that telepsychiatry is as effective as face-to-face treatment and testing and does improve symptoms. Similarly, a meta-review of systematic reviews of remote monitoring for heart failure included 17 systematic reviews that summarized studies of implanted devices as well as telemonitoring. The authors rated seven of these as high-quality reviews, combined the results across these reviews, and concluded “that remote monitoring improves mortality and quality of life as well as reduces hospitalizations and, as a consequence, health care costs.”105 However, the authors noted that the evidence bases did not provide enough detail about how to best target remote monitoring to those most likely to benefit.

Two large reviews of reviews overlapped with many of the objectives of this literature map; however, they differed in both content and conclusions. Elbert 2014106 updated a 2010 review of reviews by Ekeland111 by summarizing reviews published 2009−2012. Their definition of eHealth was narrower than the definition of telehealth used for this map (i.e., excluded interventions that are not home based like tele-ICU) and they excluded studies of mental health and behavior change. They included 31 reviews; 7 concluded that eHealth is effective and/or cost effective and 13 concluded the evidence was promising, while 11 reported limited or inconsistent proof. The authors of this review concluded that while larger more rigorous studies could provide more definite proof of effectiveness, the evidence has been and continues to be promising        and efforts could be better focused on evaluating the implementation of eHealth interventions that have been shown to be effective. McLean 2013 identified 80 systematic                      reviews of telehealth (using a definition similar to ours, but including phone only support as well) published between 1997 and November 2011 and summarized these as part of an evaluation for the National Health Service in the United Kingdom.109 They concluded “While reported improvements in surrogate clinical endpoints and hospitalizations are encouraging, the evidence overall remains equivocal.”109 The author attributed this to the failure of large trials to show benefit, the focus of research on evaluations of smaller projects that are not scaled in order to assess long term impact, failure to include patient and broader societal perspectives, and flawed economic analysis that did not consider downstream effects on the distribution of services.

While our literature map shares some objectives and conclusions with these reviews of reviews, by definition the scope, analysis and presentation are different. Our literature map scope is broader than many reviews of reviews, but the key difference is that the analysis is more descriptive and uses figures and tables to allow comparisons across subtopics within the literature on telehealth, in order to facilitate identification of topics for which the available evidence can support decision making.

Future Research

We identified groups of studies in clinical focus areas that have been included in reviews, but in ways that have made them less useful for decisionmaking. Specifically in many cases reviews have identified and grouped existing research together in ways that do correspond to current policy or practice issues. This evidence could be reanalyzed in order to support decisions.

Additionally, there are pending reviews and individual studies on several topics that can add to the evidence base. In a field with such a wealth of information, the key will be to help decisionmakers identify what important questions are truly still outstanding and develop a research agenda for both systematic reviews and primary studies that will answer these questions. This evidence map provides a foundation for that effort.

Another less traditional approach would involve a different kind of “mapping.” It would involve outlining potential benefits of telehealth, mapping these to the goals and/or measures in health reform programs, such as value-based purchasing. Then, based on this, a research agenda for both primary research and systematic reviews could be developed that explicitly addresses the questions about what role telehealth can and should play in future reforms in health care.

As evidenced in this literature map, telehealth is used for a variety of functions within multiple clinical focus areas. Use and evaluation of telehealth is also affected by characteristics of the care setting and circumstances surrounding individual patients, as depicted in Figure 12. The studies that make up the existing knowledge base for telehealth typically focus on a limited number of research questions within a single context. A narrowed research focus and tightly defined research population can reduce confounders, but it also necessitates that a large number of studies must be conducted to provide insight into the complex system of care that an individual patient, or a patient population experiences. The insights provided by narrowly defined research studies are analogous to looking at a landscape through a soda straw. The viewer can become overly confident, or miss key insights that would be more visible in a larger context. As the industry shifts toward value-based care and personalized care, the challenge will be for future telehealth research to evaluate the contributions of telehealth across care contexts. This includes within increasingly integrated care delivery models where telehealth may be but one of many modes of care that patient may simultaneously experience and where experience may vary across patients. A telehealth research network recently published a 12-point global research agenda for telehealth, in which they assert the need to “incorporate health care parameters across mediated and traditional modes of care for the benefit of providers, companies, policymakers, and the international research community.”25

Figure 12

 

 

 

 

 

 

 

 

The implementation of the triple aim, concurrent with move to value-based care, has elevated the cost of care and the patient experience to equal footing with clinical outcomes of care. This focus has implications for telehealth in that future research should help providers and health systems differentiate the value of telehealth services as an addition to traditional in-person care and the value of telehealth as a replacement for in-person care. Increasingly, decisionmakers will need evidence-based practices and guidelines to facilitate decisions regarding when to employ telehealth services. Guidelines will need to consider the context of care as well as the impact of telehealth services on the cost, quality and experience of care.

Future telehealth research will also need to look at multiple time horizons. Telehealth benefits may be seen at the time of the initial service, or over longer periods of time. As the Center for Medicare and Medicaid Services tests new payment and service delivery models, it will be important to ensure that clinical and administrative records reflect which parts of bundled services were delivered in-person or via telehealth to support longitudinal studies of the value of telehealth services within new service delivery models.

Conclusion

Our searches confirmed that there is a large, broad evidence base about the effectiveness of telehealth, including over 200 systematic reviews and hundreds of primary studies published since 2006. Although we found that many reviews are not structured or conducted in a way that would support current decisions related to telehealth, we did identify a substantial amount of evidence—58 systematic reviews that covered several important clinical focus areas and met our inclusion criteria. The largest volume of research reported that telehealth interventions have produced positive results when used in the clinical areas of chronic conditions and behavioral health and when telehealth is used for providing communication/counseling and monitoring/management. Considering both clinical areas and the functions of telehealth allowed us to create more specific subgroups and look at the variation and consistency within these as well.

Based on our analysis, advancing the use telehealth maybe best served by two different research agendas. First, as this evidence map has demonstrated, there is substantial evidence supporting subsets of telehealth uses, such as remote patient monitoring for chronic conditions. In these cases, the focus should shift from effectiveness studies to broader implementation efforts and studies of barriers to spread. Two distinct approaches are needed to address gaps in the evidence with future research. One approach to future research is to continue to elaborate on our findings as well as develop additional research in a variety of clinical topics and for different health care functions for telehealth. In areas where we did not find sufficient synthesized research, such as telehealth for consultation, in ICU and surgery, and in maternal and child health new systematic reviews may be able to organize primary research (some of which is new and some of which has been included in reviews in the past) into better reviews designed to address practice and policy considerations related to these issues. Additionally, there are clinical areas and roles for telehealth that do not yet have a sufficient evidence base to support important decisions and in these cases, more primary research is needed rather than more systematic reviews. We identified triage in urgent/primary care, management of serious pediatric conditions and the integration of behavioral and physical health as three potential topics for more primary research.

The agenda is based on expanding the orientation of future work to include new organizational and payment models. Going forward, research should be conducted in emerging models of care, particularly value-based models where use of telehealth may improve the ability to share risk and attain quality and related outcomes. These studies of telehealth should consider combinations of applications of telehealth and outcomes that are important in these new models and that evaluate the specific contribution telehealth can make in these contexts.

A recent comment published in the journal Nature reported on the use of evidence maps in conservation and development. The authors point out that failure to evaluate the existing evidence can result in unnecessary harm, that relative costs and benefits may be overlooked, and people assume evidence exists that does not. They propose that a “atlas” of evidence maps that are maintained and updated could transform work in their field.11 While this single map may not rise to the level of an atlas, our experience suggests that in a field with a large diffuse evidence base, maps are useful. This map confirms that there are important areas within telehealth with substantial evidence that can support broader implementation and spread. The fact that there are other areas where evidence is still minimal should not delay building on what we do know.

References

  1. Hetrick SE, Parker AG, Callahan P, et Evidence mapping: illustrating an emerging methodology to improve evidence-based practice in youth mental health. J Eval Clin Pract. 2010 Dec;16(6):1025-30. doi: 10.1111/j.1365-2753.2008.01112.x.  PMID: 20337833.
  2. Bates S, Coren Systematic Map Report 1: The extent and impact of parental mental health problems on families and the acceptability, accessibility and effectiveness of interventions. London: Social Care Institute for Excellence; 2006. http://www.scie.org.uk/publications/map/ma p01.pdf. Accessed June 17, 2016.
  3. Carr S, Clapton Systematic Map Report 2: The recovery approach in community-based vocation and training adult mental health day services. London: Social Care Institute for Excellence; 2007. http://www.scie.org.uk/publications/map/ma p02.pdf?res=true. Accessed June 17, 2016.
  4. Sharif N, Brown W, Rutter Systematic Map Report 3: The extent and impact of depression on BME older people and the acceptability, accessability and effectiveness of social care provision. London: Social Care Institute for Excellence; 2008. http://www.scie.org.uk/publications/map/ma p03.pdf. Accessed June 17, 2016.
  5. Coeytaux RR, McDuffie J, Goode A, et VA Evidence-based Synthesis Program Reports. Evidence Map of Yoga for High- Impact Conditions Affecting Veterans. Washington, DC: Department of Veterans Affairs (US); 2014. PMID: 25254284.
  6. Hempel S, Taylor SL, Solloway MR, et Evidence Map of Acupuncture. VA Evidence-based Synthesis Program Reports. Washington, DC: Department of Veterans Affairs (US); 2014. PMID: 24575449.
  7. Hempel S, Taylor SL, Marshall NJ, et Evidence Map of Mindfulness. VA Evidence-based Synthesis Program Reports. Washington, DC: Department of Veterans Affairs (US); 2014. PMID: 25577939.
  1. Hempel S, Taylor SL, Solloway MR, et VA Evidence-based Synthesis Program Reports. Evidence Map of Tai Chi. Washington, DC: Department of Veterans Affairs (US); 2014.
  2. Wang DD, Shams-White M, Bright OJ, et Creating a literature database of low- calorie sweeteners and health studies: evidence mapping. BMC Med Res Methodol. 2016;16(1):1. doi: 10.1186/s12874-015-0105-z. PMID: 26728979.
  3. Althuis MD, Weed Evidence mapping: methodologic foundations and application to intervention and observational research on sugar-sweetened beverages and health outcomes. Am J Clin Nutr. 2013 Sep;98(3):755-68. doi: 10.3945/ajcn.113.058917. PMID: 23824722.
  4. McKinnon MC, Cheng SH, Garside R, et Sustainability: Map the evidence. Nature. 2015 Dec 10;528(7581):185-7. doi: 10.1038/528185a. PMID: 26659166.
  5. Zundel Telemedicine: history, applications, and impact on librarianship. Bull Med Libr Assoc. 1996;84(1):71-9. PMID: 8938332.
  6. Telehealth. 2015. http://www.hrsa.gov/ruralhealth/telehealth/. Accessed June 17, 2016.
  7. gov. What is telehealth? How is telehealth different from telemedicine? http://www.healthit.gov/providers- professionals/faqs/what-telehealth-how- telehealth-different-telemedicine. Accessed March 9, 2015.
  8. van Dyk A review of telehealth service implementation frameworks. Int J Environ Res Public Health. 2014 Feb;11(2):1279-98. doi: 10.3390/ijerph110201279. PMID: 24464237.
  9. Lustig The Role of Telehealth in an Evolving Health Care Environment – Workshop Summary. Washington, DC: Institute of Medicine; 2012.
  10. Nelson B, Thune Letter to Richard Kronick, Agency for Healthcare Research and Quality. December 30, 2014.
  1. Multistakeholder Letter to Senate Committee on 2015. http://www.tiaonline.org/sites/default/files/p ages/Multistakeholder%20Letter%20to%20 Senate%20Ageing%20re%20AHRQ%20Lit %20Review%20120514.pdf. Accessed August 28, 2015.
  1. Bashshur Guest editorial: compelling issues in telemedicine. Telemed J E Health. 2013 May;19(5):330-2. doi: 10.1089/tmj.2013.9998. Epub 2013 Feb 5. PMID: 23384331
  2. Dixon B, Hook J, McGowan Using Telehealth to Improve Quality and Safety: Findings from the AHRQ Health IT Portfolio (Prepared by the AHRQ National Resource Center for Health IT under Contract No. 290-04-0016). AHRQ Publication No. 09-0012-EF. Rockville, MD: Agency for Healthcare Research and Quality; 2008.
  3. Roundtable: Harnessing the Power of Telehealth: Promises and Challenges? Testimony from Washington, DC: United States Senate Special Committee on Aging; 2014. http://www.aging.senate.gov/hearings/round table-harnessing-the-power-of-telehealth- promises-and-challenges Accessed August 25, 2015.
  4. The Role of Telehealth in an Evolving Health Care Environment: Workshop Washington, DC: The National Academies Press; 2012.
  5. Adler-Milstein J, Kvedar J, Bates Telehealth among US hospitals: several factors, including state reimbursement and licensure policies, influence adoption. Health Aff (Millwood). 2014 Feb;33(2):207- 15. doi: 10.1377/hlthaff.2013.1054. PMID: 24493762.
  6. Broderick A, Lindeman Scaling Telehealth Programs: Lessons from Early Adopters. New York, NY: Commonwealth Fund; 2013. http://www.commonwealthfund.org/Publicat ions/Case-Studies/2013/Jan/Telehealth- Synthesis.aspx. Accessed August 24, 2015.
  1. Dinesen B, Nonnecke B, Lindeman D, et Personalized telehealth in the future: a global research agenda. J Med Internet Res. 2016;18(3):e53. doi: 10.2196/jmir.5257. PMID: 26932229.
  2. Hersh WR, Hickam DH, Severance SM, et Diagnosis, access and outcomes: update of a systematic review of telemedicine services. J Telemed Telecare. 2006;12 Suppl 2:S3-31. doi: 10.1258/135763306778393117. PMID: 16989671.
  3. Medicare Payment Advisory Public Meeting. March 3, 2016. http://www.medpac.gov/documents/0316_m eeting_transcript.pdf?sfvrsn=0. Accessed June 17, 2016.
  4. Schwamm Telehealth: seven strategies to successfully implement disruptive technology and transform health care. Health Aff (Millwood). 2014 Feb;33(2):200- 6. doi: 10.1377/hlthaff.2013.1021. PMID: 24493761.
  5. Kvedar J, Coye MJ, Everett Connected health: a review of technologies and strategies to improve patient care with telemedicine and telehealth. Health Aff (Millwood). 2014 Feb;33(2):194-9. doi: 10.1377/hlthaff.2013.0992. PMID: 24493760.
  6. Clapton J, Rutter D, Sharif SCIE Systematic Mapping Guidance. London: Social Care Institute for Excellence; 2009.
  7. Bragge P, Clavisi O, Turner T, et The Global Evidence Mapping Initiative: scoping research in broad topic areas. BMC Med Res Methodol. 2011;11:92. doi: 10.1186/1471-2288-11-92. PMID: 21682870.
  8. Snilstveit B, Vojtkova M, Bhavsar A, et Evidence gap maps–a tool for promoting evidence-informed policy and prioritizing future research. World Bank Policy Research Working Paper, Report Number WPS6725. Washington, DC: The World Bank Group; 2013.
  9. Miake-Lye IM, Hempel S, Shanman R, et What is an evidence map? A systematic review of published evidence maps and their definitions, methods, and products. Syst Rev. 2016;5(1):28. doi: 10.1186/s13643- 016-0204-x. PMID: 26864942.
  1. National Research Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press; 2011.
  2. Agency for Healthcare Research and Methods Guide for Effectiveness and Comparative Effectiveness Reviews. AHRQ Publication No. 10(13)-EHC063-EF. Rockville, MD: Agency for Healthcare Research and Quality; 2014.
  3. Nelson Systematic Reviews to Answer Health Care Questions. Philadelphia, PA: Wolters Kluwer; 2014.
  4. Shea BJ, Hamel C, Wells GA, et AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol. 2009 Oct;62(10):1013-20. doi: http://dx.doi.org/10.1016/j.jclinepi.2008.10. 009. PMID: 19230606.
  5. Ammenwerth E, Schnell-Inderst P, Hoerbst
    1. The impact of electronic patient portals on patient care: a systematic review of controlled J Med Internet Res. 2012;14(6):e162. doi: http://dx.doi.org/10.2196/jmir.2238.   PMID: 23183044.
  6. Antonacci DJ, Bloch RM, Saeed SA, et Empirical evidence on the use and effectiveness of telepsychiatry via videoconferencing: implications for forensic and correctional psychiatry. Behav Sci Law. 2008;26(3):253-69. doi: 10.1002/bsl.812. PMID: 18548519.
  7. Antoniou SA, Antoniou GA, Franzen J, et A comprehensive review of telementoring applications in laparoscopic general surgery. Surg Endosc. 2012 Aug;26(8):2111-6. doi: http://dx.doi.org/10.1007/s00464-012-2175- x. PMID: 22350150.
  8. Chaudhry SI, Phillips CO, Stewart SS, et Telemonitoring for patients with chronic heart failure: a systematic review. J Card Fail. 2007 Feb;13(1):56-62. PMID: 17339004.
  1. Clarke M, Shah A, Sharma Systematic review of studies on telemonitoring of patients with congestive heart failure: a meta-analysis. J Telemed Telecare. 2011;17(1):7-14. doi: 10.1258/jtt.2010.100113. PMID: 21097564.
  2. Connelly J, Kirk A, Masthoff J, et The use of technology to promote physical activity in Type 2 diabetes management: a systematic review. Diabet Med. 2013 Dec;30(12):1420-32. doi: http://dx.doi.org/10.1111/dme.12289. PMID: 23870009.
  3. Cox NS, Alison JA, Rasekaba T, et Telehealth in cystic fibrosis: a systematic review. J Telemed Telecare. 2012 Mar;18(2):72-8. doi: http://dx.doi.org/10.1258/jtt.2011.110705. PMID: 22198961.
  4. Dang S, Dimmick S, Kelkar Evaluating the evidence base for the use of home telehealth remote monitoring in elderly with heart failure. Telemed J E Health. 2009 Oct;15(8):783-96. doi: 10.1089/tmj.2009.0028. PMID: 19831704.
  5. de Jong CC, Ros WJ, Schrijvers The effects on health behavior and health outcomes of Internet-based asynchronous communication between health providers and patients with a chronic condition: a systematic review. J Med Internet Res. 2014;16(1):e19. doi: http://dx.doi.org/10.2196/jmir.3000.   PMID: 24434570.
  6. de Waure C, Cadeddu C, Gualano MR, et Telemedicine for the reduction of myocardial infarction mortality: a systematic review and a meta-analysis of published studies. Telemed J E Health. 2012 Jun;18(5):323-8. doi: http://dx.doi.org/10.1089/tmj.2011.0158. PMID: 22468983.
  7. Eccleston C, Fisher E, Craig L, et Psychological therapies (Internet-delivered) for the management of chronic pain in adults. Cochrane Database Syst Rev. 2014;2:CD010152. doi: http://dx.doi.org/10.1002/14651858.CD0101 52.pub2. PMID: 24574082.
  1. Eland-de Kok P, van Os-Medendorp H, Vergouwe-Meijer A, et A systematic review of the effects of e-health on chronically ill patients. J Clin Nurs. 2011 Nov;20(21-22):2997-3010. doi: http://dx.doi.org/10.1111/j.1365- 2702.2011.03743.x. PMID: 21707807.
  2. Gaikwad R, Warren The role of home- based information and communications technology interventions in chronic disease management: a systematic literature review. Health Informatics J. 2009 Jun;15(2):122-
  3. doi: http://dx.doi.org/10.1177/146045820910297 3. PMID: 19474225.
  4. Gainsbury S, Blaszczynski A systematic review of Internet-based therapy for the treatment of addictions. Clin Psychol Rev. 2011 Apr;31(3):490-8. doi: http://dx.doi.org/10.1016/j.cpr.2010.11.007. PMID: 21146272.
  5. Garcia-Lizana F, Sarria-Santamera New technologies for chronic disease management and control: a systematic review. J Telemed Telecare. 2007;13(2):62- 8. PMID: 17359568.
  6. Holtz B, Lauckner Diabetes management via mobile phones: a systematic review. Telemed J EHealth. 2012 Apr;18(3):175-84. doi: http://dx.doi.org/10.1089/tmj.2011.0119. PMID: 22356525.
  7. Hailey D, Roine R, Ohinmaa A, et Evidence on the effectiveness of telerehabilitation applications. Alberta, Canada: Institute of Health Economics; 2010. http://www.ihe.ca/publications/evidence- on-the-effectiveness-of-telerehabilitation- applications. Accessed June 17, 2016.
  8. Jaana M, Pare G, Sicotte Home telemonitoring for respiratory conditions: a systematic review. Am J Manag Care. 2009 May;15(5):313-20. PMID: 19435399.
  9. Johansen MA, Berntsen GKR, Schuster T, et Electronic symptom reporting between patient and provider for improved health care service quality: a systematic review of randomized controlled trials. part 2: methodological quality and effects. J Med Internet Res. 2012;14(5):e126. doi: http://dx.doi.org/10.2196/jmir.2216.  PMID: 23032363.
  1. Kairy D, Lehoux P, Vincent C, et A systematic review of clinical outcomes, clinical process, healthcare utilization and costs associated with telerehabilitation. Disabil Rehabil. 2009;31(6):427-47. doi: http://dx.doi.org/10.1080/096382808020625 53. PMID: 18720118.
  2. Kamei T, Yamamoto Y, Kajii F, et Systematic review and meta-analysis of studies involving telehome monitoring- based telenursing for patients with chronic obstructive pulmonary disease. Jpn J Nurs Sci. 2013 Dec;10(2):180-92. doi: http://dx.doi.org/10.1111/j.1742- 7924.2012.00228.x. PMID: 24373441.
  3. Knowles SR, Mikocka-Walus Utilization and efficacy of internet-based eHealth technology in gastroenterology: a systematic review. Scand J Gastroenterol. 2014 Apr;49(4):387-408. doi: http://dx.doi.org/10.3109/00365521.2013.86 5259. PMID: 24494974.
  4. Kodama S, Saito K, Tanaka S, et Effect of Web-based lifestyle modification on weight control: a meta-analysis. Int J Obesity. 2012 May;36(5):675-85. doi: http://dx.doi.org/10.1038/ijo.2011.121. PMID: 21694698.
  5. Kumar G, Falk DM, Bonello RS, et The costs of critical care telemedicine programs: a systematic review and analysis. Chest. 2013 Jan;143(1):19-29. PMID: 22797291.
  6. Laver KE, Schoene D, Crotty M, et Telerehabilitation services for stroke. Cochrane Database Syst Rev. 2013;12:CD010255. doi: http://dx.doi.org/10.1002/14651858.CD0102 55.pub2. PMID: 24338496.
  7. Liang X, Wang Q, Yang X, et Effect of mobile phone intervention for diabetes on glycaemic control: a meta-analysis. Diabet Med. 2011 Apr;28(4):455-63. doi: http://dx.doi.org/10.1111/j.1464- 5491.2010.03180.x. PMID: 21392066.
  8. Lustria MLA, Noar SM, Cortese J, et A meta-analysis of web-delivered tailored health behavior change interventions.[Erratum appears in J Health Commun. 2013;18(11):1397]. J Health Commun. 2013;18(9):1039-69. doi: http://dx.doi.org/10.1080/10810730.2013.76 8727. PMID: 23750972.
  1. Martin S, Sutcliffe P, Griffiths F, et Effectiveness and impact of networked communication interventions in young people with mental health conditions: a systematic review. Patient Educ Couns. 2011 Nov;85(2):e108-19. doi: http://dx.doi.org/10.1016/j.pec.2010.11.014. PMID: 21239133.
  2. McLean S, Nurmatov U, Liu JL, et Telehealthcare for chronic obstructive pulmonary disease. Cochrane Database Syst Rev. 2011(7):CD007718. doi: http://dx.doi.org/10.1002/14651858.CD0077 18.pub2. PMID: 21735417.
  3. Merriel SWD, Andrews V, Salisbury Telehealth interventions for primary prevention of cardiovascular disease: a systematic review and meta-analysis. Prev Med. 2014 Jul;64:88-95. doi: http://dx.doi.org/10.1016/j.ypmed.2014.04.0 01. PMID: 24726502.
  4. Peeters JM, Mistiaen P, Francke Costs and financial benefits of video communication compared to usual care at home: a systematic review. J Telemed & Telecare. 2011;17(8):403-11. doi: http://dx.doi.org/10.1258/jtt.2011.110405. PMID: 22025744.
  5. Ramadas A, Quek KF, Chan CKY, et Web-based interventions for the management of type 2 diabetes mellitus: a systematic review of recent evidence. Int J Med Inform. 2011 Jun;80(6):389-405. doi:  http://dx.doi.org/10.1016/j.ijmedinf.2011.02. 002. PMID: 21481632.
  6. Richards D, Richardson Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clin Psychol Rev. 2012 Jun;32(4):329-42. doi: http://dx.doi.org/10.1016/j.cpr.2012.02.004. PMID: 22466510.
  7. Rietdijk R, Togher L, Power Supporting family members of people with traumatic brain injury using telehealth: a systematic review. J Rehabil Med. 2012 Nov 5;44(11):913-21. doi: http://dx.doi.org/10.2340/16501977-1058. PMID: 23027316.
  1. Rooke S, Thorsteinsson E, Karpin A, et Computer-delivered interventions for alcohol and tobacco use: a meta-analysis. Addiction. 2010 Aug;105(8):1381-90. doi: http://dx.doi.org/10.1111/j.1360- 0443.2010.02975.x. PMID: 20528806.
  2. Seto Cost comparison between telemonitoring and usual care of heart failure: a systematic review. Telemed J E Health. 2008 Sep;14(7):679-86. doi: http://dx.doi.org/10.1089/tmj.2007.0114. PMID: 18817497.
  3. Steel K, Cox D, Garry Therapeutic videoconferencing interventions for the treatment of long-term conditions. J Telemed Telecare. 2011;17(3):109-17. doi: http://dx.doi.org/10.1258/jtt.2010.100318. PMID: 21339304.
  4. Tan K, Lai Telemedicine for the support of parents of high-risk newborn infants. Cochrane Database Syst Rev. 2012;6:CD006818. doi: http://dx.doi.org/10.1002/14651858.CD0068 18.pub2. PMID: 22696360.
  5. Tran K, Polisena J, Coyle D, et Home Telehealth for Chronic Disease Management. (Technology report number 113). Ottawa, Canada: Canadian Agency for Drugs and Technologies in Health; 2008. https://www.cadth.ca/media/pdf/H0475_Ho me_Telehealth_tr_e.pdf. Accessed June 17, 2016.
  6. Urquhart C, Currell R, Harlow F, et Home uterine monitoring for detecting preterm labour. Cochrane Database Syst Rev. 2015 (1). PMID: 25558862.
  7. Wade VA, Karnon J, Elshaug AG, et A systematic review of economic analyses of telehealth services using real time video communication. BMC Health Serv Res. 2010;10:233. doi: http://dx.doi.org/10.1186/1472-6963-10- 233. PMID: 20696073.
  8. Wallace DL, Hussain A, Khan N, et A systematic review of the evidence for telemedicine in burn care: with a UK perspective. Burns. 2012 Jun;38(4):465-80. doi: http://dx.doi.org/10.1016/j.burns.2011.09.02 4. PMID: 22078804.
  1. Zhai Y-k, Zhu W-j, Cai Y-l, et Clinical- and cost-effectiveness of telemedicine in type 2 diabetes mellitus: a systematic review and meta-analysis. Medicine. 2014 Dec;93(28):e312. doi: http://dx.doi.org/10.1097/MD.00000000000 00312. PMID: 25526482.
  2. Conway A, Inglis SC, Clark Effective technologies for noninvasive remote monitoring in heart failure. Telemed J E Health. 2014 Jun;20(6):531-8. doi: http://dx.doi.org/10.1089/tmj.2013.0267. PMID: 24731212.
  3. Clark RA, Conway A, Poulsen V, et Alternative models of cardiac rehabilitation: a systematic review. Eur J Prev Cardiol. 2015 Jan;22(1):35-74. doi: http://dx.doi.org/10.1177/204748731350109 3. PMID: 23943649.
  4. Cruz J, Brooks D, Marques Home telemonitoring effectiveness in COPD: a systematic review. Int J Clin Pract. 2014 Mar;68(3):369-78. doi: http://dx.doi.org/10.1111/ijcp.12345.   PMID: 24472009.
  5. Devi R, Singh SJ, Powell J, et Internet- based interventions for the secondary prevention of coronary heart disease. Cochrane Database Syst Rev. 2015(12). PMID: 26691216.
  6. dos Santos MT, Moura SC, Gomes LM, et Telehealth application on the rehabilitation of children and adolescents. Rev Paul Pediatr. 2014 Mar;32(1):136-43. PMID: 24676201.
  7. Flodgren G, Rachas A, Farmer AJ, et Interactive telemedicine: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2015(9). PMID: 26343551.
  8. Kotb A, Cameron C, Hsieh S, et Comparative effectiveness of different forms of telemedicine for individuals with heart failure (HF): a systematic review and network meta-analysis. PLoS ONE [Electronic Resource]. 2015;10(2):e0118681. doi:  http://dx.doi.org/10.1371/journal.pone.0118 681. PMID: 25714962.
  1. Moy MF, Ray A, Buckley Techniques of monitoring blood glucose during pregnancy for women with pre-existing diabetes. Cochrane Database Syst Rev. 2015(11) PMID: 24782359.
  2. Parthiban N, Esterman A, Mahajan R, et Remote Monitoring of Implantable Cardioverter-Defibrillators: A Systematic Review and Meta-Analysis of Clinical Outcomes. J Am Coll Cardiol. 2015 Jun 23;65(24):2591-600. doi: http://dx.doi.org/10.1016/j.jacc.2015.04.029. PMID: 25983009.
  3. Rooksby M, Elouafkaoui P, Humphris G, et Internet-assisted delivery of cognitive behavioural therapy (CBT) for childhood anxiety: systematic review and meta- analysis. J Anxiety Disord. 2015 Jan;29:83-
  4. doi: http://dx.doi.org/10.1016/j.janxdis.2014.11.0 06. PMID: 25527900.
  5. Saffari M, Ghanizadeh G, Koenig Health education via mobile text messaging for glycemic control in adults with type 2 diabetes: a systematic review and meta- analysis. Prim Care Diabetes. 2014 Dec;8(4):275-85. doi: http://dx.doi.org/10.1016/j.pcd.2014.03.004. PMID: 24793589.
  6. Toma T, Athanasiou T, Harling L, et Online social networking services in the management of patients with diabetes mellitus: systematic review and meta- analysis of randomised controlled trials. Diabetes Res Clin Pract. 2014 Nov;106(2):200-11. doi: http://dx.doi.org/10.1016/j.diabres.2014.06.0 08. PMID: 25043399.
  7. van Beugen S, Ferwerda M, Hoeve D, et Internet-based cognitive behavioral therapy for patients with chronic somatic conditions: a meta-analytic review. J Med Internet Res. 2014;16(3):e88. doi: http://dx.doi.org/10.2196/jmir.2777.   PMID: 24675372.
  8. Warshaw EM, Hillman YJ, Greer NL, et Teledermatology for diagnosis and management of skin conditions: a systematic review. J Am Acad Dermatol. 2011 Apr;64(4):759-72. doi: http://dx.doi.org/10.1016/j.jaad.2010.08.026. PMID: 21036419.
  1. American Telemedicine Standards and Guidelines Working Groups. Washington, DC: The American Telemedicine Association; 2012. http://www.americantelemed.org/resources/t elemedicine-practice-guidelines/standards- working-groups#.VeC0F_lVhHx. Accessed August 28, 2015.
  2. Center for Connected Health State Telehealth Laws and Medicaid Program Policies: A Comprehensive Scan of the 50 States and District of Columbia. Sacramento: Public Health Institute Center for Connected Health Policy; 2015. http://cchpca.org/sites/default/files/resources

/50%20State%20FINAL%20April%202016.

pdf. Accessed June 17, 2016.

  1. Center for Reviews and PROSPERO. 2015. http://www.crd.york.ac.uk/PROSPERO/. Accessed August 28, 2015.

Inglis SC, Clark RA, McAlister FA, et al. Structured telephone support or telemonitoring programmes for patients with chronic heart failure. Cochrane Database

99. Inglis SC, Clark RA, McAlister FA, et al. Structured telephone support or telemonitoring programmes for patients with chronic heart failure. Cochrane Database

100.  Bashshur RL, Shannon GW, Tejasvi T, et The Empirical Foundations of Teledermatology: A Review of the Research Evidence. Telemed J E Health. 2015 Sep 22doi: 10.1089/tmj.2015.0146.

101. Fisher ES, McClellan MB, Safran Building the path to accountable care. N Engl J Med. 2011 Dec 29;365(26):2445-7. doi: 10.1056/NEJMp1112442. PMID: 22204720.

102. Burwell Setting value-based payment goals–HHS efforts to improve U.S. health care. N Engl J Med. 2015 Mar 5;372(10):897-9. doi: 10.1056/NEJMp1500445. Epub 2015 Jan 26. PMID: 25622024.

105. Conway A, Inglis SC, Chang AM, et Not all systematic reviews are systematic: a meta-review of the quality of systematic reviews for non-invasive remote monitoring in heart failure. J Telemed Telecare. 2013 Sep;19(6):326-37. doi: http://dx.doi.org/10.1177/1357633X135034 27. PMID: 24163297.

106. Elbert NJ, van Os-Medendorp H, van Renselaar W, et Effectiveness and cost- effectiveness of ehealth interventions in somatic diseases: a systematic review of systematic reviews and meta-analyses. J Med Internet Res. 2014;16(4):e110. doi: http://dx.doi.org/10.2196/jmir.2790.   PMID: 24739471.

107. Jones KR, Lekhak N, Kaewluang Using mobile phones and short message service to deliver self-management interventions for chronic conditions: a meta-review. Worldviews Evid Based Nurs. 2014 Apr;11(2):81-8. doi: http://dx.doi.org/10.1111/wvn.12030. PMID: 24597522.

108. Mair FS, May C, O’Donnell C, et Factors that promote or inhibit the implementation of e-health systems: an explanatory systematic review. Bull WHO. 2012 May 1;90(5):357-64. doi: http://dx.doi.org/10.2471/BLT.11.099424. PMID: 22589569.

109. McLean S, Sheikh A, Cresswell K, et The impact of telehealthcare on the quality and safety of care: a systematic overview. PLoS ONE [Electronic Resource]. 2013;8(8):e71238. doi: http://dx.doi.org/10.1371/journal.pone.0071 238. PMID: 23977001.

110. Morrison D, Wyke S, Agur K, et Digital asthma self-management interventions: a systematic review. J Med Internet Res. 2014;16(2):e51. doi: http://dx.doi.org/10.2196/jmir.2814.   PMID: 24550161.

111. Ekeland AG, Bowes A, Flottorp S. Effectiveness of telemedicine: a systematic review of reviews. Int J Med Inform. 2010 Nov;79(11):736-71. doi: 10.1016/j.ijmedinf.2010.08.006. PMID: 20884286

Telehealth: Mapping the Evidence for Patient Outcomes from Systematic Reviews

Appendixes

Searches for systematic reviews

Ovid MEDLINE (2006 to February 2016)

  • exp Telemedicine/
  • exp Patient Care/
  • exp Therapeutics/
  • exp Health Services/
  • exp Diagnosis/
  • exp Professional-Patient Relations/
  • exp Health Services Accessibility/
  • exp Health Behavior/
  • 2 or 3 or 4 or 5 or 6 or 7 or 8
  • exp *Telecommunications/
  • exp *Computer Communication Networks/ 12 10 or 11

13     9 and 12

14     1 or 13

  • limit 14 to english language
  • limit 15 to systematic reviews
  • limit 16 to yr=”2006 -Current”

 

Cochrane Database of Systematic Reviews (2005 to February 2016)

  • (telemedic$ or telehealth$ or teleradiol$ or teledermat$).mp.
  • (tele-medic$ or tele-heal$ or tele-radiol$ or tele-dermat$).mp.

4     (emedicine or ehealth or e-medicine or e-health).mp.

Searches for ongoing systematic reviews

PROSPERO database (August 1, 2013 to March 2016)

Protocols marked “Ongoing” with the following words in any of the fields: “telehealth” OR “telecare” OR “telemedicine” OR “eHealth” OR “mHealth.”

Searches for primary research

Ovid MEDLINE (2005 to August 2015)

  • exp telemedicine
  • mp 3 1 or 2
  • exp cancer
  • exp chronic disease 6 4 or 5
  • 3 and 6
  • limit 7 to english language
  • limit 8 to yr=”2005 -Current”
  • limit 9 to (“all infant (birth to 23 months)” or “all child (0 to 18 years)”)

——————————————————————————————————————————————————————————————————————————-

  • exp telemedicine/
  • mp.
  • 1 or 2
  • exp Pregnancy/
  • exp Postpartum Period/ 6 4 or 5
  • 3 and 6
  • limit 7 to english language 9 limit 8 to yr=”2005 – 2015″

————————————————————————————————————————————————————————————————————————————-

  • exp telemedicine/
  • mp. 3 1 or 2
  • ambulatory mp. or *Ambulatory Care/
  • urgent mp.
  • exp Triage/ or mp.
  • 4 or 5 or 6
  • 3 and 7
  • limit 8 to english language
  • limit 9 to yr=”2005 -Current”

————————————————————————————————————————————————————————————————————————————-

  • exp mp or exp Telemedicine/
  • *Dermatology/
  • exp Remote Consultation/ 4 1 or 3
  • exp Prognosis/
  • 2 and 4
  • 5 and 6
  • exp Treatment Outcome/
  • 6 and 8

Search for grey literature

Searches of websites of organizations and federal agencies

Text word searches of the following websites were conducted on 9.28.2015 and 9.29.2015

Agency and Organization 1

 

 

 

 

 

 

Agency and Organization 2

 

 

 

 

 

 

 

 

 

Appendix B.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix C.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix C.part 2

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix C.part 3

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix D.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix D. part 1

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix D. part 2

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix D. part 3

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix D. part 4

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix D. part 5 Appendix D. part 6 Appendix D. part 7

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix D. part 8

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix E

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix E part 2

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix E part 3

 

 

 

 

 

 

 

 

 

 

 

Appendix E part 4

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix F

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix F part 2

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix F part 3

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix F part 4

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix F part 5

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix F part 6

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix F part 7

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix F part 8

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix F part 9

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix F part 9

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix F part 10

 

 

 

 

 

 

 

 

 

 

 

Appendix F part 11

 

 

 

 

 

 

 

 

 

 

 

 

Appendix G

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix G part 1

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix G part 3

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix G part 4

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix G part 5

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix G part 6

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix G part 7

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix G part 8

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix G part 99

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix G part 10

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix G part 11

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix G part 12

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix G part 13

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix G part 14

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix G part 15

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix G part 17

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix G part 18

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix H

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix I

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix I part 1

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Leave a Comment

Recent Comments
    © 2020 mTelehealth, Inc. All rights reserved.