Proposed Interoperability Standards Measurement Framework Public Comments

We appreciate the opportunity to comment and are pleased to submit the attached letter with our comments.

On behalf of the Healthcare Information and Management Systems Society (HIMSS), we are pleased to provide written comments to the Office of the National Coordinator for Health Information Technology (ONC) in response to the ONC Proposed Interoperability Standards Measurement Framework (the Framework). HIMSS appreciates the opportunity to leverage our membersÕ expertise in commenting on the ONC Framework, and we look forward to continuing our dialogue with ONC on identifying, assessing, and determining the best available tools and resources to measure the state of interoperability within the health IT community. We feel that this effort will provide important information on the status of our healthcare systemÕs ability to share and use secure electronic information and inform the community on how to continue to advance interoperability to improve care delivery.

Thank you for the opportunity to comment on the proposed ONC ÒInteroperability Standards Measurement FrameworkÓ. We believe it is crucial to focus on measurement of interoperability standards in actual use AND the resulting achievement of true and useful interoperability as evident in health data/record integrity and fitness for use.

See uploaded comment document which includes specific responses to the questions posed and references closely-related CentriHealth comments submitted on 30 June 2017 regarding the "National Quality Forum Ð DRAFT Measurement Framework to Assess Nationwide Progress related to Interoperable Health Information Exchange to support the National Quality Strategy" (also included).

This is a challenge, because nationwide the different computer resources that health professionals are or have implemented into their businesses can be as different as every American family's Thanksgiving traditions. I think that the idea of tying in basic minimum computer infratructure to healthcare payer incentives is a good one. As more healthcare professionals adopt and provide patient access to healthcare electronic records via patient portels the navigation signs will increase. From a computer expert working with experience within large multi-office corporations including some with NIH research grants, communication on the front-end among and between depts, divisions and beyond is imperative. My own personal access to my EHR leads me to this comment: "Holy Cow Doctor, Where did you get this software? Now I'd like to see the machines you're trying to cram it into." Please keep me on your review and comment list as I will be exploring in more detail this subject and developing some healthcare staff and patient potal surveys.

Elizsbeth Martin
lizannm@gmail.com

Attached, please find CHIME's response to ONC's Standards Measurement Framework.

Thank you for the opportunity to comment. Please see the attached letter from the American Hospital Association.

Please see the attached Blue Cross Blue Shield Association comments document.

These comments are being submitted on behalf of the American Medical Association.

July 31, 2017

The Honorable Donald Rucker, MD
National Coordinator for Health Information Technology
U.S. Department of Health and Human Services
200 Independence Avenue SW, Suite 729-D
Washington, DC 20201

Submitted Electronically
Public Comment on ONC 2017 Interoperability Standards Measurement Framework

Dear Dr. Rucker:

On behalf of Battelle Memorial Institute, we are pleased to submit comments on ONCÕs Proposed Interoperability Standards Measurement Framework released in April of 2017. These comments are a compilation of the collective input of our technical staff on stakeholderÕs capabilities to measure, report and test on the use of standards that vary significantly across the health IT ecosystem. We understand this framework aims to help health IT developers, health information exchange organizations, and health care providers move towards a set of uniform measures to assess interoperability progress.

Our general comments are presented on the following pages. We made every effort to respond to the questions posed by ONC, after thoroughly reviewing the proposed Interoperability Standards Measurement Framework and its requirements. We believe that a large research and development business like Battelle could provide significant support to ONC in meeting these requirements. Our industry experience and capabilities as well as our collaborative relationship with Federal Government and industry stakeholders, Battelle has the capacity and technical expertise to offer a refined methodology to enhance this proposed framework. We welcome the opportunity to meet with ONC staff to discuss this input, and identify mechanisms by which Battelle can support ONC in this vital mission.

Please contact either Kathy Lesh (lesh@battelle.org) or Barry Dickman (dickman@battelle.org) with any questions.

Sincerely,

Kathy Lesh, PhD, RN-BC, CPHQ Barry Dickman, MHSA
Healthcare Quality Research Leader Program Manager, Health IT & Analytics

Battelle
2111 Wilson Blvd, Suite 1000
Arlington, VA 22201
http://www.battelle.org

This message is intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential and/or otherwise exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering the message to the intended recipient, any disclosure, dissemination, distribution, copying or other use of this communication or its substance is prohibited. If you have received this communication in error, please return to the sender and delete from your computer system.

Questions Addressed by Battelle

1) Is a voluntary, industry-based measure reporting system the best means to implement this framework? What barriers might exist to a voluntary, industry-based measure reporting system, and what mechanisms or approaches could be considered to maximize this systemÕs value to stakeholders?

The ONC continues to offer as a core mission a desire to engage with industry. Battelle believes that voluntary, industry-based Òself-reporting systemÓ regarding standards used and considered is subject to questionable accuracy and is not a valid proxy for actual true interoperability. This is evidenced by the June 2017 Office of Inspector General (OIG) report on EHR Incentive Program inappropriate payments and the Department of Justice (DOJ) complaint against eClinicalWorks. The OIG report noted that the Government Accountability Office (GAO) estimated the Centers for Medicare & Medicaid Services (CMS) inappropriately paid eligible professionals $729,424,395. They noted Òsampled eligible providers (EPs) did not maintain support for their attestationsÓ (p. ii). Regarding the DOJ complaint, eClinicalWorks is thought to be just the one that got caught. There is a belief, for example see Tweet from Farzad Mostashari, that other electronic Health Record (EHR) vendors have similar issues. EHR consumers are now even more wary of vendors.

A voluntary system needs some sort of incentive or disincentive otherwise it is not worth someoneÕs time to participate. ÒVoluntaryÓ does not imply no incentive. For example, hospital quality reporting is Òvoluntary.Ó However, if an Inpatient Prospective Payment System hospital does not submit the required quality measures, they will not receive their CMS annual payment update. Participation in the electronic health record technology certification program is voluntary. However, obtaining certification will increase sales opportunities.

Reporting/responding burden is a huge consideration. Comments in response to clarification of the ONCÐ Authorized Certification Body (ACB) surveillance or ONC direct review, which would be very similar to the ask for interoperability standards, in the Medicare Access and CHIP Reauthorization Act (MACRA) proposed rule Òexpressed concern that the proposed attestation would be unduly burdensome for health care providersÓ (81 FR 214, 77021). Clinicians and provider organizations are already overburdened with the multiple reporting requirements and multitude of surveys.

The next era of the proposed Interoperability Standards Measurement Framework requires consultation with agencies across HHS. The Federal Health IT Strategic Plan 2015Ð2020 outlines the commitments of federal agencies that use or influence the use of health IT to expedite the availability of high-quality, accurate, secure, and relevant electronic health information for stakeholders across the nation. This document should be referenced and aligned with parallel efforts to support the effort to develop a mechanism to measure interoperability. Clinicians patients and additional stakeholder groups must be integral to this planning. The proposed and revised Interoperability Standards Measurement Framework does not get to the core issue of measuring interoperability.
_
2) What other alternative mechanisms to reporting on the measurement framework should be considered (for example, ONC partnering with industry on an annual survey)?

ONC should consider if they are in the best position to do this work. In order to truly measure interoperability, work with existing organizations conducting HIT use surveys, such as the American Hospital Association and the Centers for Disease Control and Prevention, to add questions addressing use of interoperability standards may be assistive. ONC should expand Certified Health IT Product List (CHPL) requirements to include all capabilities and how they are achieved, including use of standards in each version of a product. Medical device interoperability standards should be reviewed and included.

Stakeholders such as specialty societies and patient advocacy groups can be leveraged to actively review and evaluate findings to ensure that the gaps and measurement framework development opportunities identified are clearly conveyed and understood. These findings should be part of a feedback loop, and this process used according to the principles of a learning health system.
ONC may benefit from leveraging existing work that solicits stakeholder feedback mechanisms (e.g., specific to CMS Electronic Clinical Quality Measures (eCQM) specifications, standards, and implementation). In the past, ONC has engaged frontline clinicians on projects and initiatives to inform its approach to communication, outreach, quality reporting, program implementation, and clinician quality measure development.

In the past, ONC has effectively utilized multi-stakeholder technical expert panels and recommendations to prioritize on reporting on the measurement framework.

3) Does the proposed measurement framework include the correct set of objectives, goals, and measurement areas to inform progress on whether the technical requirements are in place to support interoperability?
- Goal #1: Improve knowledge of implementation level of standards within health IT products and services (i.e., which standards are most commonly available)
We recommend simplify to Improve knowledge of which standards are most commonly available within health IT products and services.
- Goal #2: Track the use of standards by end users in deployed systems (i.e., which standards are most commonly being used and understand how often and in what manner standards are customized during implementation.
This is an action, not a goal. To be in alignment with Goal #1 suggest ÒImprove knowledge of which standards are in use by end users in deployed systems and how often standards are customized during implementation.Ó

Objectives must be measurable and usually are actions designed to meet the goal. Often there are several objectives to meet a goal. It seems odd that ONC has offered only one objective per goal.

Objective #1: Understand if specific standards are built into health IT products and available to end users (i.e., the implementation lifecycle). Having technical requirements in place to support interoperability is only the first half (or less) of the equation toward reaching an interoperable learning health system. True interoperability requires a socio-technical model in order to interoperate smoothly.

- While Objective #2: Understand the use of standards and how they are deployed into production systems to meet specific interoperability needs as well as the level of conformance or customization of standards during implementation, hints at the sociology, none of the measurement areas get to the sociology to be able to Òunderstand the useÉÓ

4) What, if any gaps, exist in the proposed measurement framework?
While simplicity is good, the proposed measurement framework is not likely to answer whether widespread interoperability can be achieved or even what level of interoperability has been achieved. Health IT developers and exchange networks will not be able to provide accurate data for measurement areas for Objective #2. Data addressing the volume of transactions by standard might be available from exchange networks, but not health IT developers. At best, the volume will only provide a number up or down. Percentage of transactions may be a better metric. None of Objective #2Õs measurement areas will capture use of terminology standards.

There is no mechanism for evaluating the state of interoperability and then working through the system to determine why it is working and what is causing it not to work if indeed it is not. We learned from Meaningful Use (MU) and View Download Transmit (VDT) requirements that just because standards are used and information is ÒexchangedÓ those transactions are not in and of themselves what interoperability is. Use of the Socio-technical model that the ONC underutilized SAFER guides are modeled after- cannot be under emphasized here.

While the health IT developers must make interoperability standards available in their products, it is up to the end user to implement the standards. Health IT developers must make it easy for end users to implement. End users, clinicians and patients are the only ones that can state whether interoperability standards are allowing for adequate sharing and understanding of health information. The definition of end user should be expanded to specifically include HIT staff at healthcare organizations, health information exchanges, patients, and their caregivers. Patients and caregivers may not be aware of the standards used, they can offer opinions as to whether data and information are interoperable.

Use of interoperability standards is not binary. It is a spectrum. Just as there are different levels of EHR adoption, there are different levels of interoperability. ONC should consider adopting /adapting a model similar to the Framework for Safe Medical Device Interoperability described by Robkin, Weininger, Preciado, and Goldman and incorporate methodology and algorithms as is done with the HIMSS Analytics Electronic Medical Record Adoption Model.

ONC has identified gaps on various elements within the Interoperability Standards Measurement Framework, derived from federal reports, stakeholder groups, and relevant public comment documents. These gaps highlight known interoperability standards measurement framework gaps and recommend engaging frontline health IT developers, exchange networks, clinicians, patients, families, and caregivers and other advocacy groups in addressing those gaps through the development, adoption, and refinement of the framework.

5) Are the appropriate stakeholders identified who can support collection of needed data? If not, who should be added?

The statement is made that ÒEnd users such as providers are not well positioned to capture (or even know about) this data nor are they necessarily the most accurate sources for these data.Ó However, health IT staff and informatics-savvy clinician end users will likely be the most accurate sources. The use of and evaluation by a multi-stakeholder panel of health IT developers, exchange networks, clinicians, patients, families, and caregivers and other advocacy groups perspectives and experiences should be part of refinement of the ONCÕs Interoperability Standards Measurement Framework.

6) Would health IT developers, exchange networks, or other organizations who are data holders be able to monitor the implementation and use of measures outlined in the report? If not, what challenges might they face in developing and reporting on these measures?

Not likely. As noted in the proposed Interoperability Standards Measurement Framework, system audit functions vary greatly. A 2013 OIG report stated that, although the recommended audit functions were available in most hospital EHRs, not all hospitals were using the audit functions fully. While this OIG report was not addressing interoperability standards audit, it indicates that just because a function is available, not all organizations will use those functions. Also, content of transmissions will not always be available to these data holders so there would be no way to assess use of terminology standards.

Understand that health IT interoperability will encounter a variety of versions of standards and specifications within the health IT community. There are many organizations and associations focused on educating and supporting this community to ensure interoperability. This requires that ONC, SDOs (Standards Developing Organizations), and implementing organizations recognize that standards are continuously evolving.

The HHS Office of the Chief Information Officer (OCIO) has recommended for its Operating-Divisions (OP/DIVs) to implement and abide by the approved Enterprise Performance Life Cycle (EPLC) Framework. This requires programs and projects to document all relevant artifacts. If implemented and followed, it supports consistency and benefits from its applied methodology. The development and testing processes inform ONC and stakeholders about the viability of a measure for broad-based implementation and documented artifacts. These processes can be conceived as a series of gates through which each standards measurement framework successfully pass to advance for consideration.

In following the recommended HHS OCIO EPLC Framework, ONC or another more appropriate agency would benefit from providing integrated end-to-end testing systems end-to-end earlier in the process. ONC would further enable its ability to showcase and leverage its Standards Implementation & Testing Environment (SITE). SITE is considered by industry as a centralized collection of testing tools and resources designed to assist health IT developers and health IT users fully evaluate specific technical standards and maximize the potential of their health IT implementations. These tools should be crowd-sourced in their effective use and highlighted as best practices for the ONCÕs Interoperability Standards Measurement Framework.

Prior to testing, the health IT developerÕs community, exchange networks, or other organizations who are data holders must ensure appropriate implementation of the interoperability standards measurement framework. This represents an opportunity for quality improvement and is deemed important by all stakeholders. After such a framework has received approval to move forward, the health IT developerÕs community, exchange networks, or other organizations use preliminary specifications to carry out initial testing. If the initial testing results are acceptable, the measures will proceed to further testing once full specifications are developed. If any implementation concerns are noted during this process, measures development could be stopped or paused until these concerns have been addressed. Iterative testingÑin conjunction with continual public input throughout development, including additional public comment opportunities, review of specifications and testing resultsÑassures that only measures which pass each critical checkpoint continue to advance toward implementation. This approach also informs ONC of successes and challenges as it works toward the goal of developing an interoperability standards measurement framework for which strong statistical evidence of a true gap in performance exists.

Robust testing ensures that measures used by ONC will function as intended to attain quality goals. Data availability is a current barrier to measure developers. Therefore, greater data transparency, integration, and consolidation are critical in promoting robust this interoperability standards measurement framework.

7) Ideally, the implementation and use of interoperability standards could be reported on an annual basis in order to inform the Interoperability Standards Advisory (ISA), which publishes a reference edition annually. Is reporting on the implementation and/or use of interoperability standards on an annual basis feasible? If not, what potential challenges exist to reporting annually? What would be a more viable frequency of measurement given these considerations?

It depends on who you are asking to complete the survey. It should be reasonably easy, and therefore feasible, for health IT developers to provide which standards are in their products on an annual basis. SDOs can provide the status of their standards. Different types of exchange networks may have greater challenges depending on their involvement in the exchange of data/information. Any additional reporting requirement on healthcare organizations will be a burden. ONC should work towards developing a format for automated reporting to reduce the burden.

ONC also needs a feedback loop to SDO's. Far too often the health IT community has borne witness to a general lack of feedback to the many SDO's who publish standards. Numerous implementations and frameworks fail to include the SDO's in difficult implementation discussions. We believe there is a significant opportunity for improvement, by supporting ONC on a Shared Services based Testing Platform. The SDO would be kept apprised of which standards were being tested, how the testing was progressing, and when applicable, what areas of the standards or specifications are proving more difficult to implement - with a potential solution being to issue a clinical protocol and improve the standard for future implementations. We believe there is a significant opportunity to ensure future standards and specifications would decrease adoption time, and improve interoperability if a Shared Services based Testing Platform were utilized.

As previously mentioned, this progress toward this next era of the proposed Interoperability Standards Measurement Framework requires consultation with agencies across HHS. HHS has developed the Federal Health IT Strategic Plan 2015Ð2020. The plan outlines the commitments of federal agencies that use or influence the use of health IT to expedite the availability of high-quality, accurate, secure, and relevant electronic health information for stakeholders across the nation. This document should be referenced and aligned with parallel efforts to its use and implementation. HHS and ONC are leading initiatives to spur innovations in health IT that will further reduce the burden of data collection for clinician quality measurement.

8) Given that it will likely not be possible to apply the measurement framework to all available standards, what processes should be put in place to determine the standards that should be monitored?

Terminology standards are going to be the greatest challenge, but are among the most important to be monitored. Use of standard terminology is essential for sematic interoperability. Even if terminology standards are integrated into products, many healthcare organizations map and then use their local terminology. Use of terminology standards is not equal. Organizations will be using International Classification of Disease Tenth Revision Clinical Modification/Procedure Coding System (ICD-10-CM/PSC) because of the payment implications, but how many are actually using SNOMED CT?

Logical Observation Identifiers Names and Codes (LOINC) is used for laboratory messaging, but can be used more widely.

9) How should ONC work with data holders to collaborate on the measures and address such questions as:

¥ How will standards be selected for measurement?
We believe this can be effectively accomplished by launching a task force from the newly organized and implemented Health IT Advisory Committee.
¥ How will measures be specified so that there is a common definition used by all data holders for consistent reporting?
We believe this can be effectively accomplished by launching a task force from the newly organized and implemented Health IT Advisory Committee.

10) What measures should be used to track the level of ÒconformanceÓ with or customization of standards after implementation in the field?

We believe that true ÒconformanceÓ is derived from the Health IT Developers and Implementers level of testing. ONC currently offers a certification testing program. ONCÕs Certification Programs should leverage proven test cases. Far too often we have observed organizations publish Certification Test Cases which the industry has not benefited from having an opportunity to "Test the Test". We believe any mature certification program will publish in draft form a series of test cases which Industry has an opportunity to vet "test out" within their systems. This allows industry an opportunity to report back on the "maturity" (status) of support and identify any potential challenges which may exist, which later on could/may prove to be politically difficult for a certifying body to execute and report on or enforce.

As shared by industry and federal reports, we remain concerned that during these efforts, certification-based test cases could be released which a limited number of EHR vendors may be able to pass, this would provide those few with a significant market advantage. Later, facing increased public pressure, the certification body may try to relax some of the published testing requirements (potentially deemed too difficult). With our proposed proactive approach to introduce test cases early to industry and to provide an opportunity for feedback - we believe ONC could avoid potentially politically difficult situations in the coming years.

Organizations such as American Hospital Association (AHA), American Medical Association (AMA), College of Healthcare Information Management Executives (CHIME), American Health Information Management Association (AHIMA), Gartner, and other committees, organizations, and advocacy groups have spoken to the lack of negative testing within health IT implementations as an industry-wide issue. We remain concerned that ONC, in an effort to address "Negative" or "Exception" calls for test cases, will introduce a sprinkling of negative test cases across the Certification Testing Program. Our concern centers on the readiness of vendorsÕ EHR systems supporting patient safety concerns in the real world. Our desire is that ONC share with organizations such as ours, the objective and approach to support "negative" and "exception" type testing. Furthermore, publishing a Certification Program which claims 5% or even 10% negative or exception test cases does not ensure the rigors appropriate for a "production ready" system. Please note, we are not proposing that the ONC Certification Program undertake a burden of forcing EHR vendors into an undertaking of 300% or 400% more testing, rather we propose developing comprehensive suites of "negative" or "exception" test cases, and fielding those in trial mode.

The Certification Program could randomly select from those suites up to 5% or 10% of the negative test cases, thereby exposing a vendor EHR to typical real-world situations. The Certification Program should not seek to claim an EHR vendor product is certified today and remains certified indefinitely. Recent news and investigations has demonstrated that many organizations which obtained their ONC Meaningful Use Stage 1 Certification, remained under product development with patch releases and product updates - while rarely returning to their ONC Certification Authorized Testing and Certification Body (ONC-ATCB) for retesting (note: some vendors have operated at higher levels of adherence than others). But, by far, a majority of EHR vendors did not return for retesting. Industry should consider doing a deeper dive of analysis into this effect if we are to recognize the risk to patient safety and ensure Certification Programs in the future have higher degrees of assurance of patient safety.

As far as assessing the amount of customization after implementation, ONCÐ Authorized Certification Body (ACB) surveillance or ONC direct review may be the only way to accurately measure the amount of post-implementation customization.

Surescripts Comments on Standards Measurement Framework attached