§170.315(g)(3) Safety-enhanced design

Clarifications:

• The application of user-centered design (UCD) during development and summative testing is limited to only those nine certification criteria specified in this certification criterion and only for which certification is sought, namely [80 FR 62670]:
  • § 170.315 (a)(1) Computerized provider order entry (CPOE) – medications
  • § 170.315 (a)(2) Computerized provider order entry (CPOE) – laboratory
  • § 170.315 (a)(3) Computerized provider order entry (CPOE) – diagnostic imaging
  • § 170.315 (a)(4) Drug-drug, drug-allergy interaction checks for CPOE 
  • § 170.315 (a)(5) Demographics
  • § 170.315 (a)(9) Clinical decision support (expires on January 1, 2025)
  • § 170.315 (a)(14) Implantable device list
  • § 170.315 (b)(2) Clinical information reconciliation and incorporation
  • § 170.315 (b)(3) Electronic prescribing
  • § 170.315 (b)(11) Decision support interventions
• As a “revised” certification criterion, the safety-enhanced design (SED) certification criterion is not “gap certification eligible.” [80 FR 62609-62610, 62670] Thus, despite the fact that some of the functionality-based certification criteria referenced by the SED criterion are “gap certification eligible” for their functionality, all of the certification criteria referenced by this SED criterion (as applicable to certification scope sought) must have UCD processes applied and new summative usability test results as the basis for certification.
• To demonstrate compliance with this certification criterion, UCD process(es) must have been applied to each capability of technology that is associated with the certification criteria named in this certification criterion. [77 FR 54188]
• If technology is presented for certification and includes capabilities to which this certification criterion would apply, but for which certification is not sought, then those other capabilities for which certification is not sought would not have to have had UCD process(es) applied because they would be beyond the scope of certification. [77 FR 54188]
• ONC-Authorized Certification Bodies (ONC-ACBs) should be notified when changes to user-interface aspects occur. ONC-ACBs are required to obtain a record of all updates to Certified Health IT Modules affecting the capabilities in certification criteria to which this “safety-enhanced design” criterion applies on a calendar quarterly basis. [80 FR 62727]
• The documentation required by this “safety-enhanced design” criterion will become a component of the publicly available testing results on which a certification is based. [77 FR 54187]
• ONC does not expect health IT developers to include trade secrets or proprietary information in these reports. [77 FR 54188]
• The ISO definition of usability is “[t]he extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use." [see also 77 FR 54186]
• Health IT developers who have already followed UCD in previous development efforts for the certification criteria identified in the SED criterion would be performing a retrospective analysis for the purposes of certification. [see also 77 FR 54188] ONC notes that the discussion of retrospective analysis provided in the 2014 Edition Final Rule was in the context of health IT being certified for the first time to the new “SED” certification criterion. As an illustration of retrospective analysis for certification to the “SED” certification criterion, if a health IT developer had followed/applied a UCD process for any or all of the certification criteria referenced by the SED certification criterion, a developer would be permitted to cite that previously applied UCD process.
• The Certification Program has modified § 170.315(g)(3) to reference the new certification criterion in § 170.315(b)(11). Certified Health IT Developers must assess user-facing functionality gaps between the requirements of § 170.315(a)(9) and § 170.315(b)(11) and, as necessary update, their safety-enhanced design (SED) testing. This means that functionality new to the (b)(11) DSI criterion, such as the functionality to modify source attributes and source attribute information at § 170.315(b)(11)(v)(B) and the functionality to
  enable users to provide feedback to evidence-based DSIs at § 170.315(b)(11)(ii)(C), would likely require user-centered design processes applied during development of those functionalities and included as part of summative testing.

Technical outcome – Developer must have applied UCD process(es) for any of the certification criteria specified in this criterion and for which certification is sought.

Clarifications:

• The reported UCD process(es) must have been applied during the design and development of the capabilities/associated criteria.
• If UCD had not been previously applied to capabilities associated with the certification criteria, the technology would ultimately need to have such UCD process(es) applied before it would be able to be certified. [77 FR 54187]
• Examples of resources that technology developers may choose to review in order to select a UCD:
  • ISO 9241-11;
  • ISO 13407;
  • ISO 16982;
  • ISO/IEC 62366;
  • ISO 9241-210;
  • NISTIR 7741;
  • NISTIR 7742;
  • NISTIR 7804; and
  • NISTIR 7865 (“A Human Factors Guide to Enhance EHR Usability of Critical User Interactions when Supporting Pediatric Patient Care”).
• Any UCD process selected by a health IT developer is appropriate, and it need not be listed in the examples we provided in order to be acceptable. [77 FR 54188]

Technical outcome – At a minimum, ten participants must have been included in the summative usability testing required for each required capability for which certification is sought.

Clarifications:

• Although only 10 participants are required, health IT developers are strongly encouraged to exceed the mandatory minimum in an effort to identify and resolve more problems. [see also 80 FR 62671]
• The cohort of users who are selected as participants will vary with the product and its intended users and should not be limited to clinicians but instead consist of test participants with the occupation and experience that aligns with the capability under testing. [see also 80 FR 62670 and 80 FR 62671]
• ONC recommends that health IT developers follow NISTIR 7804 for human factors validation testing of the final product to be certified. [see also 80 FR 62671]
• Recommended resources to consider:
  • NISTIR 7804; and
  • NISTIR 7741.

Technical outcome – The developer must submit the name, description and citation (URL and/or publication citation) for the industry or federal government standard used in UCD for the development of each required/applicable capability presented for certification.

Clarifications:

• Examples of resources that developers may choose to review in order to select a UCD process include, but are not limited to:
  • ISO 9241-11;
  • ISO 13407;
  • ISO 16982;
  • ISO/IEC 62366;
  • ISO 9241-210;
  • NISTIR 7741;
  • NISTIR 7742;
  • NISTIR 7804; and
  • NISTIR 7865 (“A Human Factors Guide to Enhance EHR Usability of Critical User Interactions when Supporting Pediatric Patient Care”).

Technical outcome – If a non-standard UCD process was used in development (i.e., § 170.315(g)(3)(iii)(A) was NOT met), the health IT developer must report the process(es), provide an outline of the process(es), a short description of the process(es), and an explanation of the reason(s) why use of any of the existing UCD standards was impractical for each required/applicable capability presented for certification.

Clarifications:

• No additional clarifications.

Technical outcome – The information specified in (g)(3)(iv)(A)-(F) must be submitted for each capability to which UCD processes were applied in product development and when summative usability testing was conducted.

Clarifications:

• All of the data elements and sections specified must to be completed, including ‘‘major findings’’ and ‘‘areas for improvement.’’ [see also 80 FR 62670]
• Health IT developers can perform many iterations of the usability testing, but the submission that is ultimately provided for summative usability testing and certification must be an expression of a final iteration. [see also 80 FR 62671]
• Only lab based summative testing is necessary to be performed in order to demonstrate compliance with this certification criterion. Nothing precludes field testing and formative testing from also being performed and we encourage technology developers to do so. [see also 77 FR 54189]
• Information must be submitted for each and every one of the criteria specified in the “SED” criterion to become part of the test results publicly available on the Certified Health IT Product List. [see also 80 FR 62725]
• To demonstrate compliance with this certification criterion this information would need to be available to an ONC–ACB for review, but the form and format for how the data would be presented for testing would not necessarily need to be NISTIR 7742 template. This documentation would become a component of the publicly available testing results on which a certification is based. [see also 77 FR 54187]
• For the requirement in (g)(3)(iv)(D), it is permissible to submit an alternative acceptable user satisfaction measure to meet the requirements of this criterion. As such, a health IT developer could meet the proposed NISTIR 7742 based approach for user satisfaction or provide documentation of an alternative acceptable user satisfaction measure. [see also 80 FR 62671]
• It is important to note a specific distinction between paragraphs (g)(3)(iii) and (g)(3)(iv). Both paragraphs are complementary, but require different information and have different applicable timing considerations.
  • Paragraph (g)(3)(iii) requires the disclosure of the UCD processes that were applied and followed in the development of the applicable capabilities named in the “SED” certification criterion and for which certification is sought.
    • If a health IT developer had previously developed a capability following a certain UCD process, we clarified that the health IT developer would not need to pick a new UCD standard as a result of our rulemaking and use it to reengineer the capability following/reapplying that UCD standard. In that respect, the timing for this paragraph (g)(3)(iii) allows for retrospective attribution. [see also 77 FR 54188]
    • ONC notes that in the 2014 Edition final rule preamble we incorrectly referenced NISTIR 7742 on page 54188. NISTIR 7742 does not require the reporting of the UCD standard/process used. Rather, in the preamble of the 2014 Edition final rule we stated that requirement. [see also 77 FR 54188]
  • Paragraph (g)(3)(iv) requires that information/sections from NISTIR 7742 must be submitted for each capability to which user-centered design processes were applied.
    • In so doing, a health IT developer cannot use summative usability test results to meet this paragraph that were not generated for the specific edition of certification criteria for which certification is sought.
    • In other words, if a health IT developer had followed a UCD process in 2010 to design its CPOE functionality, performed and provided summative usability test results in 2013 for the purposes of 2014 Edition certification, the health IT developer must produce a “fresh” set of summative usability test results for testing and certification. See also the clarification note about gap certification eligibility in the general clarification section above.
  • Once certified to the SED certification criterion, a developer can request inherited certified status for a newer version of its certified product consistent with 45 CFR 170.550(k). In so doing, the Certified Health IT Developer may attribute (“carry forward”) the summative usability test results created for the previously certified product version so long as the scope of referenced SED remains the same. In instances where an additional SED referenced certification criterion/criteria is/are added the summative usability test results applicable to those addition in scope would be needed.

Technical outcome – The test scenarios participants used during the summative usability testing must be submitted as part of the test results report.

Clarifications:

• In accordance with NISTIR 7804 Technical Evaluation, Testing, and Validation of the Usability of Electronic Health Records (EUP) (page 8), we recommended that the test scenarios be based upon an analysis of critical use risks for patient safety, which can be mitigated or eliminated by improvements to the user interface design. [see also 80 FR 62670]
• NIST recently developed an additional recommended resource for test scenarios: NISTIR 7804-1: Technical Evaluation, Testing, and Validation of the Usability of Electronic Health Records: Empirically Based Use Cases for Validating Safety-Enhanced Usability and Guidelines for Standardization.