SMART Health IT
Computational Health Informatics Program
Boston Children’s Hospital
October 19, 2026
Re: Docket No. FDA-2026-N-7874, Considerations for the Regulation of Generative AI-Enabled Medical Devices: Discussion Paper and Request for Feedback
Submitted on behalf of the SMART Health IT team (the team that originated and maintains the SMART on FHIR ecosystem)
Dear CDRH and Digital Health Center of Excellence Leadership,
Thank you for the opportunity to comment. We share CDRH’s objective of a risk-based, least burdensome approach to generative AI (GenAI)-enabled devices, and we agree that premarket testing alone cannot fully characterize these devices.
Our comments focus on the interfaces through which a GenAI-enabled device reads a patient’s record and, where appropriate, returns output to clinical workflows. HL7 Fast Healthcare Interoperability Resources (FHIR) defines a standard data format for exchanging clinical data. The SMART Health IT team at Boston Children’s Hospital originated and led the development of SMART on FHIR, the open standard for connecting applications to electronic health records (EHRs) that federal regulation now requires of every certified EHR, and the FHIR Bulk Data Access specification (Bulk FHIR) for population-scale export. We also originated CDS Hooks, the HL7 specification for invoking decision support services from within EHR workflows.
These interfaces are relevant to the paper’s proposed evaluation and monitoring approaches, which depend on retrieving health record data in a form that can be tested, sampled, and audited. Many of these standards are required under the Health IT Certification Program of the Office of the National Coordinator for Health Information Technology (ONC), though performance remains uneven across vendors. We address the discussion questions by number. The FHIR tools we cite below are publicly available and open source.
Part 1. Premarket evaluation (Section V)
1) The interface that delivers clinical data is part of the deployment context. The paper proposes evaluating “the final user-facing device, as configured and intended to be deployed” (Section V.A), with benchmark datasets “representative of the deployment contexts” (Section V.B.2). For a device that reads a patient’s chart, the form in which the chart arrives is part of that context. A device that ingests a record through SMART on FHIR receives US Core FHIR resources, in the versions ONC adopts for certification, with known structure, terminology bindings, and known gaps. A device that ingests a proprietary extract, a screen capture, or pasted text receives something different. A device tested on SMART on FHIR data may not perform the same way on a proprietary extract, and the reverse.
Recommendation. Where a device consumes the record through a standards-based interface, CDRH should encourage sponsors to state the interface in the device description, with any limitations encountered, and to construct benchmarking and clinical-confirmation inputs in a form consistent with nationally certified EHR standards, namely US Core FHIR resources delivered through SMART on FHIR or Bulk FHIR. Open-source tools exist for this. Our Cumulus tools turn a Bulk FHIR export into a de-identified dataset for population-scale analysis, and our CumulusQ tools compute FHIR-native data quality and characterization metrics on the export, so that a sponsor can show what a benchmarking dataset contains and where it is incomplete (JAMIA, 2024). (Questions 9, 10, 11)
2) A certified route to population-scale real patient records exists. Several proposals presume that a sponsor or institution can assemble the records of a large, defined population: retrospective evaluation on real inputs, shadow deployment, subgroup performance (element R.2), and estimation of the “anticipated distribution of real-world inputs” (Question 11). A mechanism for this exists and is not as widely known as it should be.
Bulk FHIR is the HL7 standard for exporting the records of a population rather than one patient at a time. A requester names a group of patients and receives their records as files of FHIR resources in the same US Core form that SMART on FHIR delivers for a single patient. SMART Backend Services provides a standardized mechanism for authorizing unattended applications to retrieve these exports. Our team developed the specification with support from ONC, and since December 31, 2022, certified EHRs have been required to support it for the USCDI data elements under 45 CFR 170.315(g)(10). The Centers for Medicare & Medicaid Services (CMS) uses the same standard in three of its own APIs: the Beneficiary Claims Data API, which delivers Medicare claims to accountable care organizations; Data at the Point of Care, which delivers them to treating providers; and AB2D, which delivers Medicare Part A and Part B claims to Part D plan sponsors. Claims delivered this way can be joined to the EHR record, and the combination matters for monitoring: readmissions, emergency visits, and prescriptions filled often appear in claims and not in any one institution’s record. Bulk FHIR is the route by which a sponsor could draw benchmarking inputs from a health system’s actual population, an institution could retrieve the records of the patients a device touched, and a third party could deliver a sequestered dataset in a form the device already consumes.
The capability is uneven in practice. A multi-vendor evaluation published in 2024 tested five sites in 2023 and found that at three of the sites, exports of about a thousand patients took 26 to 35 hours, that one attempted export of about 10,000 patients ran for nearly two weeks without a successful download, that date filtering was supported at some sites and not others, and that sites needed from two days to four months of setup before a first request could be made. The paper’s interest in synthetic data (Questions 12 and 13) is understandable in this light. Synthetic records are most valuable where they cover cases real data cannot, and least valuable as a stand-in for real populations that certified interfaces are already required to deliver. Generating synthetic records as US Core FHIR resources lets the same metrics compare synthetic and real data directly. Bulk FHIR has two further limits. First, an export can finish and still be missing data. Our BulkUP tool, from the EHR Good Neighbor project, measures export speed and checks which data are present. Second, an export taken later shows the chart as it is now, not as it was when the device ran. To know what the device saw, its inputs have to be saved each time it is used (comment 3).
Considerations. ONC sets certification requirements for these interfaces, so we do not propose new technical requirements. We make three points. First, FDA’s evidence expectations already depend on ONC’s certification requirements. The data FDA would want sponsors to use includes the data ONC requires certified EHRs to export. If the export does not work, producing the evidence becomes far more costly. Work to make standardized extraction more performant is under way: the CMS Health Technology Ecosystem includes a Bulk FHIR pledge, under which EHR vendors, health systems, and payers commit to usable performance at the scale of hundreds of thousands to millions of records and to transparent performance measurement, and a workgroup on implementation. The framework should say this, so that sponsors know to ask for the export and so that FDA sees where it falls short. Monitoring also depends on capabilities that certification does not yet require, notably a standard way for a requester to dynamically define a cohort, such as the patients on whom a device was invoked. Second, an FDA demand signal for this interface would matter beyond GenAI-enabled devices. Programmatic access to EHR data at population scale is one resource that serves many national priorities at once: AI evaluation and monitoring, quality measurement, public health reporting, clinical research, and surveillance of drugs and devices. Each has been pursued with its own pipeline, built and paid for separately, when they could instead all draw on the same certified export. In 2022 we convened federal agencies, payers, providers, and vendors on exactly this point (Multi-solving Population Data Use with SMART Bulk FHIR Access). The finding was that the export becomes a common data source for all of these uses once the agencies that depend on it say so. At that meeting Amy Abernethy, formerly FDA’s Principal Deputy Commissioner, described FDA as the demand signal for high-quality datasets. CMS has sent its signal through its own APIs and, more recently, the pledge. A statement from CDRH that its evidence expectations for GenAI-enabled devices rest on this interface would be the signal that reaches device sponsors, and it would strengthen every other use of the same data. Third, our team can brief CDRH staff on Bulk FHIR. (Questions 11, 12, 13)
Part 2. Postmarket monitoring (Section VI)
3) Postmarket monitoring needs a retrievable record of each invocation, and CDS Hooks provides the foundation for capturing it. Shadow deployment (Section V.C.1), sample-based clinician review, and degradation monitoring (Section VI.A) all presuppose that an institution or sponsor can retrieve what the device was given at the moment it ran, what it generated, and what the user did, and link that interaction to downstream patient outcomes. Today, the interaction record usually sits in the manufacturer’s proprietary telemetry, separated from the clinical outcomes in the health system’s EHR.
HL7’s CDS Hooks, originated by our team and widely implemented across EHRs, already defines the standard structure for workflow-integrated decision support: a structured request carrying patient context and prefetched FHIR data, a response returning informational or actionable suggestion cards, and a feedback endpoint recording whether the clinician accepted or overrode the suggestion with reasons. ONC certification (45 CFR 170.315(b)(11)) also requires EHRs to export clinician feedback data for certain decision support interventions. To serve full GenAI postmarket monitoring, this foundation needs two extensions: persisting the exchange and user action (including ignored outputs) as a retrievable FHIR audit record in the EHR, and extending the pattern to GenAI interactions beyond synchronous alerts, such as ambient documentation and multi-turn chat.
Recommendation. CDRH should encourage sponsors of workflow-triggered GenAI devices to integrate via CDS Hooks and to retain, for each GenAI invocation, a structured record of the inputs as received (or versioned FHIR references resolving to the chart at that moment), the output, the model and prompt version, and the user action, in a form the deploying institution can retrieve through its standard FHIR interfaces.
Persisting this invocation record in FHIR, so that a Bulk FHIR export can return it alongside the rest of the chart, addresses the retrospective gap noted in comment 2: the invocation record preserves what the device saw at the moment of use, while periodic Bulk FHIR exports supply downstream clinical outcomes (Question 19) and allow data-quality tools to detect input or environment drift (Question 22). It also lets institutions audit devices locally or run federated analyses without relying on vendor logs, while letting manufacturers monitor devices against standard exports the institution already supports (Question 21). Where CDRH accepts greater premarket uncertainty in exchange for postmarket monitoring (Question 18), it could ask sponsors to show that invocation records can be captured, retrieved, and linked to outcomes through standard interfaces at intended deployment sites, and coordinate with ONC to align certified decision-support feedback exports with CDS Hooks and Bulk FHIR. (Questions 18, 19, 21, 22)
4) For devices that act on the record through SMART on FHIR, authorization scopes bound what the device can do through that connection. The agentic element (A.1) asks whether an agent recognizes when an action would exceed its safety envelope. For an application connected through SMART on FHIR, part of the envelope is enforced at the EHR rather than by the model. SMART App Launch scopes determine whether the application can read or write and which resource types it can touch. An application granted read-only scopes cannot place an order through that connection, however it reasons.
Recommendation. For EHR-integrated GenAI devices and agents, CDRH should expect sponsors to document the scopes the device requests and why, and should treat scope minimization as evidence relevant to the scope-maintenance element (S.2) and the agentic element (A.1). For clinician-supervised workflows, CDS Hooks can present suggested actions for clinician review instead of having the agent write orders autonomously. (Questions 1, 26)
5) Patients are connecting their records to GenAI tools through the patient-access API, and that access changes the risk. Question 3 asks what might mitigate risk when a patient cannot independently evaluate an output. Under the ONC Cures Act Final Rule, certified EHRs must offer patients a standards-based API built on SMART on FHIR (45 CFR 170.315(g)(10)), through which a patient can direct their record to an application of their choosing, and patients are using it to connect their records to general-purpose assistants. A function working from the patient’s actual medication list, laboratory results, and problem list is answering a different question from one working from the patient’s recollection. In the paper’s insulin-dose example, access to documentation on the current dose, recent glucose values, and renal function changes the risk.
Recommendation. CDRH should treat authorized access to the patient’s structured record as relevant to the risk assessment of patient-facing functions, and should be explicit that the patient-access API and an EHR’s conformance to it are not device functions, so that device oversight does not become a reason to restrict patients’ access to their data. (Questions 3, 5)
Closing
The paper asks whether CDRH should accept more premarket uncertainty in exchange for stronger postmarket evidence (Question 18). Whether that evidence can be produced is in large part a question of interfaces. Our comments reduce to three points:
- For devices that read the health record, the standards-based interface is part of the deployment context, and test inputs should take the same form.
- Bulk FHIR is the certified route to population-scale real patient records. CDRH should know where it performs and where it does not, and coordinate with ONC.
- Postmarket monitoring needs a structured, retrievable record of each invocation, and a sponsor proposing to rely on monitoring in place of premarket evidence should show that the record can be captured and retrieved at the intended deployment sites.
We appreciate the opportunity to comment and would welcome continued engagement.
Respectfully submitted,
Ken Mandl, MD, MPH, Dan Gottlieb, MPA, and Josh Mandel, MD, on behalf of the
SMART Health IT Team
Computational Health Informatics Program
Boston Children’s Hospital