SMART/HL7 Bulk Data Export (Flat FHIR)

Put simply, Flat FHIR is a standard to enable “push button” population health.  An ACO, a researcher, or a public health authority should be able to, without special effort, extract a cohort of patients from the EHR for further analysis. 

A learning healthcare system requires the ability to aggregate and analyze data at a population level, to better manage population health, deliver value-based care, and conduct discovery science.  Access to data is a fundamental requirement for improving the health of individuals and populations. APIs for working with population level data, often referred to as bulk data, are needed to support this system.

Today, health systems perform bulk data export through proprietary pipelines, requiring teams of IT professionals to perform manual mapping to and from delimited text formats like comma separated values (CSV), for very single reporting and analytics pipeline, of which there are many. These manual processes are extremely expensive and time consuming, and are not sustainable.

The existing FHIR and SMART APIs work well for accessing small amounts of data, but large exports perform poorly, since it would require hundreds of thousands of individual API requests.

On behalf of the Office of the National Coordinator for Health Information Technology (ONC) , The Boston Children’s Hospital Computational Health Informatics Program and SMART hosted a meeting in December 2017 to discuss standardizing bulk data exports from EHR systems and data warehouse environments. These discussions resulted in a community effort to define a FHIR Bulk Data API specification, which was balloted through HL7 in early 2019.

The initial specification focused on a specific set of initial use cases, and is currently used by CMS to provision data, including in its 2019 Data to Point of Care program.

Resources:

  • Bulk Data Reference Server https://bulk-data.smarthealthit.org/
    The first bulk-data server to be created. Supports the standard completely, except for the optional “_typeFilter” parameter. Designed for client developers, the server includes a browser-based bulk-data client designed to be launched from within the bulk-data server configuration page. As a client-side app, it works only with open servers – useful for developers, presentations and demonstrations. It is also a great way to download some sample bulk data as NDJSON without having to know how the bulk-data works.
  • Bulk Data Client https://github.com/smart-on-fhir/sample-apps-stu3/tree/master/fhir-downloader
    A bulk-data command-line client written in NodeJS. Supports streaming and gzip. Includes a detailed graphical output. Supports JWKS authentication.
  • Bulk Data Tester https://bulk-data-tester.smarthealthit.org/
    Web based online tester for bulk data servers. Contains big open source test suite and test runner. Can be used online, offline as CLI or as a remote service thanks to it’s public HTTP testing API.
  • Bulk Data Tools https://github.com/smart-on-fhir/bulk-data-tools
    A NodeJS library for working with bulk data in different formats and mostly for converting the data between those formats. Some utility functions for reading directories, parsing and others are also included.