The brief
One of the first projects I worked on was aimed at developing a Proof of Concept for a Health Data Service. Commissioned by the Department of Health, the work sought to determine whether it was feasible—both operationally and technologically—to link multiple health datasets and provide secure, streamlined access for researchers across the United Kingdom.
The scope of the challenge was substantial. The data landscape included national assets such as ONS Mortality data and NHS Hospital Episode Statistics, alongside regional datasets held by NHS Trusts and Screening Programmes. Our mandate comprised two central objectives:
- To design and validate a governance framework for accessing health data in the UK
- To complete a proof of concept which demonstrates the viability of the service
Establishing a Governance Framework
Accessing health data requires the highest standards of scrutiny, encompassing information governance, security requirements, ethics approvals, and scientific review. Rather than approaching these domains as procedural hurdles, we engaged with each corresponding body to understand their priorities, constraints, and existing processes.
We conducted a structured series of discovery sessions, during which representatives were invited to articulate what mattered most within their remit. These discussions helped us map current pain points, clarify governance expectations, and introduce the principles of a more scalable, future-oriented model. Through sustained engagement, careful analysis, and iterative refinement, we secured consensus from all three oversight bodies on a governance framework that balanced accessibility with the necessary level of scrutiny.
Understanding Data Requirements
In parallel, we conducted workshops with data owners and researchers to develop a detailed understanding of user needs. These sessions focused on clarifying how datasets were collected and maintained, the types of transformations required, and the output formats and refresh patterns necessary to support research activity.
This work resulted in the development of comprehensive data catalogues and the design of preliminary ETL processes. These artefacts served as critical foundations, ensuring that subsequent technical work aligned with both the practical realities of the data and the operational needs of researchers.
Building the Data Service
The technical development phase centred on designing and prototyping the data service itself. Adopting an agile delivery approach, we moved rapidly through cycles of design, testing, and refinement. This allowed us to identify limitations early and validate solutions directly with stakeholders.
Using the data catalogues created earlier, we tested and validated the ETL processes required to ingest data consistently. Once datasets were successfully landed, we implemented the matching processes needed to link them—supported by the governance rules and ETL frameworks established at the outset. The final stage involved securely transferring the linked datasets to researchers, enabling them to conduct their analyses with confidence in the provenance and integrity of the data.
Results and Impact
The Proof of Concept achieved its objectives and received strong positive feedback from stakeholders. It demonstrated that both the governance model and the technical architecture were viable, and it played a significant role in shaping the subsequent business case for additional investment. Ultimately, the work provided a foundation for scaling national health data services, broadening their scope and enhancing their contribution to public health research across the UK.
Want to talk about your data service?