NHS Healthcare
Microsoft Fabric
2025
9 min read
NHS Integrated Care Board Microsoft Fabric Real-Time Patient Data: 8-Hour Lag to 20 Minutes
Sector
NHS Healthcare
Region
Midlands, England
Population Served
approx. 1.2 million
Engagement
24 Weeks
Numlytics built NHS Microsoft Fabric real-time patient data infrastructure for an NHS Integrated Care Board in the Midlands - migrating all 22 Azure Data Factory pipelines (including 5 completely undocumented ones) to Microsoft Fabric, implementing near-real-time ingestion via Fabric Eventstream connected to the EPIC HL7 FHIR interface, and reducing patient data lag from 8 hours to under 20 minutes for clinical bed management operations.
The Challenge: Clinical Teams Making Real-Time Decisions on Eight-Hour-Old Data
The ICB had invested seriously in its data infrastructure over three years, building 22 Azure Data Factory pipelines ingesting patient pathway data from EPIC, the regional NHS Spine, an estates management platform, a workforce scheduling tool, and a locally developed SQL Server data warehouse. The setup had served the organisation reasonably well during a period of lower demand. As winter pressure activity intensified, the limitations became dangerous.
-
Eight-hour patient data lag: All 22 pipelines ran on overnight batch schedules. By the time bed managers and pathway coordinators arrived, data in their Power BI dashboards was already 8 hours old and growing less useful throughout the day, a genuine operational risk during busy winter periods
-
Five completely undocumented pipelines: Built by a contractor who left in 2022 with no documentation and no monitoring. When one failed silently for eleven days without detection, it was only discovered when a senior clinician noticed waiting time figures had not changed across two consecutive weekly reports; the ICB had been reporting incorrect performance data to the board during that period
-
Four-week ad-hoc reporting backlog: The data team of five was overwhelmed with SQL query requests from clinical and operational teams. Directorate managers were routinely making workforce and resource decisions using unofficial Excel trackers rather than waiting for data team support
-
No real-time clinical operations capability: Bed managers had no live view of occupancy, discharge notifications, or admission status - all operational decisions were made on yesterday's numbers
One undocumented pipeline had been failing silently for eleven days before anyone noticed. It was only discovered when a senior clinician observed that waiting time figures had not changed across two consecutive weekly reports. The ICB had been reporting incorrect performance data to the board during that period. The silent failure scenario had to be structurally eliminated, not patched.
The Numlytics Approach:
Numlytics designed this NHS Microsoft Fabric real-time patient data programme as a five-phase engagement, beginning with a complete audit of all 22 pipelines before any architecture decisions, and ending with self-service analytics for 160 clinical and operational staff.
-
Pipeline Audit and Risk Assessment (Weeks 1–6)
A complete audit of all 22 ADF pipelines, which took longer than estimated because of the five undocumented ones. Reverse-engineering required SQL Server query tracing, Azure Monitor log analysis, and interviews with engineers who had worked alongside the original contractor. By week six, Numlytics had a complete dependency map, a failure mode analysis for each pipeline, and a prioritised migration plan. The five undocumented pipelines were designated for full rebuild rather than migration, the only responsible approach when there is no documentation to validate against.
-
Microsoft Fabric Lakehouse Architecture - OneLake Medallion (Weeks 7–13)
Numlytics designed and built the target Microsoft Fabric Lakehouse on OneLake using a three-tier Medallion structure. The Bronze layer handles raw ingest from all six source systems. The Silver layer applies clinical data quality rules, handles SNOMED code mapping and ICD-10 classification, and produces conformed patient pathway records. The Gold layer contains reporting-ready aggregates: waiting time calculations, bed occupancy rates, referral-to-treatment pathways, and workforce utilisation metrics. A data quality validation step at the Silver boundary flags incomplete or anomalous records before they reach reporting.
-
Near-Real-Time Ingestion via Fabric Eventstream (Weeks 12–17)
For the four highest-priority operational data streams, bed occupancy, A&E attendance, inpatient admissions, and discharge notifications - Numlytics implemented real-time ingestion using Fabric Eventstream connected to the EPIC HL7 FHIR interface. These streams now land in the Bronze Lakehouse layer within two minutes of the source event. Clinical staff see bed status changes, discharge completions, and new admissions in their dashboards within 20 minutes, compared to the previous 8-hour batch lag. Implementing EPIC HL7 FHIR integration required close collaboration with the ICB's clinical systems team over two additional weeks.
-
Pipeline Migration and Rebuild (Weeks 13–20)
The 17 documented pipelines were migrated to Fabric Data Factory with improved scheduling, monitoring dashboards, and automated alerting for failures. The 5 undocumented pipelines were rebuilt from scratch using the dependency maps produced during discovery - with full unit testing and documentation before deployment. All 22 pipelines now have health dashboards in Fabric, failure alerts routed to the data team, and a monthly pipeline review process. The eleven-day silent failure scenario has been structurally eliminated.
-
Self-Service Analytics and Clinical Training (Weeks 20–24)
Numlytics designed a governed self-service analytics layer on the Gold Lakehouse tier. Certified semantic models with directorate-level row-level security allow clinical and operational teams to explore data within their own domain without data team support. Eight half-day training workshops were delivered across clinical, commissioning, finance, and workforce teams. Copilot was enabled for the ICB senior leadership group using endorsed semantic models as the sole authorised data source.
The Results
20 min
Patient Data Latency
Was an 8-hour overnight batch - bed managers working from live operational data
22
Pipelines Migrated
All migrated or rebuilt, 5 undocumented pipelines rebuilt with full documentation
62%
Ad-Hoc Requests ↓
Within 8 weeks of self-service analytics launch - 4-week backlog cleared
160
Self-Service Users
Clinical and operational staff independently using analytics - Excel trackers eliminated
"Before this project, our bed managers were starting every shift looking at data that was eight hours old. During a busy winter period, that is a genuinely dangerous gap. The Eventstream work Numlytics delivered changed the operational picture completely. Within six weeks of go-live, our site teams were making discharge and admission decisions based on data that was less than twenty minutes old. The undocumented pipeline situation was also resolved properly, not just patched. We now have complete confidence in what is running on our infrastructure and why."
— Director of Data and Digital, NHS Integrated Care Board, Midlands