A FHIR server performance number is only as honest as the harness behind it. Health Samurai published an open-source benchmark on 2026-06-29 that tests Aidbox, HAPI FHIR, Medplum, and the Microsoft FHIR Server against the same hardware, the same synthetic data, and the same k6 load profile. The authoring engineer, Marat Surmashev, framed reproducibility as the headline feature, not the leaderboard. The repo and its CI rerun the suite daily, which is the part that makes the methodology hold up to outside scrutiny. For the FHIR engineering reference, the surrounding patterns sit next to this on the site.
What the Harness Actually Pins
The hardware in this run is a single bare-metal machine with 64 CPU cores and 500 GB of RAM. Each FHIR server gets the same allocation: 8 vCPU and 24 GB of memory, with Medplum running eight 1-vCPU 3-GB replicas to match its supported deployment shape. The databases are pinned too: PostgreSQL 18 for Aidbox, HAPI, and Medplum, and SQL Server 2022 Developer Edition for the Microsoft server. The load generator is Grafana k6. The dataset is Synthea, 1,000 patient records, around 2 million resources. These choices are visible in the repo, so a reviewer can argue with them on the merits rather than guess.
This level of pinning is the kind of practice that integration engineers ask for when comparing engines under load. The reasoning is the same as in the top FHIR integration engines for multi-hospital workflows shortlist: comparable numbers require comparable conditions, and comparable conditions require a script everyone can re-run.
The Reproducibility Choices Worth Calling Out
- Docker compose brings up all four servers with the same resource limits each run. The container topology is in the repo, not in a slide deck.
- k6 scripts drive CRUD, bundle import, and search workloads with the same concurrency profile against every server. Workload code lives next to the configs, which is the standard reviewers expect.
- Daily CI rerun. The benchmark is re-executed against the latest server images every day and the dashboard updates. A regression after a release does not have to wait for the next manual run.
- Open repo and license. The project is hosted at the public benchmark repo at https://github.com/HealthSamurai/fhir-server-performance-benchmark, configurations and all. Forks can adjust the workload profile and report different numbers under the same harness.
What the Harness Does Not Pretend to Do
Health Samurai is the company that builds Aidbox; the brief acknowledges that openly. The harness handles the bias question by being legible: anyone can change the resource limits, swap in a different Synthea seed, or extend the workload to a heavier dataset, then publish a fork. The current snapshot uses a 1,000-patient corpus that fits in memory, and the project notes that the next post in the series tests at scale. That is a known limitation, not a hidden one.
For the orchestration side of the same systems, the FHIR workflow orchestration 2026 reference covers the surrounding territory.
The trade-off in any benchmark is between a workload that is easy to defend and a workload that mirrors production. This repo errs toward the first, then makes it cheap to push toward the second. That is the part the team gets right.
