Fire Arrow Server 2.0.0

Fire Arrow Server 2.0.0 has been released. This is a major release driven by two large bodies of work.

The first is a performance push. Coordinated load tests against production deployments, together with work to optimize real client access patterns — particularly around CarePlan Task materialization — surfaced a number of bottlenecks. This release rewrites the hot paths behind Task materialization and subscription matching and adds bounds and caching throughout.

The second is a systematic review of code quality, security, and scalability. Recent reports of language-model tooling finding previously unknown security issues in old, well-tested code prompted us to run the same kind of review across Fire Arrow Server. Despite already performing manual review and automated quality and security checks on every change, the review turned up a surprising number of issues. We have addressed them here.

Because these two efforts touch a large part of the server — authorization, logging, subscriptions, binary storage, GraphQL, HFQL, the admin proxy, scheduled jobs, and the shipped production configuration — the combined change surface is large enough that we are declaring this a 2.0.0 release. Customers are strongly advised to test 2.0.0 in a staging environment before upgrading production systems.

• (breaking) CarePlan/$materialize now reconciles by default instead of only adding Tasks; pass _reconcile=false for the previous additive behavior
• (breaking) Several defaults now fail safe: binary uploads are restricted to a content-type allow-list, multi-node API-token deployments must set an explicit signing secret, the terminology-upload and subscription-trigger operations are off by default, and the built-in MCP bridges have been removed with MCP disabled by default
• (breaking) Group/$everything, multi-search, and GraphQL now reject over-large or unsigned requests with 413/errors instead of silently truncating
• (breaking) Container health probes now target the liveness endpoint instead of the aggregate health status
• (security) Broad hardening pass across authorization, logging, GraphQL, HFQL, binary storage, and the admin proxy, including PHI redaction in logs, tighter multi-node cache and identity bounds, and injection-hardened query and error paths
• (feature) Large performance improvements to CarePlan Task materialization and subscription matching, validated with production load tests
• (feature) FHIR Binary resources are now offloaded to Azure Blob Storage, with idempotent content-addressed uploads and optional orphan-blob cleanup
• (feature) Safer production image defaults and configurable Azure client timeouts and retries
• (bugfix) CarePlan scheduling edits now preserve in-flight Tasks instead of deleting and recreating them; monthly and yearly schedules no longer drift
• (bugfix) Numerous multi-node and large-dataset correctness and reliability fixes across subscriptions, scheduled jobs, licensing, and uploads
• (maintenance) Broad internal code-quality pass and Fire Arrow database migrations that apply automatically on startup

Performance and Scalability

Load testing against production deployments identified two dominant bottlenecks: CarePlan Task materialization and subscription matching. Both have been substantially reworked.

Faster CarePlan Task Materialization

Deployments materializing many Tasks per CarePlan (for example, a plan producing one Task per day over a long horizon) saw materialization slow down under $subscribe-due-events, $renew-due-events, PUT /CarePlan, and Task CRUD. The server now batches Task database writes, orders inserts and updates so they can be sent to the database together, and reads only the Tasks that actually changed instead of rescanning the full set.

Measured against a production-realistic PostgreSQL workload (100 CarePlans, ~101 Tasks each, one-day intervals):

Scenario	Before	After	Improvement
Cold materialization (per CarePlan)	1637 ms	615 ms	2.7×
Cold materialization (wall clock)	84.4 s	35.5 s	2.4×
Reconcile after schedule drift (per CarePlan)	2245 ms	358 ms	6.3×
Reconcile after schedule drift (wall clock)	114.6 s	21.5 s	5.3×

The reconcile path in particular was rewritten to scan the CarePlan's Tasks once rather than twice. This is what made it practical to change the $materialize default (see CarePlan Task Materialization Changes below).

Faster Subscription Matching at Scale

Under HAPI FHIR's default matching, every incoming resource change is compared against all registered subscriptions one by one. At high subscription counts — the load tests exercised thousands of per-CarePlan subscriptions — this became the delivery bottleneck.

2.0.0 adds an in-memory inverted index that narrows the candidate set for simple token and reference criteria before matching, giving near-constant-time candidate lookup. In a load test with 1,000 subscriptions delivering via REST hook:

Metric	Before	After	Improvement
Delivery throughput	78/s	493/s	6.3×
Matching latency (p95)	21,186 ms	3,181 ms	6.6×

The optimization is on by default and can be disabled with fire-arrow.subscription.optimized-matching.enabled=false (SUBSCRIPTION_OPTIMIZED_MATCHING=false). It is a pre-filter only — the server still applies the same tenant/partition checks to every candidate before delivery, so matching behavior and isolation are unchanged.

Bounded Reads and Caching on Hot Paths

Several read paths that previously loaded unbounded result sets into memory now stream and page, and repeated lookups are cached:

• Authorization lookups that previously issued one database read per search result (organization hierarchy walks, general-practitioner patient sets) now batch and cache those reads.
• The Azure User Delegation Key used to sign pre-signed attachment URLs is now cached and refreshed ahead of expiry. Previously a search returning many attachments could trigger one Azure control-plane call per attachment; that per-attachment cost is eliminated (fire-arrow.binary-storage.presigned-url.delegation-key-refresh-seconds, default 300).
• CarePlan Task scans, subscription searches, and Task deletes now page rather than loading the entire set at once.

CarePlan Task Materialization Changes

CarePlan Task materialization received the most significant behavior changes in this release. The overarching goal was to stop deleting and recreating Tasks unnecessarily, so that Task identities and in-flight work survive routine schedule changes.

$materialize Reconciles by Default

CarePlan/$materialize supports a _reconcile parameter. When reconciling, the server makes the non-terminal Task set an exact match for the CarePlan's current in-window schedule:

• Tasks whose occurrence is no longer scheduled are deleted.
• Tasks whose due time moved are refreshed in place, keeping their id, status, and in-flight work.
• Newly scheduled occurrences with no Task are created.

In earlier 1.x releases this was opt-in (_reconcile=true), because reconciling was more expensive than a plain additive fill. With the reconcile path now scanning the CarePlan's Tasks only once, that cost gap is gone.

Starting with 2.0.0, $materialize reconciles by default. A call with no _reconcile parameter now cleans up orphaned Tasks and refreshes drifted due times in addition to adding missing Tasks.

If you run a $materialize tick purely to roll the horizon forward and rely on the previous add-only behavior, pass _reconcile=false to opt back into the pure additive path.

Scheduling Edits Preserve In-Flight Tasks

Previously, editing a CarePlan's schedule (or a $subscribe-due-events / $renew-due-events call) deleted all non-terminal Tasks and recreated them. Any Task a user had already started could be discarded and replaced, and Task ids churned on every edit.

The subscribe, renew, and CarePlan-update paths now reconcile at the occurrence level instead. A timing change refreshes the affected Tasks in place; removing an activity deletes only the Tasks for that activity. Tasks that are still valid keep their id, and in-flight Tasks are never discarded.

Two consequences worth noting for integrators:

• The reconcile after a CarePlan edit runs asynchronously, just after the update commits. There is a brief window where a stale Task can still be visible before reconcile completes, whereas the old wipe happened synchronously during the write.
• There is no automatic reconcile from the background sweeper. If a reconcile cannot run (for example, it cannot acquire its per-CarePlan lock within the timeout and exhausts its retries), it is recovered by the next CarePlan edit, the next subscribe/renew, or an explicit $materialize?_reconcile=true.

Idempotent, Multi-Node-Safe Materialization

Task materialization is now idempotent across nodes. Each occurrence maps to a deterministic Task id derived from the CarePlan and occurrence, so a retry or a race between two nodes updates the same Task instead of creating a duplicate. Subscription delivery notifications now also carry a stable notificationId (and an idempotency key on minimized payloads), so webhook consumers can deduplicate a resource-change event that is delivered more than once.

Rematerialization on Timezone Change

Editing a CarePlan's resource-level timezone extension now triggers rematerialization immediately. Previously a timezone-only edit was picked up only later, when some other change or the background sweeper next reconciled the plan, so due times could remain in the old timezone in the meantime.

Calendar-Correct Monthly and Yearly Schedules

Monthly and yearly schedules no longer drift. Previously a "monthly" schedule advanced by a nominal 30 days and a "yearly" schedule by 365 days, so occurrences slipped by roughly five days per year and a 12-occurrence monthly plan could be truncated before delivering all twelve. Monthly and yearly recurrences now step by calendar month and year, so a plan scheduled for the 15th stays on the 15th and delivers every occurrence.

Stricter Schedule Validation and Backpressure

Two smaller behavior changes affect misconfigured or heavily loaded deployments:

• An unrecognized Task status in configuration (initial-status, due-status, or terminal-status) now fails fast the first time it is used, instead of silently defaulting to requested and producing Tasks in the wrong state.
• When the CarePlan-events async executor is saturated, the default is now to apply backpressure to the submitting thread rather than dropping the event. The previous fail-fast behavior can be restored with fire-arrow.careplan-events.async.rejection-policy=abort.

Security and Authorization Hardening

The security and code-quality review touched most subsystems. The changes below are grouped by the surface they affect. None of them change the authorization model itself; they close gaps where the server could behave more loosely than intended, leak identifiers into logs, or accept malformed input.

Protected Health Information in Logs

Two diagnostic log paths could emit direct patient identifiers. Response logging recorded full request URLs and search parameters — including values such as identifier (MRNs), subject, and _id — and the authorization timing trace recorded short string arguments in full at INFO.

Both now redact direct-identifier search parameters and identifier values in routine logs while preserving non-identifying parameters (_count, _sort, _include, and similar) so the logs remain useful for diagnosing search and paging behavior. As part of the same change, diagnostic logging now runs regardless of whether authentication is enabled, so it is available in unsecured development deployments too.

A separate fix removed an incidental access filter that had been folded into GraphQL response logging. Compartment narrowing for GraphQL already happens at the search layer for all request and resource types; the logging-layer filter was incomplete and is now gone, leaving response logging as pure diagnostics.

Multi-Node Cache and Identity Bounds

Fire Arrow Server's authorization caches are per-node. On a multi-node deployment, a change on one node is reflected on other nodes only after their cached entries expire. That behavior is now the explicit, documented contract, and the identity cache's expiry was reduced from 600 seconds to 120 seconds so the cross-node window is uniform and short. The server warns at startup if either cache expiry is configured above 300 seconds.

Injection-Hardened Query and Error Paths

Several places that built structured output by string concatenation now use a proper encoder, and query endpoints that accept caller-supplied expressions were tightened:

• HFQL (the SQL-like FHIR query interface) no longer rewrites queries with regular expressions. Queries are parsed into a syntax tree, narrowing is applied to that tree, and the result is re-serialized with correct value escaping. Configurable complexity caps reject pathological queries (fire-arrow.hfql.max-where-clauses, max-in-list-size, max-select-clauses), and continuation predicates are validated to prevent expression injection between pages.
• GraphQL now enforces a maximum query depth and complexity (fire-arrow.graphql.max-query-depth, default 12; max-query-complexity, default 200) and a maximum result size, and rejects a query that would fan out into too many sub-searches. Pagination cursors are now signed; see Upgrade Notes.
• Error and $me responses that previously interpolated caller-influenced values into hand-built JSON now build that JSON with a proper encoder, so a crafted value can no longer break out of or inject fields into the response.

Denial-of-Service Caps

Read paths that could be driven to exhaust memory now have explicit ceilings. Group/$everything and the sorted multi-search path now fail with 413 Payload Too Large and an OperationOutcome when a result set exceeds their configured caps, instead of silently truncating (fire-arrow.paging.group-everything-max-patients, default 500; fire-arrow.paging.sorted-window-cap). A _total=accurate count is now bounded by a configurable row ceiling (fire-arrow.authorization.multisearch.accurate-total-row-ceiling, default 100000), and multi-search fan-out is capped (max-sub-searches, default 16).

API Token and Locking Robustness

• API-token validation is now cached (keyed by a hash of the token, never the plaintext), and token roll uses a conditional update so a concurrent roll cannot orphan a token. Privilege-dropped tokens are distinguished from full tokens at the authentication layer.
• The distributed-locking subsystem was hardened for multi-node use: identity-protection locks are released reliably even when a request is rejected before commit, the JDBC lock time-to-live is now configurable (fire-arrow.mutex.jdbc.lock-ttl-seconds, default 300), and a deployment that configures identity protection with a JVM-local lock provider is now rejected at startup rather than reporting protection as enabled while nodes race.

Admin Proxy Hardening

The backend-for-frontend proxy that serves the administration UI now streams request and response bodies rather than buffering them in memory, normalizes request paths with a standards-based routine that rejects encoded path-traversal attempts, verifies the access token signature before deriving the admin hint from it, and matches the admin scope exactly instead of by substring.

Subscription Payload Safety

Subscription payload minimization — which strips protected health information out of the Task delivered to webhook consumers — now fails the delivery if it cannot confirm the minimized payload took effect, rather than logging a warning and delivering the original full payload. It also now preserves the source basedOn reference (for example, the originating RequestGroup or ServiceRequest) on minimized Tasks, so hierarchical-completion consumers can correlate a webhook Task with its immediate source without an extra read.

Binary Storage

Binary storage gained a new capability and a round of hardening.

FHIR Binary Resources Offloaded to Azure

Large FHIR Binary resources are now offloaded to Azure Blob Storage through HAPI FHIR's binary-storage extension point, the same as attachment content referenced by Attachment.url. Previously Binary content was stored inline in the database, which did not scale. This activates when fire-arrow.binary-storage.enabled=true and HAPI's own binary storage is left off. Existing inline Binary resources continue to read normally; only new content above the size threshold is written to Azure, under a blob layout kept separate from Fire Arrow's own attachment layout. No migration is required.

Idempotent Uploads and Orphan Cleanup

Uploads are now content-addressed: the stored blob is named from the server-computed hash of the uploaded bytes, so retrying an identical upload returns the same URL rather than creating a second copy. A new, opt-in orphan-blob garbage collector can reconcile stored blobs against the database and reclaim confirmed orphans. It is off by default, and dry-run by default when enabled, so it reports candidate deletions before removing anything (fire-arrow.binary-storage.gc.*).

Streaming Uploads and Bounded Azure Calls

Both upload paths now stream to Azure without materializing the full payload in memory, and the declared size is checked before the body is read, so a multi-gigabyte upload no longer risks running the server out of memory before validation runs. The Azure client now has bounded request timeouts and retries (fire-arrow.binary-storage.azure.client.*), so a slow or unreachable storage backend can no longer hold a request thread indefinitely.

Tenant Isolation, Ownership, and Content-Type Validation

For deployments that enable tenant partitioning, blobs are now stored under a tenant-scoped path, and reversing a pasted pre-signed URL now verifies that the URL belongs to the resource being saved. Uploaded content types are now validated against an allow-list with magic-byte sniffing; see Upgrade Notes for the default change. Pre-signed URLs can optionally be bound to the requesting client IP (fire-arrow.binary-storage.presigned-url.bind-to-ip).

Operational and Deployment Changes

Safer Production Image Defaults

The shipped Azure production configuration and image were hardened so the server is safe to run without a long list of undocumented overrides. Notable changes include a larger default database connection pool (HIKARI_MAXIMUM_POOL_SIZE, default 30), a per-instance scheduler identity derived from the hostname, JVM heap sizing relative to the container memory limit with exit-on-out-of-memory, capped search prefetch and page sizes, and clearer warnings on settings that are unsafe for multi-node use. The terminology-upload and subscription-trigger operations are now off by default; see Upgrade Notes.

Health Probes Target Liveness

The container health check previously probed the aggregate /actuator/health endpoint, whose status reflects the worst of all indicators. A license-expiry warning or a transient database blip could therefore flip liveness down and cause the orchestrator to restart-loop every pod. The health check now targets the liveness endpoint by default; see Upgrade Notes for the probe-command change.

Multi-Node Scheduled Jobs and Tenant Scoping

The CarePlan due, expiry, and subscription sweepers now run once per tenant partition in partitioned deployments (previously they only processed the default partition), serialize per tenant so one busy tenant cannot starve the others, and hold their distributed lock until their transaction commits so a second node cannot redo work a first node already committed. A cluster configured for distributed locking that is missing its lock registry now refuses to start rather than silently running jobs on every node.

MCP Removed and Disabled by Default

The custom Fire Arrow MCP bridges have been removed, and MCP is disabled by default across all shipped profiles. See Upgrade Notes.

Bug Fixes

CarePlan completion checks could roll back a Task update

The CarePlan completion check ran inside the user's Task-update transaction. A failure in that secondary check could mark the transaction for rollback, so a legitimate Task PATCH/PUT was rejected because of a problem in the completion logic rather than the write itself. The completion check now runs after the Task update has committed, in a separate transaction, so it can no longer roll back the user's write.

License expiry timer could stop permanently

An unexpected error in the license expiry timer task silently killed the timer, so no license-termination action would ever fire afterward. The timer now catches such failures, records them, and reschedules itself, so it stays alive.

Slow identity provider could hang startup

The token decoders used unbounded network timeouts when fetching identity-provider discovery documents, so a slow or hanging provider could block server startup indefinitely. Discovery requests now use bounded connect and read timeouts.

Polling-only subscriptions could stall startup

Polling-only ("nop") CarePlan-event subscriptions could enter a retry storm and stall startup on deployments that do not use the Azure Storage Queue channel. These are now drained without requiring the queue. A related opt-in setting reindexes the search index on startup when it is found empty (fire-arrow.search.reindex-on-startup-when-index-missing, default false), for deployments on ephemeral storage.

Subscription delivery could hang the dispatcher

A hung send to Azure could pin HAPI's subscription dispatcher indefinitely. Sends now use an explicit timeout (send-timeout-seconds, default 10); on timeout the delivery is treated as a transient failure and retried, with the stable notificationId keeping the retry safe for consumers to deduplicate.

Dependency Upgrades and Maintenance

This release includes the internal code-quality and security-review pass described above, along with a number of correctness fixes to internal utilities (schedule expansion bounds, numeric overflow handling, and recursion limits) that protect against pathological input.

Two Fire Arrow database migrations are included and apply automatically on startup when Flyway is enabled: one adds a version column to the subscription delivery-state table, and one adds a partition column to the composite-search table (relevant only when tenant partitioning is enabled). No shared HAPI FHIR schema migration is required for this release.

Upgrade Notes

Because of the breadth of this release, please test in staging before upgrading production. The items below require action or a conscious decision on upgrade.

Change	What to check or do
$materialize reconciles by default	A call with no _reconcile parameter now deletes orphaned Tasks and refreshes drifted due times in addition to adding missing ones. If a horizon-rolling tick relied on add-only behavior, pass _reconcile=false.
Binary upload content-type allow-list	Uploads are now restricted to a default allow-list (image/png, image/jpeg, image/gif, image/webp, application/pdf, text/plain). Deployments that relied on accepting any content type must set fire-arrow.binary-storage.allow-all-content-types=true.
Binary-upload authorization wildcard	A binary-upload authorization rule with resource: "*" is now rejected at startup. Enumerate the concrete resource types binaries may attach to.
Multi-node API-token signing secret	A clustered deployment that enables API tokens without an explicit signing secret now fails fast at startup. Set API_TOKEN_HMAC_SECRET (API tokens are off by default, so this only affects deployments that opt into them on multiple nodes).
Terminology upload / subscription trigger off by default	The $upload-external-code-system and $trigger-subscription operations are no longer registered by default. Set terminology_uploader_enabled / subscription_triggering_enabled to re-enable them.
MCP removed and off by default	The built-in Fire Arrow MCP bridges are removed and MCP is disabled by default. Opt in per environment with spring.ai.mcp.server.enabled=true (using the upstream HAPI MCP surface).
GraphQL pagination cursors are signed	Existing unsigned cursors are rejected; clients simply restart pagination. Set an explicit fire-arrow.graphql.cursor-hmac-secret in production so cursors survive restarts.
Group/$everything and multi-search caps	These now return 413 when a result set exceeds the configured cap instead of silently truncating. Direct very large cohorts to bulk $export, or raise the caps (fire-arrow.paging.group-everything-max-patients, sorted-window-cap).
Health probe target	The container health check now targets the liveness endpoint. Update probe commands to HealthCheck liveness and HealthCheck readiness; no-argument calls keep working but now hit the liveness endpoint rather than the aggregate status.
Unknown Task status now fails fast	A misconfigured initial-status / due-status / terminal-status now fails at first use. Confirm these config values are valid FHIR Task statuses.
Test authentication provider gating	The test auth provider (which skips JWT signature validation) can now be enabled only via the -Dfire-arrow.allow-test-auth-provider=true JVM flag. Normal deployments are unaffected.

Aside from the items above, no further action is required.