All queries from from FHIR server to Cosmos DB failing

Emma 0 Reputation points
2024-05-08T06:10:25.21+00:00

We are running an instance of the Azure FHIR Server using Cosmos DB. Since at least 12 hours ago, all queries to the FHIR server are failing with the following exception and trace:

2024-05-08T05:41:26.650255158Z fail: Microsoft.Health.Fhir.CosmosDb.Features.Storage.CosmosFhirDataStore[0]
2024-05-08T05:41:26.650270158Z CosmosFhirDataStore. 'Search Parameters' failed. Elapsed: 00:00:00.0006074. Memory: 186966472.
2024-05-08T05:41:26.650273858Z Microsoft.Health.Fhir.Core.Exceptions.InitializationException: Failed to initialize search parameters
2024-05-08T05:41:26.650277358Z at Microsoft.Health.Fhir.Core.Features.Definition.SearchParameterDefinitionManager.EnsureInitialized() in /repo/src/Microsoft.Health.Fhir.Core/Features/Definition/SearchParameterDefinitionManager.cs:line 106
2024-05-08T05:41:26.650280958Z at Microsoft.Health.Fhir.Core.Features.Definition.SearchParameterDefinitionManager.GetSearchParameters(String resourceType) in /repo/src/Microsoft.Health.Fhir.Core/Features/Definition/SearchParameterDefinitionManager.cs:line 112
2024-05-08T05:41:26.650284658Z at Microsoft.Health.Fhir.Core.Features.Definition.SearchableSearchParameterDefinitionManager.GetSearchParameters(String resourceType) in /repo/src/Microsoft.Health.Fhir.Core/Features/Definition/SearchableSearchParameterDefinitionManager.cs:line 49
2024-05-08T05:41:26.650310959Z at Microsoft.Health.Fhir.Core.Features.Conformance.CapabilityStatementBuilder.SyncSearchParamsAsync(String resourceType) in /repo/src/Microsoft.Health.Fhir.Core/Features/Conformance/CapabilityStatementBuilder.cs:line 185
2024-05-08T05:41:26.650314759Z at Microsoft.Health.Fhir.Core.Features.Conformance.CapabilityStatementBuilder.SyncSearchParametersAsync() in /repo/src/Microsoft.Health.Fhir.Core/Features/Conformance/CapabilityStatementBuilder.cs:line 348
2024-05-08T05:41:26.650318059Z at Microsoft.Health.Fhir.CosmosDb.Features.Storage.CosmosFhirDataStore.Build(ICapabilityStatementBuilder builder) in /repo/src/Microsoft.Health.Fhir.CosmosDb/Features/Storage/CosmosFhirDataStore.cs:line 806
2024-05-08T05:41:26.650655764Z fail: Microsoft.Health.Fhir.Core.Features.Conformance.SystemConformanceProvider[0]
2024-05-08T05:41:26.650665064Z SystemConformanceProvider: Failed running 'Microsoft.Health.Fhir.CosmosDb.Features.Storage.CosmosFhirDataStore' when building a new CapabilityStatement.
2024-05-08T05:41:26.650668964Z Microsoft.Health.Fhir.Core.Exceptions.InitializationException: Failed to initialize search parameters
2024-05-08T05:41:26.650672264Z at Microsoft.Health.Fhir.Core.Features.Definition.SearchParameterDefinitionManager.EnsureInitialized() in /repo/src/Microsoft.Health.Fhir.Core/Features/Definition/SearchParameterDefinitionManager.cs:line 106
2024-05-08T05:41:26.650675665Z at Microsoft.Health.Fhir.Core.Features.Definition.SearchParameterDefinitionManager.GetSearchParameters(String resourceType) in /repo/src/Microsoft.Health.Fhir.Core/Features/Definition/SearchParameterDefinitionManager.cs:line 112
2024-05-08T05:41:26.650678965Z at Microsoft.Health.Fhir.Core.Features.Definition.SearchableSearchParameterDefinitionManager.GetSearchParameters(String resourceType) in /repo/src/Microsoft.Health.Fhir.Core/Features/Definition/SearchableSearchParameterDefinitionManager.cs:line 49
2024-05-08T05:41:26.650682665Z at Microsoft.Health.Fhir.Core.Features.Conformance.CapabilityStatementBuilder.SyncSearchParamsAsync(String resourceType) in /repo/src/Microsoft.Health.Fhir.Core/Features/Conformance/CapabilityStatementBuilder.cs:line 185
2024-05-08T05:41:26.650686065Z at Microsoft.Health.Fhir.Core.Features.Conformance.CapabilityStatementBuilder.SyncSearchParametersAsync() in /repo/src/Microsoft.Health.Fhir.Core/Features/Conformance/CapabilityStatementBuilder.cs:line 348
2024-05-08T05:41:26.650689565Z at Microsoft.Health.Fhir.CosmosDb.Features.Storage.CosmosFhirDataStore.Build(ICapabilityStatementBuilder builder) in /repo/src/Microsoft.Health.Fhir.CosmosDb/Features/Storage/CosmosFhirDataStore.cs:line 806
2024-05-08T05:41:26.650692965Z at Microsoft.Health.Fhir.Core.Features.Conformance.SystemConformanceProvider.GetCapabilityStatementOnStartup(CancellationToken cancellationToken) in /repo/src/Microsoft.Health.Fhir.Core/Features/Conformance/SystemConformanceProvider.cs:line 146


The response content is the following:

{
    "resourceType": "OperationOutcome",
    "id": "ee3a3476-4e9c-4f28-a960-699f23b1cab6",
    "meta": {
        "lastUpdated": "2024-05-08T05:51:35.2237241+00:00"
    },
    "issue": [
        {
            "severity": "error",
            "code": "unknown",
            "diagnostics": ""
        }
    ]
}


We are not sure if the issue lies with the FHIR server instance or the connection with the Cosmos DB instance.

We have 3 environments with the same setup and all of them have the same issue. We have not deployed any updates or made any changes that could explain these issues.

Azure Cosmos DB
Azure Cosmos DB
An Azure NoSQL database service for app development.
1,533 questions
Azure Health Data Services
Azure Health Data Services
An Azure offering that provides a suite of purpose-built technologies for protected health information in the cloud.
155 questions
{count} votes