Query items (container.query_items) for aio CosmosClient works for query that returns few items but fails when the number of items returned increase

Byiringiro, Nissi Fidele 0

Hello, I'm relatively new to aio.CosmosClient which is a Cosmos DB Client for Python to be used with async code.
This is the snippet of my code in one function:

async_iter = container.query_items(
    query=sql_query,
    parameters=query_params,
    partition_key=my_partition_key
)

try:
    # Iterate asynchronously
    async for item in async_iter:
        yield item
except Exception as e:
	raise Exception(f"{e}")

And I use the following code to iterate through the results in another function:

result = []
async for item in items:
    result.append(item)

The issue is that this code works when the query returns fewer items but fails with the following error when the query has to return a lot of items.
Error details: 0, message='', url=URL(url_to_my_docs)

My query should return 14,000+ items, it works when I apply a TOP 500, but fails with the error mentioned above when I remove the TOP 500.

Can anyone advise on what to do in this case?

Thank you

Oury Ba-MSFT 17,631 Reputation points Microsoft Employee

2024-06-28T21:17:15.0366667+00:00

@Byiringiro, Nissi Fidele Thank you for reaching out.

Please check if you are using the latest version of aio.comosclient. There might be new updates or fixes in the new version.

Please do also share the query that was used.

Regards,

Oury
Byiringiro, Nissi Fidele 0 Reputation points

2024-07-02T11:40:33.2666667+00:00
@Oury Ba-MSFT Hi, I have azure-cosmos==4.7.0 installed. I believe it is the latest one.
My query is

SELECT * FROM c WHERE STRINGEQUALS (c.CloudAccount.AccountId, @AccountId, true) AND c.ResourceType IN ('ec2-instance', 'ec2-image')

I'm querying the resources metadata stored in the cosmos db, where the account id is the one I passed in as a parameter, and the resource types are the ones mentioned in the query.

To be a bit specific, the query should return 22 ec2-instances, and 14,000+ ec2-images.
When I filter only on ec2-instances, things work fine, but when I add the ec2-images, then it doesn't work.

Regards,

Nissi
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Oury Ba-MSFT 17,631 Microsoft Employee

@Byiringiro, Nissi Fidele

Could you please try the below code using continuation toke.

query = "SELECT * FROM c WHERE STRINGEQUALS(c.CloudAccount.AccountId, @AccountId, true) AND c.ResourceType IN ('ec2-instance', 'ec2-image')"

 # Execute the query    
items = container.query_items(query=query, max_item_count=500, enable_cross_partition_query=True)

    # Paginate through results    
    continuation_token = None    
    result_list = []
    while True:
        pager = items.by_page(continuation_token)        
        page = await pager.fetch_next_page()
        if not page:
            break
        for item in page:
            result_list.append(item)
        continuation_token = pager.continuation_token
        # Introduce a .1-second wait here optional
        await asyncio.sleep(0.1)

    # Now result_list contains all the items
return result_list

Share via

Query items (container.query_items) for aio CosmosClient works for query that returns few items but fails when the number of items returned increase