Estimate creating index on cosmos DB - mongo

Migrate mongodb 1

Hi, I migrated database from mongo-db into cosmos db.
I followed below url
https://datamigration.microsoft.com/scenario/mongo-to-cosmos?step=1

After migrating data into cosmos db, I created index on collection by query:
db.getCollection(" collection_name").createIndex({ "modified_at": 1 })
returned:
{
"createdCollectionAutomatically": false,
"numIndexesBefore": 2,
"numIndexesAfter": 3,
"ok": 1
}

my collection have 9GB data size and around 8M documents.

After create index, I check with below query:
db.collection_name.find({ "modified_at" : {$gte: "2020-08-18"} })
This query returned time-out, maybe index didn't create completely.

I checked index-progress follow url:
https://video2.skills-academy.com/en-us/azure/cosmos-db/mongodb-indexing#track-index-progress
Using: db.currentOp({"command.createIndexes": { $exists : true } })
It returned:
{
"inprog" : [],
"ok" : 1
}

It means that index created completely? I don't think so because my query still run timeout.

After 3 hours, I run find query again: db.collection_name.find({ "modified_at" : {$gte: "2020-08-18"} }), at that time, query return very fast. Maybe at that time, index created completely.

How to estimate time creating index for collection, could you help me estimate maximum time creating index finish?
Currently, I set 150k RU per second. If I want to speed up of create index, what should I do?

Anurag Sharma 17,606 Reputation points

2020-08-20T09:11:52.917+00:00

Hi @Migrate mongodb , apologies you are facing the time out issue with index creation, and thanks for providing so detailed description of the issue.

We have reached out to internal product team on the same and will update you as soon as we receive the inputs on this.
Migrate mongodb 1 Reputation point

2020-08-20T10:18:35.75+00:00

@AnuragSharma-MSFT thank you very much. If you need more any information, please let me know.
Anurag Sharma 17,606 Reputation points

2020-08-21T08:23:33.347+00:00
Hi @Migrate mongodb , Depending upon the collection size indexes could take some time to be created. One way to handle this is to create the indexes in background and continue working on collections by using {background: true} as in below query. This will not make your queries getting timed out.

db.people.createIndex( { zipcode: 1}, {background: true} )

You can refer to below thread for more details:
https://stackoverflow.com/questions/40374585/how-mongodb-background-indexing-works

Meanwhile, I am still discussing the estimated time to build the indexes with the team and will get back as soon as i get any updates.
Migrate mongodb 1 Reputation point

2020-08-21T11:37:15.207+00:00

Hi @AnuragSharma-MSFT, I want to track index progress. Why I couldn't run " db.currentOp({"command.createIndexes": { $exists : true } })" as url: https://video2.skills-academy.com/en-us/azure/cosmos-db/mongodb-indexing#track-index-progress
Anurag Sharma 17,606 Reputation points

2020-08-21T20:46:38.043+00:00

Hi @Migrate mongodb , yes you have rightly mentioned and you can run the command to track the index progress as mentioned in the article you shared.

The above command with background that was shared was related to not blocking your other commands execution (e.g. db.collection_name.find({ "modified_at" : {$gte: "2020-08-18"} })) when you are creating the indexes and would help with the issue of time out.

If possible, could you please try checking it and let us know if you face similar issue? Meanwhile as mentioned by me, we are trying to get the estimated time of index creation too.
Anurag Sharma 17,606 Reputation points

2020-08-26T07:02:02.167+00:00

Hi @@Migrate mongodb , just wanted to follow up if you were able to try creating index 'background: true' and then noticed any time out issue.

Also, I am still awaiting the response by product team on estimated time of index creation. But the article you mentioned states "Because index updates consume Request Units (RUs) at a lower priority than other database operations, index changes won't result in any downtime for writes, updates, or deletes". So if there are other operations going on, the index creation time would be impacted. Also the size of collection and mongodb overall would be other factors influencing the time too.

Share via

Estimate creating index on cosmos DB - mongo

Your answer