Difference in Confusion Metrics between Vision Portal & using Python SDK

Rishi Khemka 60

Hi,

Ive been using the custom vision api for object detection purposes, currently im using an object detection model with Products on Shelves as the domain. Im doing this implementation (starting from creating a project) to annotation to uploading the preprocessed images along with their annotations to training & evaluating the model via python sdk in jupyter notebook.

Parallelly I am keeping a watch on the vision portal to ensure what im implementing is reflecting in vision studio. After training my model, i check the performance metrics using the following code(along with values, please find the image below)

User's image

I didnt tweak the probability or the overlap threshold even one bit, checked on the portal here are the values displayed

User's image

Can you please help me understand why is their a difference in values between studio & python sdk implementation?

santoshkc 6,310 Reputation points Microsoft Vendor

2024-05-30T11:32:05.9966667+00:00

Hi @Rishi Khemka,

Thank you for reaching out to Microsoft Q&A forum!

It's not uncommon to see differences between the performance values displayed in the Custom Vision Studio and using Python SDK. This can happen due to a variety of reasons, such as different versions of the SDK, differences in the model architecture, different preprocessing steps, or different evaluation sets.
Also, I would recommend reviewing the documentation on training and evaluating custom models in Custom Vision to ensure that you are following best practices and using consistent evaluation metrics. You can also try retraining the model with a larger dataset or adjusting the model architecture to see if that improves the performance metrics.

I hope you understand! Thank you.
Rishi Khemka 60 Reputation points

2024-05-30T12:14:33.1066667+00:00

Hi how can i check what version of SDK is the portal using ? i just did pip install for the same and upgrade it as well

how can we tweak the model architecture, i didnt tweak any parameter except for no of training iteration i.e 10 even that is default

how can preprocessing steps be different, i first preprocessed the images in my local, annotated them then uploaded it on the portal and did the training in my local itself
and then just printed the confusion metrics parameters and check studio for the same - here NO PREDICTION IS BEING DONE
santoshkc 6,310 Reputation points Microsoft Vendor

2024-05-31T07:29:50.9533333+00:00

Hi @Rishi Khemka,

Thank you for your follow-up query.

To check the version of the SDK being used in the Custom Vision portal, you can refer to: Microsoft Azure SDK for Python. It's possible that there may be differences in the way the images were preprocessed between your local machine and the portal. For example, the portal may have applied additional preprocessing steps that you did not apply locally, or vice versa.

See: Evaluation metrics.

I hope you understand! Thank you.
Rishi Khemka 60 Reputation points

2024-06-04T07:52:41.4433333+00:00

Hi, as mentioned above im just seeing the training results. I didnt train using the portal
i trained using the python sdk and same results should be reflecting in the portal
Im unable to understand the gap here
Rishi Khemka 60 Reputation points

2024-06-05T09:46:10.11+00:00

@santoshkc please respond .. my query is pretty simple
i did preprocessing, generated annotations using opensource python tool in XML format, uploaded those to the portal and then used python sdk to train
now once trained i printed the accuracy metrics in using python which was having different values than what portal was showing .. i did no change at all just wanted to see the performance during training, so without a quick test checked the performance tag on portal. Why the different results?

Also a followup question that i have to ask is, does custom vision do its own preprocessing(particularly normalization of pixel values) before doing the modelling activity?
santoshkc 6,310 Reputation points Microsoft Vendor

2024-06-05T10:59:45.5+00:00

Hi @Rishi Khemka,

Sorry for the delay in response. The differences in the performance metrics are due to differences in the evaluation sets or the way the metrics are calculated. The evaluation set used in the portal may be different from the one used in the Python SDK, or the metrics may be calculated differently.

Regarding your follow-up question, Custom Vision does perform some preprocessing on the images before training the model. The Custom Vision service uses a machine learning algorithm to analyze images. The algorithm trains the data and calculates its own accuracy by testing itself on the same images. However, the specific preprocessing steps may vary depending on the training parameters used. If you have specific preprocessing requirements, you can try preprocessing the images yourself before uploading them to the portal.

I hope you understand! Thank you.

Share via

Difference in Confusion Metrics between Vision Portal & using Python SDK