Hi, We are building an application that utilizes real-time object tracking. We attempted to create an object detection project for the General (compact) domain and exported it to various platforms: TensorFlow (Python) TensorFlow.js (Browser) TensorFlow.js (Node.js) However, the prediction time isn't suitable for real-time applications. It's significantly slower than the model performance described for GPU or CPU in the official documentation: https://video2.skills-academy.com/en-us/azure/ai-services/custom-vision-service/select-domain#compact-domains For a test image with dimensions of 512x512, the prediction times we get are as follows: Browser: ~2400 ms (using customvision-tfjs) Node.js: ~180 ms (using customvision-tfjs-node) Python: ~500 ms (using TensorFlow) But even with images that are about 35% smaller, the prediction times remain largely unchanged. We have a few questions: Are there any optimizations we can implement to achieve inference times more in line with the official documentation? Is Microsoft still actively developing these libraries? We've noticed that it's been nearly 2 years since the customvision-tf libraries received an update. The official tfjs library is now at version 4.10.0, yet the exported examples still rely on 3.12.0. What's the YOLO version on which the exported models are based? Any insights would be appreciated.

Custom Vision - Object Detection TensorFlow Prediction Very Slow

Alexander 5

Hi,

We are building an application that utilizes real-time object tracking. We attempted to create an object detection project for the General (compact) domain and exported it to various platforms:

TensorFlow (Python)
TensorFlow.js (Browser)
TensorFlow.js (Node.js)

However, the prediction time isn't suitable for real-time applications. It's significantly slower than the model performance described for GPU or CPU in the official documentation:

https://video2.skills-academy.com/en-us/azure/ai-services/custom-vision-service/select-domain#compact-domains

For a test image with dimensions of 512x512, the prediction times we get are as follows:

Browser: ~2400 ms (using customvision-tfjs)
Node.js: ~180 ms (using customvision-tfjs-node)
Python: ~500 ms (using TensorFlow)

But even with images that are about 35% smaller, the prediction times remain largely unchanged.

We have a few questions:

Are there any optimizations we can implement to achieve inference times more in line with the official documentation?
Is Microsoft still actively developing these libraries? We've noticed that it's been nearly 2 years since the customvision-tf libraries received an update. The official tfjs library is now at version 4.10.0, yet the exported examples still rely on 3.12.0.
What's the YOLO version on which the exported models are based?

Any insights would be appreciated.

Ramr-msft 17,736 Reputation points

2023-08-16T10:00:51.13+00:00
@Alexander Thanks for the question, Regarding your first question, there are several optimizations you can implement to achieve inference times more in line with the official documentation. Here are some suggestions:

Use a GPU: Custom Vision supports GPU acceleration, which can significantly improve inference times. You can use a GPU-enabled machine to run your inference code or use Azure Machine Learning to deploy your model to a GPU-enabled cluster.

Optimize your model: You can try optimizing your model by reducing its size, using quantization, or pruning. These techniques can help reduce the number of parameters and operations required to run your model, resulting in faster inference times.

Use a different model architecture: You can try using a different model architecture that is optimized for real-time object detection, such as YOLO or SSD. These models are designed to be fast and accurate and can help you achieve better inference times.

Is Microsoft still actively developing these libraries?. I have forwarded to the team to add to the backlog.

Regarding your third question, the exported models are based on YOLOv3-tiny. YOLOv3-tiny is a lightweight version of the YOLOv3 model that is optimized for real-time object detection on devices with limited computational resources.
Alexander 5 Reputation points

2023-08-17T21:04:43.56+00:00
@Ramr-msft , thanks for the quick response.

Our testing was done on an Apple M1 GPU, and from the latency observed on the browser, it appears that the web version is not using the GPU at all. However, even assuming we are only using the CPU, the latency of the Node.js version (the fastest in our tests) is still over 5 times what your documentation claims (35ms on CPU).

What made Azure's offering attractive to us was the promise to "Customize and embed state-of-the-art computer vision image analysis for specific domains" with "No machine learning expertise required." So, I believe having to figure out how to quantize your models to achieve the performance indicated in the documentation should not be a requirement. In addition, I hope I'm wrong but, I don't think Custom Vision has an option to quantize or change the model architecture.

Is there anything else we can do, or is Custom Vision simply not a good solution for real-time object detection?

Are there plans to update the model architecture for compact domains to a more recent version?

Thanks.
Ramr-msft 17,736 Reputation points

2023-08-31T04:49:40.8833333+00:00

@Alexander Custom vision is not optimal for real time object detection and isn’t optimal for detecting subtle differences in images. We have created backlog item to update the model architecture for compact domains to a more recent version.

Share via

Custom Vision - Object Detection TensorFlow Prediction Very Slow

Your answer