Low or No Code Picture Identification solution recommendation and tutorial

Karyn Close (Work) 5

I have a repository of safety data sheets where I can and have extracted text and used the information the way I want to. However, each sheet contains some images I'm looking to extract. Example: Pictogram with red triangle that depicts some sort of hazard warning. I'm looking to extract those images and place them on a label for the hazardous component. If you google the type of sheets I'm referring to, you'll see what I mean. They are standard.

I've looked at Cognitive services (Computer Vision & Custom Vision) to get this done but can't seem to figure out the solution and a few paths lead me down a coding path which I'm trying to avoid.

I'm either not connecting the dots on how it works or just don't have the right knowledge to get it done.

To be clear, I am not looking to extract text from an image. I'm looking to understand that the document contains the image, which image(s) the document has, and then create a label that contains the found image(s).

Since there are only 9 standard images, I could store them somewhere. But, what would be the solution to match them up. There is no text on these images. The safety sheets could also contain other images that have nothing to do with what I'm looking for. Also, I have 10s of thousands of these documents.

Can anyone help me?

romungi-MSFT 45,961 Reputation points Microsoft Employee

2023-06-21T13:37:01.0733333+00:00

@Karyn Close (Work) Custom vision projects can be trained to identify an object or classify an image based on labels used during training. In this case if you are looking to identify an object in an image with a no code method then the custom vision studio will help you to train a model and you can use the model to be deployed as a prediction endpoint to pass images or image URLs to identify any of your trained labels.

Since, you are using documents which could be in a word or pdf format the input needs to be pre-processed to any of the supported image formats rather than passing an entire document and trying to classify the images in the document.

A general use case scenario for your scenario for thousands of documents would be to develop an app which uses the custom vision endpoint where in the app takes an input of a document and extracts images which are then passed to the API to classify each image. I hope this helps!!
Karyn Close (Work) 5 Reputation points

2023-06-21T14:39:08+00:00

@romungi-MSFT I did investigate customvision.ai and didn't get anywhere with it. I guess I'm not understanding. I was able to create a project and train it but then had no idea where to go with that information. In Power Automate, there don't seem to be many options to utilize what I've created.

The documents being processed are solely PDF and not Word. Just like the text I have scraped from the PDFs, I want to grab those specific images as described and no other images.

I understand that the images as a whole from the PDF would need to be evaluated and that is where I can't find any tutorials or articles explaining how to do this.

I understand your last paragraph. And, that is what I am in search of in terms of a tutorial or explanation. Anything I have found online has been partial at best and I'm just going through the motions with those pieces of information and not understanding the "why" behind what I'm doing which doesn't help with understanding. That is why I'm reaching out with this post. I thought that I would be able to configure the image retrieve in basically the same manner as I did the text but that doesn't appear to be the case.

The other challenge I faced with Custom Vision is that it requires 5 images per tag and I can't seem to upload the same photo multiple times. There are only 9 industry images and each image has a variety of classifications. There aren't 5 images per classification/tag. So, this is prohibiting me from training. Is there a way around this?

Still in need of help and understanding.
romungi-MSFT 45,961 Reputation points Microsoft Employee

2023-06-23T09:39:00.6666667+00:00

@Karyn Close (Work) With respect to custom vision, 5 images per tag is a requirement to train the model and usually the images used for training are images of an object or group of objects with different views or angles. For example, for object detection to detect a car in an image, you can upload different images of cars of various sizes, brands, moving, stationary etc. In your case if you have just a limited set of images you could try to edit that image and save it with different edits and upload them with different names. This is just a workaround and not a recommended way to add images for a tag.

W.r.t power automate, I do not have much experience of using it. So, I hope other users from community could help on it. Thanks!!

Share via

Low or No Code Picture Identification solution recommendation and tutorial

Your answer