Extract Text from Images using Power App and Power Automate

  • Article

  • Introduction
  • Business Use Case
  • Test the flow
  • Summary

Introduction

AI Builder is a Power Platform capability that provides you the AI models to take leverage of artificial intelligence to optimize your business as well as automate process. We can either use the already existing prebuilt models that suits our needs to use the Power of AI in Power Apps or Power Automate or create our own custom AI models to suit some specific business needs. You can read more about AI Models in the Official documentation here. The out of the box pre built available models in AI Builder are:

  • Business card reader    
  • Category classification     
  • Entity extraction                   
  • ID reader
  • Invoice Processing          
  • Key phrase extraction    
  • Language detection              
  • Receipt processing            
  • Sentiment analysis                
  • Text recognition                    
  • Text translation      

In this article we will see how to use the AI Builder Model – Text recognition to extract text from an image and process it further

Business Use Case

We have a requirement where Field Police Officers would capture the Number plate image of speeding cars and sent is as an attachment in a mail for further processing their fine dues in the back end. We will be using the Text Recognition AI model to extract the car number from the image and create a SharePoint list item automatically via flow so that any manual effort for text extraction can be avoided.

Implementation

We will be using ‘when a new email arrives’ trigger to kickstart the flow when a speeding ticket email arrives. Followed by that we will add an apply to each condition so that we can iterate through multiple image attachments to process the text extraction.

Within the loop, we will use the Get Attachment action to extract the images which has come as email attachments. We will then add the ‘AI Builder action Recognize text in an image or a PDF document’ action to the flow and feed the ‘Content Bytes’ Dynamic output of the Get attachment action as the AI Builder Model Input.

Now we will use a compose action to extract the Vehicle number from the extracted text output which is a JSON object

We will us the below expression to fetch the Results Array and then index into the lines attribute which contains the vehicle number in the key ‘text’.

outputs('Recognize_text_in_an_image_or_a_PDF_document')?['body/responsev2/predictionOutput/results'][0]?['lines'][0]?['text']

Finally we will add the create item action to save the extracted text(obtained from the Compose Output action) as a list item along with the Image Attachment Name(corresponds to the vehicle owner)

The loop will execute as many times as the number of attachments and extract the number plate information in the multiple images and write back the records to SharePoint List.

Thus the overall flow will look like :

Test the flow

Lets test the flow by attaching multiple vehicle number images as attachment to the mail and sent it to the destination mail box which will trigger the flow.

It will loop through all the 4 attachments and extract the information as shown below

The extracted data will be written back to the SharePoint List.

Summary

Thus, we saw how to use the Text recognition AI Model to extract text from an image that was sent as a mail attachment and write it back to the SharePoint List for further processing.