Issue with Custom Model Data Extraction via API in Azure Document Intelligence

AB INITIO 0

Dear Support Team,

I am currently using Azure Document Intelligence and have successfully trained a custom model that is performing well on your website. The model accurately detects and displays the values I need in a table format, which is exactly what I expected.

However, I am encountering an issue when using the API. Instead of receiving the specific data that the model was trained to detect, the API returns all the text, with options for "all lines," "all words," or "all data in table format." This is not the desired output, as I specifically trained the custom model to extract only the data I need, not all the text.

Could you please assist me in understanding why the custom model is working as intended on the website but not through the API? I would greatly appreciate any guidance or troubleshooting steps you can provide to resolve this issue.

Thank you for your support.
AB INITIO

santoshkc 7,865 Reputation points Microsoft Vendor

2024-08-28T11:09:25.5933333+00:00

Hi @AB INITIO,

Thank you for reaching out to Microsoft Q&A forum!

While your custom model is working correctly on the Azure Document Intelligence portal, the API may not be configured to extract the specific fields you trained it to recognize. Ensure that your API request is targeting the correct model ID and that the extraction mode is set to use the custom model, not default text extraction. Check that you are correctly interpreting the response by focusing on the specific fields or tables section rather than the full text output.

Retraining the model with more documents might be necessary if the API results are inconsistent or not as expected. Additional training data can improve the model’s accuracy, especially if the current dataset is limited. More diverse documents can also enhance the model’s ability to generalize and correctly identify the fields of interest.

I hope you understand! Thank you.
santoshkc 7,865 Reputation points Microsoft Vendor

2024-08-29T11:42:30.4433333+00:00

Hi @AB INITIO,

Following up to see if the given response was helpful. Thank you.
AB INITIO 0 Reputation points

2024-08-29T13:43:00.58+00:00
Hi @santoshkc

Thank you for your response!

I've implemented the API call using the document_analysis_client with my custom model. However, I'm encountering an issue where some row values are missing in the output, even though these values are correctly displayed on the Azure Document Intelligence portal.

Here's the code snippet I'm using:

from azure.core.credentials import AzureKeyCredential from azure.ai.formrecognizer import DocumentAnalysisClient from oauth2client.service_account import ServiceAccountCredentials with open(pdf_path, "rb") as file: poller = document_analysis_client.begin_analyze_document(model_id="credit-card-model-1", document=file) result = poller.result() for table in result.tables: for cell in table.cells: if current_row_index is None or cell.row_index != current_row_index: if row_data: print(row_data) row_data = "" current_row_index = cell.row_index row_data += cell.content + " "

As you can see, I'm iterating through the table cells to capture the content, but the rows appear to be incomplete when I print them out. The missing values are visible on the Azure portal, so it seems the model is recognizing them, but they aren't being included in the API response.

Could this be an issue with how the API is handling or returning the table data? Any insights on what might be causing this discrepancy and how to resolve it would be greatly appreciated.

Thank you for your help!!
AB INITIO 0 Reputation points

2024-08-29T13:44:10.0466667+00:00

@santoshkc

Also whole table data is not showing , there are more 150 row data are there in the PDF but I am getting only around 30 - 50 data but all data is getting in Azure website

santoshkc 7,865 Microsoft Vendor

Hi @AB INITIO,

Thank you for your follow-up query. I understand that you're experiencing inconsistencies with missing table data in the API response compared to the Azure Document Intelligence portal. To help resolve this, I tried reproducing the issue using a document with 150 rows of data, and I was able to extract all rows successfully using the following code:

from azure.core.credentials import AzureKeyCredential
from azure.ai.documentintelligence import DocumentIntelligenceClient
from azure.ai.documentintelligence.models import AnalyzeResult

endpoint = "https://ENFPOINT.cognitiveservices.azure.com/" 
key = "KEY" 
model_id = "model-doc"

document_intelligence_client = DocumentIntelligenceClient(endpoint=endpoint, credential=AzureKeyCredential(key))

# Make sure your document's type is included in the list of document types the custom model can analyze
with open(r"FILE_PATH", "rb") as f:
    poller = document_intelligence_client.begin_analyze_document(
    model_id=model_id, analyze_request=f, content_type="application/octet-stream"
        )
result: AnalyzeResult = poller.result()

if result.documents:
    for idx, document in enumerate(result.documents):
            print(f"--------Analyzing document #{idx + 1}--------")
            print(f"Document has type {document.doc_type}")
            print(f"Document has document type confidence {document.confidence}")
            print(f"Document was analyzed with model with ID {result.model_id}")
            if document.fields:
                for name, field in document.fields.items():
                    field_value = field.get("valueString") if field.get("valueString") else field.content
                    print(
                        f"......found field of type '{field.type}' with value '{field_value}' and with confidence {field.confidence}"
                    )

    # iterate over tables, lines, and selection marks on each page
    for page in result.pages:
        print(f"\nLines found on page {page.page_number}")
        if page.lines:
            for line in page.lines:
                print(f"...Line '{line.content}'")
        if page.words:
            for word in page.words:
                print(f"...Word '{word.content}' has a confidence of {word.confidence}")
        if page.selection_marks:
            print(f"\nSelection marks found on page {page.page_number}")
            for selection_mark in page.selection_marks:
                print(
                    f"...Selection mark is '{selection_mark.state}' and has a confidence of {selection_mark.confidence}"
                )

    if result.tables:
        for i, table in enumerate(result.tables):
            print(f"\nTable {i + 2} can be found on page:")
            if table.bounding_regions:
                for region in table.bounding_regions:
                    print(f"...{region.page_number}")
            for cell in table.cells:
                print(f"...Cell[{cell.row_index}][{cell.column_index}] has text '{cell.content}'")
    print("-----------------------------------")

Output:

User's image

I hope this helps! Thank you.

AB INITIO 0 Reputation points

2024-08-30T12:15:53.6133333+00:00
Hi @santoshkc

Thankyou for your response !!

I checked your code and modified my code based on that I am getting errors

Error 1 : TypeError: DocumentAnalysisClient.begin_analyze_document() missing 1 required positional argument: 'document
The error is due to missing of document argument

So I decided to add this argument along with arguments you mentioned but then I am getting the another error ( Error 2 )

Error 2: TypeError: azure.ai.formrecognizer._generated.v2023_07_31.operations._document_models_operations.DocumentModelsOperations.begin_analyze_document() got multiple values for keyword argument 'analyze_request'

Error 3 : TypeError: azure.ai.formrecognizer._generated.v2023_07_31.operations._document_models_operations.DocumentModelsOperations.begin_analyze_document() got multiple values for keyword argument 'content_type'

Looks like analyze_request and content_type are not valid arguments

Code :

import os from azure.core.credentials import AzureKeyCredential from azure.ai.formrecognizer import DocumentAnalysisClient from azure.ai.documentintelligence.models import AnalyzeResult from oauth2client.service_account import ServiceAccountCredentials endpoint = os.getenv('ENDPOINT') key = os.getenv('KEY') pdf_path = "C:\\Users\\***\\Downloads\\Bank of America-Credit-Card-Statement.pdf" model_id = "credit-card-model-1" document_intelligence_client = DocumentAnalysisClient( endpoint=endpoint, credential=AzureKeyCredential(key) ) with open(pdf_path, "rb") as file: poller = document_intelligence_client.begin_analyze_document( model_id=model_id, analyze_request=file, content_type="application/octet-stream" ) result: AnalyzeResult = poller.result() for table in result.tables: for cell in table.cells: if current_row_index is None or cell.row_index != current_row_index: if row_data: print(row_data) print("=====================>") row_data = "" current_row_index = cell.row_index row_data += cell.content + " "

the way I trained the model is like getting value required value into an table format, where I am getting the rows ( not full rows ) but the some rows are missing major data I needed but it is showing in website perfectly

From above 2 images ( Azure Portal Image ) the rows have values in Account column and Individual column but the comes to API , when the each cell details there is no Account and Individual data is showing

I also saw that you used word by extraction but it will give all the words in pdf file but for that I don't need train the model I can just inbuilt extraction.

Why Azure is giving correct data in their portal but giving data along other data when comes to API

For example when upload a PDF and analyze that then I am getting good results in Portal when same model used in API other text values are receiving in the response along with the data I needed , which is not accurate as in portal
AB INITIO 0 Reputation points

2024-08-30T12:16:16.7+00:00

@santoshkc
santoshkc 7,865 Reputation points Microsoft Vendor

2024-08-30T14:07:13.0266667+00:00

Hi @AB INITIO,

Thank you for your follow up query.

It appears you're facing issues with inconsistencies between the data extracted by your custom model on the Azure Document Intelligence portal and the API responses. This discrepancy is particularly concerning as it affects the reliability of the data you extract through the API. The errors you've encountered suggest that some parameters might be incorrectly used in the API call. To address this, I recommend double-checking the API documentation for the correct usage of parameters and ensuring that your API call aligns with how the model was trained to extract specific data.

If the issue persists, contacting Azure Support may help resolve any underlying problems with the API’s behavior compared to the portal.
AB INITIO 0 Reputation points

2024-09-02T10:35:40.1766667+00:00

Hi @santoshkc

Thank you for your response.

I've been using the code provided directly from the Microsoft Azure portal for our project with Azure Document Intelligence. The trained model works well when used through the Azure portal, but we're noticing differences in accuracy when trying to get the same data through the API. This inconsistency is causing issues because the data from the API isn't as reliable.

The model itself seems to be performing correctly within the portal, but the API responses aren't matching up. I’ve double-checked the API documentation to ensure we're using the correct parameters, but the accuracy differences still persist.

If there’s any known difference between how the portal and API handle data extraction, it would be helpful to know. This information is important to share with our client so that I can explain why the API isn’t working as well as the portal. We've already invested a lot of time trying different methods and training several models, so understanding this gap is crucial for deciding the project's next steps.

I appreciate your help in resolving this issue and look forward to any advice you can provide.

Best regards
AB INITIO
santoshkc 7,865 Reputation points Microsoft Vendor

2024-09-02T14:56:28.6566667+00:00

Hi @AB INITIO,

Differences in accuracy between the Azure Document Intelligence Portal and API responses can stem from several factors. Discrepancies in data formatting or document types between the Portal and API can also impact accuracy. Additionally, if you’re using a custom model, confirm that your training and validation data are the same across both platforms.

There is no known difference between how the Azure document intelligence portal and API handle data extraction. Both the portal and API use the same underlying technology to extract data from documents. However, there may be some differences in the way the data is presented or the level of control you have over the extraction process.

For example, the portal may provide a more user-friendly interface for configuring and testing the extraction process, while the API may provide more fine-grained control over the extraction process. Additionally, the portal may provide additional features or functionality that are not available through the API.

I hope you understand! Thank you.
AB INITIO 0 Reputation points

2024-09-03T05:35:17.3033333+00:00

Hi @santoshkc

Thank you for your explanation.

The reason we trained a new model was to ensure that we only receive the specific data we need. However, when using the API, I’m still getting all types of data from the document, including the data I purposely removed during the training process. This is frustrating because it defeats the purpose of having a custom-trained model.

If the API is going to return all the data anyway, I’m questioning why we invested in training a new model when I could have just used the inbuilt ones. The model works perfectly when tested on the portal, but when it comes to the API, Azure is giving back all the data, which isn’t what we need.

My client is not using the portal, so I need a solution that allows me to get data through the API just like how it appears in the portal. If Azure is performing some sort of data cleanup in the portal to ensure the results are accurate and relevant, then why isn’t that same process included in the code provided through the portal? Instead, the code from the portal gives all the raw data, and it’s up to us to clean it up to match what we see in the portal. This raises the question of why we even need to train a model in the first place if the API won’t return accurate or filtered data like the portal does.

I hope this clarifies the issues we’re facing, and I look forward to hearing how we can address them.

Best regards,
AB INITIO
santoshkc 7,865 Reputation points Microsoft Vendor

2024-09-03T08:30:46.9266667+00:00

Hi @AB INITIO,

Sorry for the trouble you are facing, I recommend raising support request through the Azure portal.
AB INITIO 0 Reputation points

2024-09-04T13:24:55.86+00:00

Hi @santoshkc

Thank you for the suggestion. I attempted to raise a support ticket directly through the Azure portal. However, I encountered an issue where Azure is requesting an upgrade to a paid support plan (starting at $29/month), even though I have the Basic plan active. I believe that all kinds of support should be included in the Basic plan, except for the Free plan. Could you please guide me on how I can get direct support from Azure under my current plan? Your assistance would be greatly appreciated.

Thank you.
AB INITIO 0 Reputation points

2024-09-06T11:02:49.7833333+00:00

Hi @santoshkc

Any suggestions ?
santoshkc 7,865 Reputation points Microsoft Vendor

2024-09-06T15:33:27.4433333+00:00

Hi @AB INITIO,

We are reaching out to the internal team to get more information and will get back to you as soon as we have an update.

Share via

Issue with Custom Model Data Extraction via API in Azure Document Intelligence

Your answer