Custom Vision Stuck on 'Training'

Hazel Erickson 1 Reputation point
2021-07-26T22:57:20.297+00:00

I cannot train any models on Custom Vision. They stay stuck on 'Training' regardless if I choose quick or advanced training, and it stays stuck even after the maximum training time. This problem started a couple days ago, and I haven't had any problems in the past training with the same data/ methods.

This is an example of the iteration details while the training job is in progress:

Iteration status: Training
Iteration Name: Iteration 1
Iteration training time (mins): 0
Iteration reserved budget time (hours): 12
Iteration training type: Advanced
Iteration classification type: None
Iteration error details: None

Yesterday I set a training job for 12 hours and it still had 0 mins for training time after 12 hours, so I had to manually delete the iteration.

I am used both the python SDK and azure custom vision GUI to start the training job, with the same results.

Azure Machine Learning
Azure Machine Learning
An Azure machine learning service for building and deploying models.
2,892 questions
Azure AI Custom Vision
Azure AI Custom Vision
An Azure artificial intelligence service and end-to-end platform for applying computer vision to specific domains.
247 questions
{count} votes

1 answer

Sort by: Most helpful
  1. romungi-MSFT 45,961 Reputation points Microsoft Employee
    2021-07-29T11:52:45.17+00:00

    @Hazel Erickson A backend issue caused training jobs to fail or they got stuck. This issue is now fixed.
    All jobs that were pending should be in failed or success state now.

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.