Job created by CreateMigrationJobEncrypted method fails to read xml metadata files

Robert Dudus 0 Reputation points
2024-01-16T17:23:10.7866667+00:00

Hi, I am trying to get job created by Microsoft.SharePoint.Client.Site.CreateMigrationJobEncrypted to read my metadata files. I have tried various ways but nothing works. Let me describe them and various errors I'm getting.

Method 1:

I provision containers for data & metadata and a queue (queue stores progress messages):

// ... getting context and such first

ClientResult<ProvisionedMigrationContainersInfo> containers =     clientContext.Site.ProvisionMigrationContainers();

clientContext.ExecuteQuery();

ClientResult<ProvisionedMigrationQueueInfo> queue = clientContext.Site.ProvisionMigrationQueue();

clientContext.ExecuteQuery();

Then I get encryption key from the returned containers structure.


var encryptOption = new EncryptionOption

{

    AES256CBCKey = containers?.Value.EncryptionKey,

};

Then I upload data files (they do not matter ATM as I never reach the point that they get used).

Then I upload metadata files to the container which should store metadata for import job. I also use encryptOption.AES256CBCKey to encrypt them, I follow AES CBC 256 standard, and I generate IV using System.Security.Cryptography.Aes. Then I add metadata for the generated blobs (each blob represents one xml metadata file). Metadata I add only sets IV on a blob. I also create the needed snapshots. Here's the part which sets IV.


// ... other things

// Convert IV to a string format (e.g., Base64) for storing as metadata

string ivBase64 = Convert.ToBase64String(iv);

// Set the IV as metadata

IDictionary<string, string> metadata = new Dictionary<string, string>

{

    { "IV", ivBase64 }

};

blobClient.SetMetadata(metadata);

// 3 Create snapshot for all files

blobClient.CreateSnapshot();

Then I create job and execute it:


var jobReadInfo = clientContext.Site.CreateMigrationJobEncrypted(

  Guid.Parse(targetWebId), // gWebId

  containers.Value.DataContainerUri, // azureContainerSourceUri

  containers.Value.MetadataContainerUri, // azureContainerManifestUri

  queue.Value.JobQueueUri, //azureQueueReportUri

  encryptOption);

clientContext.ExecuteQuery();

When I read the log generated by the job I get this kind of errors (a bit different each call):


[1/16/2024 7:36:58 AM] Start Time: 1/16/2024 7:36:58 AM. 

[1/16/2024 7:36:58 AM] Correlation Id: 805102a1-c02f-7000-f07b-b7c87b4b59b8  

[1/16/2024 7:36:58 AM] [Progress] Initializing Import. 

[1/16/2024 7:36:58 AM] [Progress] Downloaded SystemData.xml: SnapShotTime == ; MD5 == b7y78P4VFYpmAW7Tx79NfA==; Size == 400 bytes 

[1/16/2024 7:36:58 AM] [FatalError] [-2146232000] [System.Xml.XmlException] [Data at the root level is invalid. Line 1, position 1.]  

[1/16/2024 7:36:58 AM] [Debug]    at System.Xml.XmlTextReaderImpl.Throw(Exception e) 

   at System.Xml.XmlTextReaderImpl.ParseRootLevelWhitespace() 

   at System.Xml.XmlTextReaderImpl.ParseDocumentContent() 

   at System.Xml.XsdValidatingReader.Read() 

   at Microsoft.SharePoint.Deployment.SPImport.EnsureVersion() 

   at Microsoft.SharePoint.Deployment.SPImport.Run()

or


[1/16/2024 7:40:11 AM] Start Time: 1/16/2024 7:40:11 AM. 

[1/16/2024 7:40:11 AM] Correlation Id: af5102a1-2057-8000-099d-2916a2a0a0a5  

[1/16/2024 7:40:11 AM] [Progress] Initializing Import. 

[1/16/2024 7:40:11 AM] [Progress] Downloaded SystemData.xml: SnapShotTime == ; MD5 == Kej88c87xr7WGI1nYq9uxA==; Size == 400 bytes 

[1/16/2024 7:40:11 AM] [FatalError] [-2146232000] [System.Xml.XmlException] [Invalid character in the given encoding. Line 1, position 1.]  

[1/16/2024 7:40:11 AM] [Debug]    at System.Xml.XmlTextReaderImpl.Throw(Exception e) 

   at System.Xml.XmlTextReaderImpl.InvalidCharRecovery(Int32& bytesCount, Int32& charsCount) 

   at System.Xml.XmlTextReaderImpl.GetChars(Int32 maxCharsCount) 

   at System.Xml.XmlTextReaderImpl.ReadData() 

   at System.Xml.XmlTextReaderImpl.SwitchEncoding(Encoding newEncoding) 

   at System.Xml.XmlTextReaderImpl.ParseXmlDeclaration(Boolean isTextDecl) 

   at System.Xml.XmlTextReaderImpl.Read() 

   at System.Xml.XsdValidatingReader.Read() 

   at Microsoft.SharePoint.Deployment.SPImport.EnsureVersion() 

   at Microsoft.SharePoint.Deployment.SPImport.Run() 

[1/16/2024 7:40:11 AM] [Progress] Import did not complete. 

[1/16/2024 7:40:11 AM] Finish Time: 1/16/2024 7:40:11 AM. 

[1/16/2024 7:40:11 AM] Duration: 00:00:00 

[1/16/2024 7:40:11 AM] Byte imported: 0.00 at 0.00 bytes per second 

[1/16/2024 7:40:11 AM] Total Processed Objects: 0 

[1/16/2024 7:40:11 AM] Finished with 0 warnings. 

[1/16/2024 7:40:11 AM] Finished with 1 errors.

This looks like the import job is trying to read the encrypted by me files as plain text when parsing its XML. It should decrypt them first but it does not.

Method 2:

Now, I am thinking that maybe I don't need to encrypt these files. I dropped the encryption of xml metadata files. But I kept the rest the same. So I still call CreateMigrationJobEncrypted method and provide encryptionOption to it provided by the generated containers variable (containers?.Value.EncryptionKey).

The error I'm getting now is:


[1/16/2024 6:54:53 AM] Start Time: 1/16/2024 6:54:52 AM. 

[1/16/2024 6:54:53 AM] Correlation Id: 174f02a1-209c-7000-f07b-bf1736dce42f  

[1/16/2024 6:54:53 AM] [Progress] Initializing Import. 

[1/16/2024 6:54:53 AM] [Error] Unable to download SystemData.xml with exception 'Must specify IV in Manifest or blob metadata' 

[1/16/2024 6:54:53 AM] [Error] Unable to download Requirements.xml with exception 'Must specify IV in Manifest or blob metadata' 

[1/16/2024 6:54:53 AM] [Warning] Import requirement file Requirements.xml was not found no verifications ran. 

[1/16/2024 6:54:53 AM] [Error] Unable to download ExportSettings.xml with exception 'Must specify IV in Manifest or blob metadata' 

[1/16/2024 6:54:53 AM] [FatalError] [-2147024894] [System.IO.FileNotFoundException] [Missing Export Settings file stream]  

[1/16/2024 6:54:53 AM] [Debug]    at Microsoft.SharePoint.Deployment.SPImport.DeserializeExportSettings() 

   at Microsoft.SharePoint.Deployment.SPImport.Run() 

[1/16/2024 6:54:53 AM] [Progress] Import did not complete. 

[1/16/2024 6:54:53 AM] Finish Time: 1/16/2024 6:54:53 AM. 

[1/16/2024 6:54:53 AM] Duration: 00:00:00 

[1/16/2024 6:54:53 AM] Byte imported: 0.00 at 0.00 bytes per second 

[1/16/2024 6:54:53 AM] Total Processed Objects: 0 

[1/16/2024 6:54:53 AM] Finished with 1 warnings. 

[1/16/2024 6:54:53 AM] Finished with 4 errors. 

:54 AM] Total Processed Objects: 0 

[1/16/2024 6:53:54 AM] Finished with 0 warnings. 

[1/16/2024 6:53:54 AM] Finished with 1 errors.

Which makes me think, that I need to encrypt these files when using CreateMigrationJobEncrypted method.

Method 3:

I've decided to use CreateMigrationJob method instead of CreateMigrationJobEncrypted. It does not need encryptOption argument. I did not encrypt xml metadata files and when I attempted to created job with this method it failed (on ExecuteQuery):


var jobReadInfo = clientContext.Site.CreateMigrationJob(

  Guid.Parse(targetWebId), // gWebId

  containers.Value.DataContainerUri, // azureContainerSourceUri

  containers.Value.MetadataContainerUri, // azureContainerManifestUri

  queue.Value.JobQueueUri);

clientContext.ExecuteQuery();  // throws exception

Exception thrown is:


Error: Invalid use of SPOContainer.

I can only assume that these containers which are created for the migration require encryption as they provide us back encryption key. This method could potentially be used with self provided not encrypted containers. I did not try this way.

Conclusion:

I am not sure what to do with this except of trying my own containers. But I would like to avoid it.

Could someone here point me in the right direction?

Thanks!

Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,874 questions
Azure
Azure
A cloud computing platform and infrastructure for building, deploying and managing applications and services through a worldwide network of Microsoft-managed datacenters.
1,054 questions
OneDrive
OneDrive
A Microsoft file hosting and synchronization service.
943 questions
SharePoint Development
SharePoint Development
SharePoint: A group of Microsoft Products and technologies used for sharing and managing content, knowledge, and applications.Development: The process of researching, productizing, and refining new or existing technologies.
2,784 questions
SharePoint Server Development
SharePoint Server Development
SharePoint Server: A family of Microsoft on-premises document management and storage systems.Development: The process of researching, productizing, and refining new or existing technologies.
1,594 questions

3 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Nehruji R 4,216 Reputation points Microsoft Vendor
    2024-01-18T08:12:28.37+00:00

    Hello Robert Dudus,

    Greetings ! Welcome to Microsoft  Q&A Platform.

      I understand that you are facing issues in creating Jobs by CreateMigrationJobEncrypted method which fails to read xml metadata files.

    CreateMigrationJob

    This method creates a new migration import job and queues it up for later processing by a separate timer job. The job will consume a well formed (pre-defined format) import package that is located in the Azure Blob Storage Containers specified in this method. The SLA for migration job processing is controlled through pre-configured queue and work load throttling settings, and there's no guaranteed SLA or return time for a submitted job. SharePoint Online Import Migration API | Microsoft Learn.  

    Please refer - Troubleshoot SharePoint Migration Tool - Migrate to Microsoft 365 | Microsoft Learn which describes the detailed view of error cause and troubleshooting steps briefly.  

    Also, Please raise a Q&A thread in  https://video2.skills-academy.com/answers/topics/office-sharepoint-server-development.html SharePoint forum with appropriate tagging on particular service area. So that related team engineer will reach you shortly for better assistance/traction.  

    For more options, here are the list of SharePoint forums: https://video2.skills-academy.com/answers/topics/office-sharepoint-online.html https://video2.skills-academy.com/answers/topics/office-sharepoint-server-development.html https://video2.skills-academy.com/answers/topics/office-sharepoint-server-administration.html https://video2.skills-academy.com/en-us/answers/topics/office-sharepoint-server-customization.html.

    Hope this answer helps! Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

    0 comments
  2. Robert Dudus 0 Reputation points
    2024-01-18T20:30:11.6633333+00:00

    I realised I have added IV to the beginning of every file I encrypted. It is not needed as it is added to the blob's metadata. With it removed "Method 1" I described works now.

  3. RaytheonXie_MSFT 33,251 Reputation points Microsoft Vendor
    2024-01-19T06:31:58.1266667+00:00

    Hello Robert Dudus,

    I'm glad to hear you solve the problem ,if you have any issue about SharePoint, you are welcome to raise a ticket in this forum.

    By the way, since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others." and according to the scenario introduced here: Answering your own questions on Microsoft Q&A, I would make a brief summary of this thread:

    Issue Symptom:

    Job created by CreateMigrationJobEncrypted method fails to read xml metadata files

    Solution:

    Have added IV to the beginning of every file encrypted. After remove it the method 1 works.

    ---You could click the "Accept Answer" button for this summary to close this thread, and this can make it easier for other community member's to see the useful information when reading this thread. Thanks for your understanding!

    0 comments