AzCopy log file for MD5 mismatch

Hello Zeno GH,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you seek to understand how an MD5 mismatch during the download process will be logged in the AzCopy log file, specifically asking whether it will be categorized under DOWNLOADFAILED and what the log entry would look like in case of such a mismatch.

Yes, in AzCopy, if there is an MD5 mismatch during the download process, it will be logged under the DOWNLOADFAILED category in the AzCopy log file. This is because an MD5 mismatch indicates that the file's integrity has been compromised during the download process, which is effectively a failure in the download operation.

The below is what the log entry might look like for an MD5 mismatch:

{
  "time": "2024-05-30T12:34:56.789Z",
  "level": "ERROR",
  "msg": "DOWNLOADFAILED",
  "source": "https://mystorageaccount.blob.core.windows.net/mycontainer/myblob",
  "destination": "/local/path/to/myblob",
  "error": {
    "code": "MD5Mismatch",
    "message": "The MD5 hash of the downloaded file does not match the expected hash."
  }
}

Also, you can use a script to scan AzCopy log files for MD5 mismatches. The example provided here is written in Python:

import json
import os
log_directory = "<path-to-log-directory>"
def check_md5_mismatches(log_directory):
    for filename in os.listdir(log_directory):
        if filename.endswith(".log"):
            with open(os.path.join(log_directory, filename), 'r') as log_file:
                for line in log_file:
                    log_entry = json.loads(line)
                    if log_entry.get("msg") == "DOWNLOADFAILED" and log_entry["error"].get("code") == "MD5Mismatch":
                        print(f"MD5 mismatch found in log {filename}:")
                        print(json.dumps(log_entry, indent=4))
# Run the function
check_md5_mismatches(log_directory)

Therefore, by using AzCopy with --put-md5 and --check-md5, you can ensure file integrity during downloads. If an MD5 mismatch occurs, it will be logged under the DOWNLOADFAILED category. Monitoring these logs and implementing steps to verify and resolve MD5 mismatches can help maintain data integrity. The provided Python script can help automate the detection of such mismatches in AzCopy logs.

Accept Answer

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

** Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful ** so that others in the community facing similar issues can easily find the solution.

Best Regards,

Sina Salam