Search code examples
azureazcopy

AzCopy V10 Download Directory containing over a Million Files

I have a directory with over a million files. I can selectively download a file without any issues using this command:

azcopy copy https://XXXXX/YYYY/10000000-0000-0000-0000-000000000000 C:\temp\YYYY

However, when I try to selectively download files based on a pattern, I get an error. This is the command I used:

azcopy copy https://XXXXX/YYYY C:\temp\YYYY --recursive --include-pattern '1000*'

I get the following error:

Description=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:XXXXXXXXXXXXXXXXXX
Time:2019-11-20T16:24:57.6698636Z, Details:
   AuthenticationErrorDetail: Issuer validation failed. Issuer did not match.
   Code: AuthenticationFailed
   GET https://XXXXX/YYYY?comp=list&include=metadata&restype=container&timeout=901
   Authorization: REDACTED
   User-Agent: [AzCopy/10.3.2 Azure-Storage/0.7 (go1.13; Windows_NT)]
   X-Ms-Client-Request-Id: [zzzzzzzzzzzzzzzzzz]
   X-Ms-Version: [2018-03-28]
   --------------------------------------------------------------------------------
   RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
   Content-Length: [422]
   Content-Type: [application/xml]
   Date: [Wed, 20 Nov 2019 16:24:57 GMT]
   Server: [Microsoft-HTTPAPI/2.0]
   X-Ms-Error-Code: [AuthenticationFailed]
   X-Ms-Request-Id: [zzzzzzzzzzzzzzzzz]

Any ideas on what I can try.

Solution

  • For --include-pattern, you should use double quotes to embrace the value, like "1000*", not '1000*'.

    And also use the latest version 10.3.2 of azcopy, follow this link to download.

    Here are the steps I followed, and works fine:

    1.In azure portal -> nave to your storage account -> in the left pane, click "Shared access signature", then generate a sas-token(you should copy the sas token, which can be used in step2). Screenshot as below:

    enter image description here

    2.Then build your source url, like https://yy3.blob.core.windows.net/t11/folder1/folder2?sas-Token.

    3.The complete command like below, note use double quotes for the value of --include-pattern:

    azcopy.exe copy "https://yy3.blob.core.windows.net/test1/folder1/folder2?sas-token" C:\temp\YYYY --recursive --include-pattern "1000*"

    4.The test result as below, only the matched files are copies:

    enter image description here