The following script runs a hive job from my local machine running Windows 8.1 and copies the output to a local file. It has been working fine up to now, but after upgrading to Windows 10 it is erroring out when I call Get-AzureHDInsightJobOutput. I have tried uninstalling Windows Azure Powershell, and reinstalling, but that did not help. Not sure how to proceed now, anyone have any ideas?
# Retrieve a single tagged stream from project hive database into a single file
#
#
param(
[string]$table = 'sensor_part',
[string]$bld = 'B24',
[string]$tag = 'B24/AHU/LR/_1/OAT'
)
$clusterName = 'Crunch88'
$HiveQuery = "select * from ${table} where bld='${bld}' and tag='${tag}'"
$jobDefinition = New-AzureHDInsightHiveJobDefinition -Query $HiveQuery -StatusFolder '/tagtransfer'
$job = Start-AzureHDInsightJob -Cluster $clusterName -JobDefinition $jobDefinition
Write-Host "Wait for the Hive job to complete ..." -ForegroundColor Green
Wait-AzureHDInsightJob -Job $job -WaitTimeoutInSeconds 72000
Get-AzureHDInsightJobOutput -StandardError -JobId $job.JobId -Cluster $clusterName
Write-Host "Copy the standard output to a local file" -ForegroundColor Green
Get-AzureHDInsightJobOutput -Cluster $clusterName -JobId $job.JobId -DownloadTaskLogs -TaskLogsDirectory "tasklogs"
# AzCopy /Y /Source:${blobcontainer} /Dest:${tsvpath} /SourceKey:${blobkey} /Pattern:${dirout}/stdout
Here is the results of a run with the error message:
PS C:\Windows\System32\WindowsPowerShell\v1.0> F:\88acres\3-hive\hiveretrieval\error.ps1
WARNING: When submitting a query use the -RunAsFile switch to prevent errors with query lengths or special characters
Wait for the Hive job to complete ...
Cluster : Crunch88
ExitCode : 0
Name : Hive: select * from sensor
PercentComplete : map 100% reduce 0%
Query : select * from sensor_part where bld='B24' and tag='B24/AHU/LR/_1/OAT'
State : Completed
StatusDirectory : /tagtransfer
SubmissionTime : 30-Aug-15 12:26:18
JobId : job_1438448805059_0015
Get-AzureHDInsightJobOutput : Could not load file or assembly 'Microsoft.WindowsAzure.Storage, Version=3.0.3.0, Culture=neutral,
PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.
At F:\88acres\3-hive\hiveretrieval\error.ps1:18 char:1
+ Get-AzureHDInsightJobOutput -StandardError -JobId $job.JobId -Cluster ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Get-AzureHDInsightJobOutput], FileNotFoundException
+ FullyQualifiedErrorId : System.IO.FileNotFoundException,Microsoft.WindowsAzure.Management.HDInsight.Cmdlet.PSCmdlets.GetAzu
reHDInsightJobOutputCmdlet
Copy the standard output to a local file
Get-AzureHDInsightJobOutput : Could not load file or assembly 'Microsoft.WindowsAzure.Storage, Version=3.0.3.0, Culture=neutral,
PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.
At F:\88acres\3-hive\hiveretrieval\error.ps1:21 char:1
+ Get-AzureHDInsightJobOutput -Cluster $clusterName -JobId $job.JobId - ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Get-AzureHDInsightJobOutput], FileNotFoundException
+ FullyQualifiedErrorId : System.IO.FileNotFoundException,Microsoft.WindowsAzure.Management.HDInsight.Cmdlet.PSCmdlets.GetAzu
reHDInsightJobOutputCmdlet
Update: I have enabled fusion logging. The results indicate that it is binding to 5.0.0.0 instead of 3.0.3.0. Not sure how to fix this.
Failed log entry for 3.0.3.0 follows:
*** Assembly Binder Log Entry (04-Sep-15 @ 08:54:11) ***
The operation failed.
Bind result: hr = 0x80070002. The system cannot find the file specified.
Assembly manager loaded from: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
Running under executable C:\Windows\System32\WindowsPowerShell\v1.0\powershell_ise.exe
--- A detailed error log follows.
=== Pre-bind state information ===
LOG: DisplayName = Microsoft.WindowsAzure.Storage, Version=3.0.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
(Fully-specified)
LOG: Appbase = file:///C:/Windows/System32/WindowsPowerShell/v1.0/
LOG: Initial PrivatePath = NULL
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = powershell_ise.exe
Calling assembly : Microsoft.Hadoop.Client, Version=1.5.9.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35.
===
LOG: This bind starts in LoadFrom load context.
WRN: Native image will not be probed in LoadFrom context. Native image will only be probed in default load context, like with Assembly.Load().
LOG: Using application configuration file: C:\Windows\System32\WindowsPowerShell\v1.0\powershell_ise.exe.Config
LOG: Using host configuration file:
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config.
LOG: Post-policy reference: Microsoft.WindowsAzure.Storage, Version=3.0.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
LOG: GAC Lookup was unsuccessful.
LOG: Attempting download of new URL file:///C:/Windows/System32/WindowsPowerShell/v1.0/Microsoft.WindowsAzure.Storage.DLL.
LOG: Attempting download of new URL file:///C:/Windows/System32/WindowsPowerShell/v1.0/Microsoft.WindowsAzure.Storage/Microsoft.WindowsAzure.Storage.DLL.
LOG: Attempting download of new URL file:///C:/Windows/System32/WindowsPowerShell/v1.0/Microsoft.WindowsAzure.Storage.EXE.
LOG: Attempting download of new URL file:///C:/Windows/System32/WindowsPowerShell/v1.0/Microsoft.WindowsAzure.Storage/Microsoft.WindowsAzure.Storage.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft SDKs/Azure/PowerShell/ServiceManagement/Azure/HDInsight/Microsoft.WindowsAzure.Storage.DLL.
LOG: Assembly download was successful. Attempting setup of file: C:\Program Files (x86)\Microsoft SDKs\Azure\PowerShell\ServiceManagement\Azure\HDInsight\Microsoft.WindowsAzure.Storage.dll
LOG: Entering run-from-source setup phase.
LOG: Assembly Name is: Microsoft.WindowsAzure.Storage, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
WRN: Comparing the assembly name resulted in the mismatch: Major Version
ERR: The assembly reference did not match the assembly definition found.
ERR: Run-from-source setup phase failed with hr = 0x80131040.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft SDKs/Azure/PowerShell/ServiceManagement/Azure/HDInsight/Microsoft.WindowsAzure.Storage/Microsoft.WindowsAzure.Storage.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft SDKs/Azure/PowerShell/ServiceManagement/Azure/HDInsight/Microsoft.WindowsAzure.Storage.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft SDKs/Azure/PowerShell/ServiceManagement/Azure/HDInsight/Microsoft.WindowsAzure.Storage/Microsoft.WindowsAzure.Storage.EXE.
LOG: All probing URLs attempted and failed.
***
Assembly Binder Log Entry (04-Sep-15 @ 08:54:11) ***
And here is the successful version for 5.0.0.0:
*** Assembly Binder Log Entry (04-Sep-15 @ 08:54:10) ***
The operation was successful.
Bind result: hr = 0x0. The operation completed successfully.
Assembly manager loaded from: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
Running under executable C:\Windows\System32\WindowsPowerShell\v1.0\powershell_ise.exe
--- A detailed error log follows.
=== Pre-bind state information ===
LOG: DisplayName = Microsoft.WindowsAzure.Storage, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
(Fully-specified)
LOG: Appbase = file:///C:/Windows/System32/WindowsPowerShell/v1.0/
LOG: Initial PrivatePath = NULL
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = powershell_ise.exe
Calling assembly : Microsoft.WindowsAzure.Commands.Sync, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35.
===
LOG: This bind starts in LoadFrom load context.
WRN: Native image will not be probed in LoadFrom context. Native image will only be probed in default load context, like with Assembly.Load().
LOG: Using application configuration file: C:\Windows\System32\WindowsPowerShell\v1.0\powershell_ise.exe.Config
LOG: Using host configuration file:
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config.
LOG: Post-policy reference: Microsoft.WindowsAzure.Storage, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
LOG: GAC Lookup was unsuccessful.
LOG: Attempting download of new URL file:///C:/Windows/System32/WindowsPowerShell/v1.0/Microsoft.WindowsAzure.Storage.DLL.
LOG: Attempting download of new URL file:///C:/Windows/System32/WindowsPowerShell/v1.0/Microsoft.WindowsAzure.Storage/Microsoft.WindowsAzure.Storage.DLL.
LOG: Attempting download of new URL file:///C:/Windows/System32/WindowsPowerShell/v1.0/Microsoft.WindowsAzure.Storage.EXE.
LOG: Attempting download of new URL file:///C:/Windows/System32/WindowsPowerShell/v1.0/Microsoft.WindowsAzure.Storage/Microsoft.WindowsAzure.Storage.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft SDKs/Azure/PowerShell/ServiceManagement/Azure/Compute/Microsoft.WindowsAzure.Storage.DLL.
LOG: Assembly download was successful. Attempting setup of file: C:\Program Files (x86)\Microsoft SDKs\Azure\PowerShell\ServiceManagement\Azure\Compute\Microsoft.WindowsAzure.Storage.dll
LOG: Entering run-from-source setup phase.
LOG: Assembly Name is: Microsoft.WindowsAzure.Storage, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
LOG: Where-ref bind Codebase does not match what is found in default context. Keep the result in LoadFrom context.
LOG: Binding succeeds. Returns assembly from C:\Program Files (x86)\Microsoft SDKs\Azure\PowerShell\ServiceManagement\Azure\Compute\Microsoft.WindowsAzure.Storage.dll.
LOG: Assembly is loaded in LoadFrom load context.
Update: Added the required assembly using a Nuget package. Still didn't help. Think I somehow need to tell Nuget to make it available for the PowerShellISE, but don't see a parameter for that... This must be close to a solution.
PS C:\WINDOWS\system32> nuget install WindowsAzure.Storage -version 3.0.3
Attempting to resolve dependency 'Microsoft.Data.OData (ò 5.6.0)'.
Attempting to resolve dependency 'System.Spatial (= 5.6.0)'.
Attempting to resolve dependency 'Microsoft.Data.Edm (= 5.6.0)'.
Attempting to resolve dependency 'Newtonsoft.Json (ò 5.0.6)'.
Attempting to resolve dependency 'Microsoft.Data.Services.Client (ò 5.6.0)'.
Attempting to resolve dependency 'Microsoft.WindowsAzure.ConfigurationManager (ò 1.8.0.0)'.
Installing 'System.Spatial 5.6.0'.
Successfully installed 'System.Spatial 5.6.0'.
Installing 'Microsoft.Data.Edm 5.6.0'.
Successfully installed 'Microsoft.Data.Edm 5.6.0'.
Installing 'Microsoft.Data.OData 5.6.0'.
Successfully installed 'Microsoft.Data.OData 5.6.0'.
Installing 'Newtonsoft.Json 5.0.6'.
Successfully installed 'Newtonsoft.Json 5.0.6'.
Installing 'Microsoft.Data.Services.Client 5.6.0'.
Successfully installed 'Microsoft.Data.Services.Client 5.6.0'.
Installing 'Microsoft.WindowsAzure.ConfigurationManager 1.8.0.0'.
Successfully installed 'Microsoft.WindowsAzure.ConfigurationManager 1.8.0.0'.
Installing 'WindowsAzure.Storage 3.0.3.0'.
Successfully installed 'WindowsAzure.Storage 3.0.3.0'.
PS C:\WINDOWS\system32> F:\88acres\3-hive\hiveretrieval\error.ps1
WARNING: When submitting a query use the -RunAsFile switch to prevent errors with query lengths or special characters
Wait for the Hive job to complete ...
Cluster : Crunch88
ExitCode : 0
Name : Hive: select * from sensor
PercentComplete : map 100% reduce 0%
Query : select * from sensor_part where bld='B24' and tag='B24/AHU/LR/_1/OAT'
State : Completed
StatusDirectory : /tagtransfer
SubmissionTime : 04-Sep-15 08:03:50
JobId : job_1440975278889_0021
Get-AzureHDInsightJobOutput : Could not load file or assembly 'Microsoft.WindowsAzure.Storage, Version=3.0.3.0, Culture=neutral,
PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.
At F:\88acres\3-hive\hiveretrieval\error.ps1:18 char:1
+ Get-AzureHDInsightJobOutput -StandardError -JobId $job.JobId -Cluster ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Get-AzureHDInsightJobOutput], FileNotFoundException
+ FullyQualifiedErrorId : System.IO.FileNotFoundException,Microsoft.WindowsAzure.Management.HDInsight.Cmdlet.PSCmdlets.GetAzu
reHDInsightJobOutputCmdlet
I didn't find a good solution for this issue. I ran into this same issue after I installed to Visual Studio 2015, so if someone has a good solution I'd love to hear it. At this point in time... I just took the Microsoft.WindowsAzure.Storage.dll version 3.0.3.0 and copied it to my \Program Files (x86)\Microsoft SDKs\Azure\PowerShell\ServiceManagement\Azure\HDInsight folder (making a backup of the newer version first). It works fine now, but this is only a temporary work around.