We are currently working on making an Azure MachineLearning Studio experiment operational.
Our most recent iteration has a webjob that accepts a queue message, gets some data to train the model, and consumes the ML Experiment webservice to put a trained model in a blob location.
A second webjob accepts a queue message, pulls the data to be used in the predictive experiment, gets the location path of the trained .ilearner model, and then consumes THAT ML Experiment webservice.
The data used to make the predictions is passed in as an input parameter, and the storage account name, key, and .ilearner path are all passed in as global parameters--Dictionary objects defined according to what the data scientist provided.
Everything appears to work correctly--except in some cases, the predictive experiment fails, and the error message makes it clear the wrong .ilearner file is being used.
When a non-existent blob path is passed to the experiment webservice, the error message reflects there is no such blob, so it's clear the webservice is at least validating the .ilearner's existence.
The data scientist can run it locally, but has to change the name of the .ilearner file when he exports it locally through PowerShell. Ensuring each trained model has a unique file name did not resolve this issue.
All files, when I view them in the Azure Storage Explorer, appear to be getting updated as expected based on last-modified dates. It's almost like there's a cached version of the .ilearner somewhere that isn't being overridden properly.
After ruling out all possibility of passing in the wrong file, our data scientist took a closer look at the experiment itself. He discovered that it was defaulting to one hardcoded .ilearner path he had been using in development.
At one point in time, he had created webservice parameters to override this value (hence why I had them defined in my webservice call), but they had been removed during one of the redesigns of the experiment with anyone noticing, because the webservice will apparently accept superfluous arguments.
The webservice was accepting my global parameters, and apparently even validating them. But since they weren't wired to anything inside the experiment the passed .ilearner file info was never applied to anything--the hardcoded .ilearner was being applied no matter what.
We were all very surprised there was no exception thrown about passing in parameters to the webservice that weren't actually defined. Had that happened, we would have gotten to the bottom of it much more quickly.
tl/dr: The experiment wasn't properly configured to accept an .ilearner file path (or Account Name, or Account Key) as a parameter, and the webservice was happily accepting and ignoring the parameter arguments without raising any alarm since it had the hardcoded value to run with.