vSphere VMware.Vim Clone_VM 'The operation is not allowed in the current state.'

Question

My team is developing an application that needs to clone existing templates in our vSphere environment. We're using VMware.Vim in a C# application to do this. We're replacing an already existing implementation that uses PowerShell.

Below is the code that is throwing the error. We are eventually going to load balance based on memory usage, but currently we are selecting the host by random. That's why there is some extra code with collecting all of the hosts and then picking one.

When it gets to CloneVM_Task, an exception with the message 'The operation is not allowed in the current state.' is thrown. The exception doesn't give me much to work with and I can't find any useful logs in vSphere. vSphere just says "An error prevented the virtual machine from being cloned" in the events log. We're using version 6.7. I'm new to VMWare, so any help is appreciated. Their documentation is lacking, to say the least.

public async void CreateVirtualMachineAsync(NewVMRequest newVMRequest)
{
    var appliance = await _applianceService.GetAppliance(newVMRequest.Appliance);

    var vimClient = new VimClientImpl
    {
        IgnoreServerCertificateErrors = true, ServiceUrl = appliance.ServiceUrl
    };
    vimClient.Login(appliance.User, appliance.Password);

    var datacenter = GetDatacenter(vimClient);

    var hostCollection = GetListOfHosts(vimClient, datacenter);

    var randomHost = PickRandomHost(hostCollection);

    var sourceVm = GetSelectedVm(vimClient, newVMRequest);

    if (sourceVm == null)
    {
        _logger.LogDebug($"Could not find virtual machine {newVMRequest.Source} to use for template");
        _logger.LogError($"Could not find virtual machine {newVMRequest.Source} to use for template", null);
        return;
    }

    var selectedStore = ConnectToDataStore(vimClient);

    var cluster = GetCluster(vimClient);

    var mySpec = CreateCloneSpec(selectedStore, randomHost, cluster, sourceVm);

    vimClient.WaitForTask(sourceVm.CloneVM_Task(sourceVm.Parent, newVMRequest.Name, mySpec));

    vimClient.Disconnect();
}

private VirtualMachineCloneSpec CreateCloneSpec(Datastore selectedStore, ManagedObjectReference randomHost, ClusterComputeResource cluster, VirtualMachine sourceVm)
{
    var mySpec =  new VirtualMachineCloneSpec
    {
        Location = new VirtualMachineRelocateSpec
        {
            Datastore = selectedStore.MoRef,
            Transform = VirtualMachineRelocateTransformation.sparse,
            Host = randomHost,
            Pool = cluster.ResourcePool
        },
        Config = new VirtualMachineConfigSpec()
    };

    var networkDevice = new VirtualDevice();
    foreach (var vDevice in sourceVm.Config.Hardware.Device)
    {
        if (vDevice.DeviceInfo.Label.Contains("Network"))
        {
            networkDevice = vDevice;
        }
    }

    var devSpec = new VirtualDeviceConfigSpec
    {
        Device = networkDevice, FileOperation = VirtualDeviceConfigSpecFileOperation.create
    };
    mySpec.Config.DeviceChange = new[] { devSpec };

    return mySpec;
}

private Datacenter GetDatacenter(VimClient vimClient)
{
    var entities = vimClient.FindEntityViews(typeof(Datacenter), null, null, null);
    return (Datacenter)entities.First();
}

private List<ManagedObjectReference> GetListOfHosts(VimClient vimClient, Datacenter datacenter)
{
    var hostCollection = new List<ManagedObjectReference>();
    var hostFolderMoRef = datacenter.HostFolder;
    var hostFolder = (Folder)vimClient.GetView(hostFolderMoRef, null);
    var childEntityMoRefs = hostFolder.ChildEntity;
    foreach (var childEntityMoRef in childEntityMoRefs)
    {
        var thisCluster = (ClusterComputeResource)vimClient.GetView(childEntityMoRef, null);
        var clusterHostMoRefs = thisCluster.Host;
        foreach (var clusterHostMoRef in clusterHostMoRefs)
        {
            var hostSystem = (HostSystem)vimClient.GetView(clusterHostMoRef, null);
            hostCollection.Add(hostSystem.MoRef);
        }
    }

    return hostCollection;
}

private ManagedObjectReference PickRandomHost(List<ManagedObjectReference> hostCollection)
{
    var rand = new Random();
    return hostCollection[rand.Next(0, hostCollection.Count)];
}

private VirtualMachine GetSelectedVm(VimClient vimClient, NewVMRequest newVMRequest)
{
    var filter = new NameValueCollection
    {
        {"name", newVMRequest.Source},
        {"Config.Template", newVMRequest.UseTemplate.ToString().ToLower()}
    };
    var entityViews = vimClient.FindEntityViews(typeof(VMware.Vim.VirtualMachine), null, filter, null);

    if (entityViews.Count == 0)
    {
        return null;
    }
    return (VirtualMachine)vimClient.FindEntityViews(typeof(VMware.Vim.VirtualMachine), null, filter, null).First();
}

private Datastore ConnectToDataStore(VimClient vimClient)
{  
    var myDs = vimClient.FindEntityView(typeof(Datastore), null, null /*dsFilter*/, null);
    return (Datastore)myDs;
}

private ClusterComputeResource GetCluster(VimClient vimClient)
{
    var appClusters = vimClient.FindEntityViews(typeof(ClusterComputeResource), null, null, null);
    return (ClusterComputeResource)appClusters?.FirstOrDefault();
}

Answer 1

For the most part, your code looks good. I would suggest changing the spec file and using the least possible information to create a clone from existing template. Once you have success, you can add more details and see if that errors out.

Start off with this and see if this gets you going.

private VirtualMachineCloneSpec CreateCloneSpec(Datastore selectedStore, ManagedObjectReference randomHost, ClusterComputeResource cluster, VirtualMachine sourceVm)
{
    var mySpec =  new VirtualMachineCloneSpec
    {
        Location = new VirtualMachineRelocateSpec
        {
            Datastore = selectedStore.MoRef,
            Host = randomHost,
            Pool = cluster.ResourcePool
        },
        PowerOn = false,
        Template = false
    };

    return mySpec;
}

If the above goes through, then add the VirtualMachineRelocateTransformation to transform your thick VM as thin VM and network connections as needed.