How can I leverage luigi for Openstack tasks

Question

I want to use Luigi to manage workflows in Openstack. I am new to Luigi. For the starter, I just want to authenticate myself to Openstack and then fetch image list, flavor list etc using Luigi. Any help will be appreciable.

I am not good with python but I tried below code. I am also not able to list images. Error: glanceclient.exc.HTTPNotFound: The resource could not be found. (HTTP 404)

import luigi
import os_client_config
import glanceclient.v2.client as glclient
from luigi.mock import MockFile
import sys
import os

def get_credentials():
    d = {}
    d['username'] = 'X'
    d['password'] = 'X'
    d['auth_url'] = 'X'
    d['tenant_name'] = 'X'
    d['endpoint'] = 'X'
    return d

class LookupOpenstack(luigi.Task):
    d =[]

    def requires(self):
        pass
    def output(self):
        gc = glclient.Client(**get_credentials())
        images = gc.images.list()
        print("images", images)
        for i in images:
            print(i)

        return MockFile("images", mirror_on_stderr=True)

    def run(self):
        pass

if __name__ == '__main__':
    luigi.run(["--local-scheduler"], LookupOpenstack())

Answer 1

The general approach to this is just write python code to perform the tasks you want using the OpenStack API. https://docs.openstack.org/user-guide/sdk.html It looks like the error you are getting is addressed on the OpenStack site. https://ask.openstack.org/en/question/90071/glanceclientexchttpnotfound-the-resource-could-not-be-found-http-404/

You would then just wrap this code in luigi Tasks as appropriate- there's nothing special about doing with this OpenStack, except that you must define the output() of your luigi tasks to match up with an output that indicates the task is done. Right now it looks like the work is being done in the output() method, which should be in the run() method, the output method should just be what to look for to indicate that the run() method is complete so it doesn't run() when required by another task if it is already done.

It's really impossible to say more without understanding more details of your workflow.