we would like to make sure that the MLFLOW experiment management platform fits our needs and workflow.
We work with image processing CNNs like Yolo, UNET, and RetinaNet based on an NVIDIA TAO framework.
What we actually need is a tool that concentrates on one place (in a nice and representative way comfortable for comparison) at least the three following things for each experiment:
a- chosen by user typical meta parameters that were used to train a network (such as batches, subdivisions, max batches, etc) b- a link to the dataset the network was trained on, located on our cloud storage (such as one-drive, google drive or google cloud) or a list of filenames or a link to a file storage cloud or online drive suggested by MLFLOW service if there is such a thing. c- a result of running the trained network - the number of detected objects
Thus the question is:
Does the MLFLOW fit our needs? If not ill be glad if anyone could suggest a relevant alternative. Thank you
I use Comet.ml and it addresses all 3 of your points.
With Comet, parameter tracking is as easy as calling the experiment.log_parameter function. You can also use the diff tool to compare two experiments by their hyper parameters and even group experiments by hyper-paremeters!
Comet has the concept of artifacts. You can upload your dataset as an artifact and version it. You can also have remote artifacts!
Comet has a feature call the Image Panel. This allows users to visualize their model performance on the data across different experiment runs. For the object detection use case, use experiment.log_image to log your images in which you have you drawn your model predicted bounding box on! You will then see in the Image Panel different experiments and how they each one of them are drawing their predictions side by side