Search code examples
azure-blob-storagemosaic-decisionscompressed-files

Does Mosaic supports ingesting compressed data?

We have a scenario of uploading compressed files into Blob container in Microsoft Azure and then read it.

Is it possible in Mosaic to do it and if yes, what is the way to achieve it?

We have files in .gz format.

Solution

  • Yes you can upload and read compressed files in Mosaic through Azure Reader. Currently, Mosaic supports two compression types - .ZIP & .GZ

    To read compressed files in Mosaic's Azure Reader node you can follow below steps -

    1. In Path field, provide the path of the compressed folder as shown in screen shot below enter image description here
    2. Make the toggle button for Is Compressed is True
    3. Select the compression type - (either .ZIP or .GZ)
    4. In compressed path we will have to provide the file without the compressed extension. e.g. if the compressed file is ‘ABC.csv.gz’ then in compressed path it would be ‘ABC.csv’ Similarly for files compressed in .zip format, the compressed path will be the path of files within that compressed folder. e.g. compressed folder is ‘ABC.zip’ then compressed path would be ‘ABC/file.csv’
    5. Select the format of the file and Validate.