Search code examples
command-linehdfsmlrun

Unexpected duration for v3io content delete

When I deleted content of NoSqlTarget (key-value storage) in MLRun/v3io via standard command line utility such as:

rm -rf /v3io/project/test-featureset/

It took approx. 1 hour for 1 million of files and in case of 20 millions of files it was near to one day.

Do you have tips/tricks, how to use faster delete?

Solution

  • I got very useful tip for speedup delete process, use this command line:

    hdfs dfs -rm -f -r v3io://projects/test-featureset/

    The new performance was excellent, 1 million of files per 3 minutes.

    BTW: the command hdfs dfs is possible to use for speedup other operations such as -ls -cp -find, more information see the link