How do Dask dataframes handle larger-than-memory datasets?

Question

The documentation of the Dask package for dataframes says:

Dask dataframes look and feel like pandas dataframes, but operate on datasets larger than memory using multiple threads.

But later in the same page:

One dask DataFrame is comprised of several in-memory pandas DataFrames separated along the index.

Does Dask read the different DataFrame partitions from disk sequentally and perform computations to fit into memory? Does it spill some partitions to disk when needed? In general, how does Dask manage the memory <--> disk IO of data to allow larger-than-memory data analysis?

I tried to perform some basic computations (e.g. mean rating) on the 10M MovieLens dataset and my laptop (8GB RAM) started to swap.

Answer 1

Dask.dataframe loads data lazily and attempts to perform your entire computation in one linear scan through the dataset. Surprisingly, this is usually doable.

Intelligently dumping down to disk is also an option that it can manage, especially when shuffles are required, but generally there are ways around this.