Dask
chirashi.dask
A Docker Compose file for Dask alone, generated from the catalog and checked by the safety inspector. No account needed.
Parallel computing library for Python that partitions arrays, dataframes and custom task graphs across cores or a cluster, mirroring the NumPy and pandas APIs on larger-than-memory data.
Dask splits a large array or dataframe into partitions, builds a task graph describing the operations across them, and executes that graph over threads, processes or a distributed cluster of workers. Because its collections follow the NumPy and pandas interfaces closely, existing analysis code often runs with little modification while handling data that does not fit in memory. A lower-level delayed and futures interface covers workloads that are neither array nor dataframe shaped. A cluster is a scheduler plus workers, and its dashboard shows the task graph, worker memory and where time is actually being spent.
You know it worked when
- The scheduler starts and its dashboard reports the expected worker count.
- A client connects and a computation returns a correct result.
- The dashboard shows tasks distributed across workers rather than concentrated on one.
- A dataframe larger than a single worker's memory completes an aggregation.
Known sharp edges
- Scheduler and workers must run matching Python and library versions, and a mismatch surfaces as deserialisation errors rather than a version warning.
- Workers spill to disk when memory fills and then pause, so a workload that looks merely slow is often thrashing rather than computing.
- Partition size drives everything: too few leave workers idle, too many drown the scheduler in task overhead.
- The scheduler and dashboard ports accept work without authentication and must stay on an internal network.