Turning HPC Systems into Interactive Data Analysis Platforms | SciPy 2019 | A. Banihirwe

This talk demonstrates how to use Dask and Jupyter on large high-performance computing (HPC) systems to scale and accelerate large interactive data analysis tasks -- effectively turning HPC systems into interactive big-data platforms. We will introduce dask-jobqueue which allows users to seamlessly deploy and scale dask on HPC clusters that use a variety of job queuing systems such as PBS, Slurm, SGE, and LSF. We will also introduce dask-mpi, a Python package that makes deploying Dask easy from within a distributed MPI environment. See the full SciPy 2019 playlist at https://www.youtube.com/playlist?list=PLYx7XA2nY5GcDQblpQ_M1V3PQPoLWiDAC Connect with us! ***************** https://twitter.com/enthought https://www.facebook.com/Enthought/ https://www.linkedin.com/company/enthought