For the last 40 to 50 years, the hard drive has been the dominant storage medium for data and has been integral to the operation of high-performance computing (HPC) systems. Due to its legacy, most traditional HPC users have developed and evolved their applications, workloads and algorithms to use spinning media efficiently. They have, with varying degrees of success, worked around its limitations: mostly latency issues due to the physical movement of the hard-drive head across the spinning platter.
But a new breed of HPC user is emerging. These new users haven’t evolved with the hard-drive and have very sophisticated file formats and data-access patterns. When you run a large supercomputer, running thousands of jobs all accessing the same shared storage, the hard-drive sees a completely random workload and quickly becomes “seek-dominated”.
Enter the SSD/Flash/NVMe storage solution, which offers ultra-low latency (due to no moving parts), and equal access time regardless of where the data is physically stored in the medium. Your limitation moves to the speed of electronics and how quickly you can “flip-bits” in silicon.
BUT Flash storage systems have traditionally been too costly to roll-out at scale – which limited its use to a caching layer or very specific and much smaller workloads.
Finally, VAST broke all the rules, used all the latest ideas and technologies, and delivered a product that pushes all the right buttons.
The results have been impressive, with many workloads getting several times faster and others scaling to much larger compute systems. No longer do you have to do large sequential reads and writes – and never mix them. Do all your reads and writes when and how you want. Use your storage how you want.
In 2019 we made the decision to replace our hard disk-based legacy storage with innovative, all-flash storage service from Vast Data, powered by Intel Optane storage and memory. One year on, benefits of the new storage platform are clear, both internally and to customers.
You can read about our journey and the impact it has had on some of our customers in this interview with Phil Schwan, DUG’s CTO, published recently in VentureBeat. Download the PDF here, or visit VentureBeat here.
Hero picture: DUG’s HPC as a Service accelerated processing of 450 hours data from the Murchison Widefield Array (MWA), a low frequency radio telescope and the first of four Square Kilometre Array (SKA) precursors to be completed. Image Credit: Peter Wheeler, ICRAR.
