Unleash Your Data for HPC & AI with the WEKA Data Platform

Our customers are using a fairly broad gamut of machine learning processing frameworks. At its core, WEKA is a POSIX compliant storage system and we've got a POSIX compliant client interface. So, you do not need to change any of your machine learning processing frameworks to use WEKA. Your favorite and processing framework from PyTorch to Caffe, they work out-of-the-box with WEKA, and they work extremely well.  

A lot of our customers are running very large models. They start off with models on a single GPU and then they go to multiple GPUs in a single client, and then they start running very large models across multiple GPUs across multiple clients. WEKA works very well regardless of the model size and how many GPUs that they're running across in parallel.  WEKA provides good performance with lots of concurrent processes reading and writing into the same file. In summary, don't worry as long as the Linux applications. They will be POSIX compliant, and they will be able to use WEKA.

Most of our work is done with applications that are running in parallel leveraging highly distributed data. I don't know what particular file formats you're looking at, or particular distributed I/O tools you're thinking about.  We've not come across any common tools in an HPC environment that's caused any issues. If you've got applications where you're particularly concerned, it would be interesting if you could share the file format and any details with me. We run it and test it for you, but I would be very surprised if it poses a problem for WEKA. 

With regards to latency, I mentioned that we are reading and writing data over the network at sub millisecond. The time it takes to pass data over a high performance network is a tiny fraction of the time it takes to do a read and write operation on an SSD. Data transfer over the network is completely negligible. We don't really worry about data locality in the same way that we did when people were using 10 GB or 1 GB networks. Transferring data over 100 GB network or above is so quick that data locality is no longer an issue. If you're using something like WEKA, you know that's got a completely optimized network stack.

That's a really good question because we tend to show off a little bit with our largest customers, but most of our customers are significantly smaller than that. We've got customers that that are starting as small as 50 to 100 TB of capacity. I would say, if you're going to invest in a network attached storage system, then that's probably a good starting point for WEKA. Because you've got to invest in the networking, you may already have enough ports, but you've got to invest in NVME devices, servers and networking to build a cluster. So it only really makes sense once you've got 50-100 TB and above. Also suitable for small scale enterprises or like academic institutions. 

Potentially it would work, but you're not really going to get the benefits of WEKA because we've built and designed our software stack for NVMe. If you wanted to use SAS flash drives for example, then it would probably work. But it's not something that's going to get the highest levels of performance, because our software stacks are designed for NVMe and similarly with spinning disc, you know you're not going to get the levels of performance. We just wouldn't recommend doing so. 

There's certain workloads that HDDs is still good for, but what we see is that most modern workloads require either an all flash system or certainly a primary flash storage, where the hot data resides these days. Though a spinning disc is getting more and more capacity on a single drive, the performance isn't necessarily going up. So through a file system, you're only going to get, let's say, 150 MBPS from a spinning disc. And you could get up to 5 GBPS, or more with PCIe Gen 5 on a on an NVMe device. Per GB or per TB of data, spinning disc is getting slower and slower over time. So for that high performance layer to your applications, it's pretty hard to ignore having enough flash tier these days.

Our POSIX client running on your computer would require a few megabytes of memory. I can't remember the exact figure. On the storage server side, we're probably looking at anywhere from 256 – 512 GB of RAM. The different services we're using to run the storage cluster, so we look for a fairly high chunk. 

We can do a live demo where you'll get a better flavor of how WEKA works, how it's administered and how it's used day-to-day. We can give people access to a demo lab where you can use some of our clients and WEKA storage cluster to do some testing. We can also provide demo licenses if you want to do some on premise testing. And finally we have the option to build a WEKA cluster in the public cloud or we can build it in your cloud environment and you can you can play around with it there. So there are a lot of different options on how to test working. 

The license is subscription based and it's licensed per usable TB. So typically our on premise customers would let's say acquire a WEKA storage system with 500 TB of capacity. Then you know you would buy a WEKA license for 500 TB and you would buy it generally for the same length of time as your hardware support. So it's subscription based for the period of time that you want to use it based on usable terabytes.