The cloud has opened up opportunities for virtually every industry. The ability to quickly acquire all the compute resources you need has made projects that once seemed infeasible completely practical. Do you have an unexpected rendering job and no available capacity on premises? Use the cloud to spin up the extra nodes. Do you want to use machine learning algorithms to analyze a large pool of gene expression data? Create a virtual supercomputer in the cloud and get years of compute in a few hours.
But life in the cloud isn’t perfect. One of the biggest gotchas for many companies is file storage. While object storage is a mature technology and actively advocated by cloud providers, file storage in the cloud comes with many limitations and pain points. Lack of protocol support, performance limitations, problematic retrofitting of legacy solutions for the cloud, and data mobility are four common problems.
Cloud file storage gotchas
In terms of protocol support, customers who rely on SMB file services often find that it’s not supported. You’ll find SMB file storage on Azure, but elsewhere in the cloud file storage is much more likely to support NFS.
Performance for cloud-based file storage is tricky. Many products either have limited scalability or performance is explicitly tied to capacity. The latter is true of the AWS Elastic File System (EFS). Customers who don’t have enough data stored but want better performance sometimes store dummy files to get more bandwidth. In other words, they resort to creating false capacity to get the performance they need.
(QF2) is a modern, file storage system that can scale to billions of files and that runs in the data center and in AWS. In QF2, cloud instances or computing nodes with standard hardware work together to form a cluster that has scalable performance and a single, unified file system. All QF2 clusters form a globally distributed but highly connected storage fabric tied together with continuous replication. In the rest of this article, we’ll concentrate on how QF2’s capabilities solve the problems people have encountered when looking for file storage in the cloud.
Using the Qumulo API interactively
You can experiment with the API interactively by clicking “API & Tools” from the dashboard. Here is an example of what you’ll see.
If you scroll down, under Files, you’ll see the method for listing the entries in a directory.
If you click on the method, it expands to show the call and the information you need to enter. For this method, you need to enter the directory you want to read and the number of files you want to list.
In this example, the directory is CERT_ROOT and the maximum number of files returned is 1000.
After you click “Try It,” you get the response header, which shows if the call succeeded.
If the call succeeded, the rest of the response shows the information you requested. Here is an example.
Using the Python library for the Qumulo API
You can also access the API with Python. The instructions for installing and using the Python library are . For example, here is how to create a file.
# create file
curl -X POST
-H "Content-Type: application/json"
-H"Authorization: Bearer $API_TOKEN"
You can see more examples of how to use Python .
Deploying QF2 for AWS with Terraform
You can use popular automation platforms to automatically deploy QF2 for AWS. As an example, you can use the Terraform template, , to deploy QF2 clusters. Set the number of nodes in the
cluster_config variable to either 1 or to a number greater than or equal to 4. You can also use a .tfvars file to automatically populate the variables.
Try QF2 for yourself
You can create a with 5TB of storage for free. You pay only for the underlying AWS infrastructure. After adding the QF2 AMI to your AWS account, work through the tutorial called “” or watch the accompanying video.
If you would like to try out a multi-node cluster, call us at +1 (206) 260-3588 or chat with us online and learn how you can qualify for 10,000 free QF2 consumption units (Qnits). With 10,000 Qnits, you’ll have more than enough time to evaluate large-scale production workloads on a four-node QF2 cluster.
New Tech Forum provides a venue to explore and discuss emerging enterprise technology in unprecedented depth and breadth. The selection is subjective, based on our pick of the technologies we believe to be important and of greatest interest to Techdailyhub readers. Techdailyhub does not accept marketing collateral for publication and reserves the right to edit all contributed content. Send all inquiries to .