Posts Tagged ‘AmazonS3’
[AWSReInvent2025] High-Performance Storage Architectures for AI/ML, Analytics, and HPC Workloads
Lecturer
Aditi is a Senior Product Manager for Amazon FSx at Amazon Web Services (AWS). With years of experience working directly with customers on high-performance workloads, she focuses on pushing the technical boundaries of what is possible with cloud storage to meet the demands of modern compute-intensive applications.
Abstract
This article examines the critical role of high-performance storage in supporting modern AI/ML, analytics, and High-Performance Computing (HPC) workloads. As organizations scale their compute resources—incorporating hundreds or thousands of CPU and GPU cores—storage often becomes the primary bottleneck, preventing linear performance scaling. We explore the technical architectures of Amazon FSx and Amazon S3, focusing on how these services address the needs of both “lift-and-shift” file-based applications and “cloud-native” S3-based data lakes. By analyzing customer use cases in genomics, media rendering, and large language model (LLM) training, we detail the methodologies for achieving peak performance at scale.
The Storage Bottleneck in Compute-Intensive Workloads
Modern high-performance workloads are characterized by their extreme reliance on massive datasets and high-core-count compute clusters. In an ideal cloud environment, adding more compute resources should lead to a proportional increase in work completed—a concept known as linear scaling. However, traditional storage solutions often fail to keep pace with the throughput demands of these clusters, leading to a performance plateau.
When storage becomes the bottleneck, compute instances sit underutilized as they compete for access to the same data store. This is particularly detrimental given that 90% to 95% of the expenditure for these workloads is typically allocated to compute resources. Consequently, an inefficient storage layer not only extends the time to insight but also significantly increases the total cost of ownership (TCO). To avoid this, storage must be architected to scale linearly alongside compute.
Navigating the Path to the Cloud: File Systems vs. Object Storage
Organizations generally approach high-performance storage on AWS from two distinct backgrounds: those with long-standing on-premises file-based workflows and those who have built native cloud applications around object storage.
The Persistence of File-Based Architectures
Despite the rise of object storage, file systems remain the preferred interface for many researchers and developers due to three primary factors: Familiar Interface: The intuitive nature of files and directories simplifies complex data management for data scientists and developers.
* Granular Permissions: File systems provide robust POSIX permissions, allowing for fine-grained control over which users can read, write, or execute specific files.
* Consistent Data Access:* For workloads where multiple users or compute nodes access the same data simultaneously, the strong consistency of file systems ensures that all parties see the most recent data updates.
Amazon FSx for High-Performance File Access
Amazon FSx addresses these needs by providing fully managed file systems that offer the performance of local storage with the scalability of the cloud. For “lift-and-shift” scenarios, FSx allows organizations to move their existing HPC and AI/ML pipelines to AWS without refactoring their applications.
Accelerating Generative AI and ML Workloads
The emergence of generative AI has placed a renewed emphasis on data strategy. Whether an organization is building a model from scratch or fine-tuning a foundational model, the quality and accessibility of its proprietary data are the primary differentiators.
Retrieval Augmented Generation (RAG)
To move beyond generic AI responses and reduce hallucinations, many organizations are implementing Retrieval Augmented Generation (RAG). RAG allows foundational models to access evolving, large-scale data lakes without requiring the data to be manually loaded into a prompt.
The RAG methodology involves:
1. Vectorization: Converting organizational data into vectors—numeric representations that capture semantic meaning.
2. Semantic Search: Using spatial similarity to compare a query vector against the data lake’s vectors to find the most relevant information.
3. Augmentation: Feeding the retrieved context back into the model to generate a more accurate and business-specific response.
Ingestion and Data Strategy with Amazon S3
Amazon S3 serves as the foundational data lake for these AI workflows due to its cost-effectiveness and virtually unlimited scalability. Organizations typically utilize two ingestion patterns:
* Batch Ingestion: Suitable for static or infrequently changing data such as historical records and product catalogs.
* Real-Time Ingestion: Essential for agentic workflows where AI models must respond to the latest available information.
Modernizing Self-Managed Databases with Amazon FSx
While fully managed services like Amazon RDS are popular, certain business and technical requirements drive organizations toward self-managed database architectures on AWS.
Drivers for Self-Managed Databases
Organizations choose to self-manage databases like Oracle, SQL Server, or SAP HANA for several reasons:
* Granular Control: The ability to choose specific versions of the database engine and the underlying operating system.
* Custom Protection Policies: Implementing specific backup intervals and recovery procedures that may not be available in managed services.
* High Resilience: Scaling databases across multiple Availability Zones or regions with custom failover configurations.
Optimization through Storage Features
A common oversight in database deployment is the potential for the storage layer to add significant value beyond simple data persistence. Amazon FSx file systems (including FSx for NetApp ONTAP, OpenZFS, and Windows File Server) enable features like:
* Snapshots and Cloning: Facilitating rapid testing and database upgrades by creating near-instantaneous copies of production environments.
* Performance Tuning: Choosing the right FSx service can significantly optimize the TCO and performance of database environments, particularly for high-transaction workloads.
Conclusion
As compute power continues to expand, the storage layer must evolve from a passive repository into a high-performance engine. By leveraging Amazon FSx and S3, organizations can eliminate storage bottlenecks, enabling their most demanding AI, HPC, and database workloads to scale linearly and cost-effectively in the cloud.
Links:
- Amazon FSx Product Page
- Amazon S3 Product Page
- AWS re:Invent 2025 – High-performance storage for AI/ML, analytics, and HPC workloads (STG336)
- AWS re:Invent 2025 – Accelerate gen AI and ML workloads with AWS storage (STG201)
- AWS re:Invent 2025 – Improve self-managed database performance and agility with Amazon FSx (STG337)
[AWSReInvent2025] Amazon S3 Performance: Architecture, Design, and Optimization for Data-Intensive Systems
Lecturer
Ian Heritage is a Senior Solutions Architect at Amazon Web Services, specializing in Amazon S3 and large-scale data storage architectures. With deep expertise in performance engineering and distributed systems, Ian Heritage helps organizations design and optimize their storage layers for high-throughput and low-latency applications, including machine learning training and real-time analytics. He is a prominent figure in the AWS storage community, known for his technical deep-dives into S3’s internal mechanics and best practices for performance at scale.
Abstract
This article explores the internal architecture and performance optimization strategies of Amazon S3, the industry-leading object storage service. It provides a detailed analysis of the differences between S3 General Purpose and the newly introduced S3 Express One Zone storage class, highlighting the architectural trade-offs between regional durability and sub-millisecond latency. The discussion covers advanced request management techniques, including prefix partitioning, request routing, and the role of the AWS Common Runtime (CRT) in maximizing throughput. By examining these technical foundations, the article offers practical guidance for architecting storage solutions that can handle millions of requests per second and petabytes of data for modern AI and analytics workloads.
S3 Storage Class Selection for High Performance
The performance of an S3-based application is fundamentally determined by the selection of the storage class. For over a decade, S3 General Purpose (Standard) has been the default choice, offering 99.999999999% (11 9s) of durability by replicating data across at least three Availability Zones. While this provides extreme reliability, the regional replication introduces a baseline latency that may be too high for certain “request-intensive” applications, such as machine learning model checkpoints or high-frequency trading logs.
To address these needs, AWS introduced S3 Express One Zone. This storage class is designed for workloads that require consistent, single-digit millisecond latency. By storing data within a single Availability Zone and utilizing a new, purpose-built architecture, Express One Zone can deliver up to 10x the performance of S3 Standard at a 50% lower request cost. This class is ideal for applications that perform frequent, small I/O operations where the overhead of regional replication would be the primary bottleneck. The choice between Standard and Express One Zone is thus a strategic decision between geographic durability and extreme performance.
Request Routing, Partitioning, and the Scale-Out Architecture
At its core, Amazon S3 is a massively distributed system that scales out to handle virtually unlimited throughput. The key to this scaling is “partitioning.” S3 automatically partitions buckets based on the object keys (names). Each partition can support a specific number of requests: 3,500 PUT/COPY/POST/DELETE requests and 5,500 GET/HEAD requests per second per prefix. For many years, users were advised to use randomized prefixes to ensure even distribution across partitions.
Modern S3 architecture has evolved to handle this automatically, but understanding prefix design remains crucial for performance. When an application’s request rate increases, S3 detects the hot spot and splits the partition to handle the load. However, this process takes time. For workloads that burst from zero to millions of requests instantly, pre-partitioning or using a wide range of prefixes is still a best practice. By spreading data across multiple prefixes (e.g., bucket/prefix1/, bucket/prefix2/), an application can linearly scale its throughput to accommodate massive concurrency, limited only by the client’s network bandwidth and CPU.
Client-Side Optimization with AWS CRT and SDKs
While the S3 service is designed for scale, the performance experienced by the end-user is often limited by the client-side implementation. To bridge this gap, AWS developed the Common Runtime (CRT) library. The CRT is a set of open-source, C-based libraries that implement high-performance networking best practices, such as automatic request retries, congestion control, and most importantly, multipart transfers.
'''
Conceptual example of enabling CRT in the AWS SDK for Python (Boto3)
'''
import boto3
from s3transfer.manager import TransferConfig
'''
The CRT allows for automatic parallelization of large object transfers
'''
config = TransferConfig(use_threads=True, max_concurrency=10)
s3 = boto3.client('s3')
s3.upload_file('large_data.zip', 'my-bucket', 'data.zip', Config=config)
The CRT automatically breaks large objects into smaller parts and uploads or downloads them in parallel. This utilizes the full network capacity of the EC2 instance and mitigates the impact of single-path network congestion. For applications using the AWS CLI or SDKs for Java, Python, and C++, opting into the CRT-based clients can result in a significant throughput increase—often double or triple the speed of standard clients for large files. Additionally, the CRT handles the complexities of DNS load balancing and connection pooling, ensuring that requests are distributed efficiently across the S3 frontend fleet.
Case Study: Optimization for Machine Learning and Analytics
Machine learning training is a premier use case for S3 performance optimization. During the training of large language models (LLMs), hundreds or thousands of GPUs must simultaneously read training data and write model “checkpoints.” These checkpoints are multi-gigabyte files that must be saved quickly to avoid idling expensive compute resources. By combining S3 Express One Zone with the CRT-based client, researchers can achieve the throughput necessary to saturate the high-speed networking of P4 and P5 instances.
In analytics, the use of “Range Gets” is a critical optimization. Instead of downloading an entire 1GB Parquet file to read a few columns, an application can request specific byte ranges. This reduces the amount of data transferred and speeds up query execution. S3 is optimized to handle these range requests efficiently, and when combined with a partitioned data layout (e.g., partitioning by date or region), it enables sub-second query responses over petabytes of data. This architectural synergy between storage class, partitioning, and client-side logic is what allows S3 to serve as the foundation for the world’s largest data lakes.