Posts Tagged ‘SRD’
[AWSReInvent2025] Maximizing Block Storage Performance for High-Intensity Workloads: A Technical Analysis of io2 Block Express and the Nitro System
Lecturer
Mark Olsen and Jody Berenblatt are distinguished engineering and product leaders at Amazon Web Services, specializing in high-performance block storage. Mark Olsen serves as a Principal Product Manager for Amazon EBS, where he focuses on the architectural evolution of Provisioned IOPS volumes to meet the demands of mission-critical enterprise applications. Jody Berenblatt, a Senior Technical Product Manager, brings extensive expertise in the integration of storage subsystems with the AWS Nitro System and the optimization of storage networking protocols. Their work has been pivotal in the development of io2 Block Express, a storage tier designed to provide SAN-like performance in the cloud.
Abstract
This article provides a comprehensive examination of the technical foundations and performance characteristics of high-intensity block storage within the Amazon Elastic Block Store (EBS) ecosystem. Centered on the io2 Block Express architecture, the analysis explores how the integration of the AWS Nitro System, the Scalable Reliable Datagram (SRD) protocol, and Multi-Attach NVMe reservations enables ultra-low latency and high-throughput capabilities for data-intensive workloads such as SAP HANA, Oracle, and Microsoft SQL Server. The discussion details the methodology for managing tail latency, the benefits of decoupled storage architectures, and the operational strategies required to maximize I/O performance in a distributed cloud environment.
Infrastructure Foundations: The Evolution of Provisioned IOPS
The landscape of enterprise computing has shifted toward workloads that demand not only high throughput but also extreme consistency in I/O operations per second (IOPS). For decades, on-premises Storage Area Networks (SANs) were the only viable option for these applications. However, the maturation of Amazon EBS, particularly the transition from io1 to the io2 Block Express architecture, has redefined the capabilities of cloud-native block storage. The fundamental challenge in high-intensity storage is the management of latency, which is often the primary bottleneck for database performance.
In traditional storage models, performance was often tethered to the physical limitations of the disk or the controller. In the modern AWS architecture, the storage is decoupled from the compute instance, connected via a dedicated high-speed network. This separation allows for independent scaling of compute and storage resources but introduces the necessity for highly optimized networking to maintain sub-millisecond latency. The io2 Block Express volumes are engineered to provide up to 256,000 IOPS and 4,000 MB/s of throughput per volume, offering a level of performance that satisfies even the most demanding transactional databases.
Architecture of io2 Block Express: Performance and Durability
The architecture of io2 Block Express represents a paradigm shift in how block storage is provisioned and managed. Unlike standard volumes, io2 Block Express is designed to handle “high-intensity” workloads, defined by their sensitivity to latency and their requirement for high durability. These volumes provide a durability rating of 99.999%, which is a ten-fold improvement over standard io1 volumes. This reliability is achieved through sophisticated replication techniques across multiple physical hardwares within an Availability Zone.
A critical innovation in this architecture is the way it handles I/O operations. By utilizing the Nitro System, the overhead of the hypervisor is removed, allowing the EBS service to communicate directly with the instance’s memory. This “Block Express” layer acts as a high-performance interface that minimizes the processing time required for each I/O request. For applications like SAP HANA, where the speed of logging and data loading is critical, the reduced overhead translates directly into faster business processing cycles.
Networking Innovations: Scalable Reliable Datagram (SRD)
Perhaps the most significant technical advancement in maximizing block storage performance is the implementation of the Scalable Reliable Datagram (SRD) protocol. Traditional TCP protocols, while reliable, are prone to “head-of-line blocking,” where a single lost packet can delay the entire stream of data. In a high-performance storage environment, this creates “tail latency”—spikes in response time that can disrupt database synchronization and performance.
SRD solves this by utilizing multipath routing. Instead of sending data down a single network path, SRD spreads the traffic across as many as 64 different paths simultaneously. If a specific network switch becomes congested or a link fails, the protocol automatically reroutes the data without the latency spikes associated with TCP retransmissions. This protocol is implemented directly in the Nitro Cards, ensuring that the heavy lifting of network management does not consume CPU cycles on the user’s EC2 instance. The result is a more consistent “p99” latency profile, which is essential for maintaining stable performance in clustered environments.
Multi-Attach NVMe Reservations and High Availability
For enterprise applications requiring high availability, the ability for multiple EC2 instances to attach to a single EBS volume is a critical requirement. io2 Block Express supports Multi-Attach, allowing up to 16 Nitro-based instances to access the same volume simultaneously. This feature is particularly valuable for clustered file systems and applications that require shared storage for failover or parallel processing.
To manage concurrent access without data corruption, AWS implemented Multi-Attach NVMe Reservations. Based on the NVMe standard for persistent reservations (similar to SCSI-3 PR), this technology allows one instance to “reserve” the volume, ensuring that only authorized nodes can perform write operations. In the event of an instance failure, the reservation can be quickly cleared and reassigned to a healthy node, minimizing downtime. This mechanism provides the coordination layer necessary for complex deployments like Oracle RAC or SAP environments, where data integrity across multiple nodes is non-negotiable.
Observability and Performance Tuning for Enterprise Workloads
Achieving maximum performance requires a sophisticated approach to observability. Many administrators focus on average latency, but in high-intensity workloads, the “outliers” or tail latency are what truly matter. AWS provides tools such as Amazon CloudWatch and EBS Volume Insights to monitor these metrics in real-time. A key metric is the “Queue Depth,” which represents the number of pending I/O requests for a volume. To reach the full potential of an io2 Block Express volume (e.g., 256,000 IOPS), the application must maintain a sufficient queue depth—often 128 or higher—to keep the storage pipeline full.
// Example AWS CLI command to modify an EBS volume to io2 with high provisioned IOPS
aws ebs modify-volume \
--volume-id vol-0123456789abcdef \
--volume-type io2 \
--iops 100000
Furthermore, the choice of the EC2 instance type is paramount. Performance is not solely a function of the storage volume; the instance must be “EBS-optimized” with sufficient dedicated bandwidth to handle the provisioned throughput. For instance, using an R5b or X2idn instance allows the application to utilize the full 4,000 MB/s throughput offered by Block Express. Failure to match the instance capability with the volume performance will lead to throttling at the instance level, regardless of how many IOPS are provisioned.