Posts Tagged ‘GenAI’
[AWSReInvent2025] High-Performance Storage Architectures for AI/ML, Analytics, and HPC Workloads
Lecturer
Aditi is a Senior Product Manager for Amazon FSx at Amazon Web Services (AWS). With years of experience working directly with customers on high-performance workloads, she focuses on pushing the technical boundaries of what is possible with cloud storage to meet the demands of modern compute-intensive applications.
Abstract
This article examines the critical role of high-performance storage in supporting modern AI/ML, analytics, and High-Performance Computing (HPC) workloads. As organizations scale their compute resources—incorporating hundreds or thousands of CPU and GPU cores—storage often becomes the primary bottleneck, preventing linear performance scaling. We explore the technical architectures of Amazon FSx and Amazon S3, focusing on how these services address the needs of both “lift-and-shift” file-based applications and “cloud-native” S3-based data lakes. By analyzing customer use cases in genomics, media rendering, and large language model (LLM) training, we detail the methodologies for achieving peak performance at scale.
The Storage Bottleneck in Compute-Intensive Workloads
Modern high-performance workloads are characterized by their extreme reliance on massive datasets and high-core-count compute clusters. In an ideal cloud environment, adding more compute resources should lead to a proportional increase in work completed—a concept known as linear scaling. However, traditional storage solutions often fail to keep pace with the throughput demands of these clusters, leading to a performance plateau.
When storage becomes the bottleneck, compute instances sit underutilized as they compete for access to the same data store. This is particularly detrimental given that 90% to 95% of the expenditure for these workloads is typically allocated to compute resources. Consequently, an inefficient storage layer not only extends the time to insight but also significantly increases the total cost of ownership (TCO). To avoid this, storage must be architected to scale linearly alongside compute.
Navigating the Path to the Cloud: File Systems vs. Object Storage
Organizations generally approach high-performance storage on AWS from two distinct backgrounds: those with long-standing on-premises file-based workflows and those who have built native cloud applications around object storage.
The Persistence of File-Based Architectures
Despite the rise of object storage, file systems remain the preferred interface for many researchers and developers due to three primary factors: Familiar Interface: The intuitive nature of files and directories simplifies complex data management for data scientists and developers.
* Granular Permissions: File systems provide robust POSIX permissions, allowing for fine-grained control over which users can read, write, or execute specific files.
* Consistent Data Access:* For workloads where multiple users or compute nodes access the same data simultaneously, the strong consistency of file systems ensures that all parties see the most recent data updates.
Amazon FSx for High-Performance File Access
Amazon FSx addresses these needs by providing fully managed file systems that offer the performance of local storage with the scalability of the cloud. For “lift-and-shift” scenarios, FSx allows organizations to move their existing HPC and AI/ML pipelines to AWS without refactoring their applications.
Accelerating Generative AI and ML Workloads
The emergence of generative AI has placed a renewed emphasis on data strategy. Whether an organization is building a model from scratch or fine-tuning a foundational model, the quality and accessibility of its proprietary data are the primary differentiators.
Retrieval Augmented Generation (RAG)
To move beyond generic AI responses and reduce hallucinations, many organizations are implementing Retrieval Augmented Generation (RAG). RAG allows foundational models to access evolving, large-scale data lakes without requiring the data to be manually loaded into a prompt.
The RAG methodology involves:
1. Vectorization: Converting organizational data into vectors—numeric representations that capture semantic meaning.
2. Semantic Search: Using spatial similarity to compare a query vector against the data lake’s vectors to find the most relevant information.
3. Augmentation: Feeding the retrieved context back into the model to generate a more accurate and business-specific response.
Ingestion and Data Strategy with Amazon S3
Amazon S3 serves as the foundational data lake for these AI workflows due to its cost-effectiveness and virtually unlimited scalability. Organizations typically utilize two ingestion patterns:
* Batch Ingestion: Suitable for static or infrequently changing data such as historical records and product catalogs.
* Real-Time Ingestion: Essential for agentic workflows where AI models must respond to the latest available information.
Modernizing Self-Managed Databases with Amazon FSx
While fully managed services like Amazon RDS are popular, certain business and technical requirements drive organizations toward self-managed database architectures on AWS.
Drivers for Self-Managed Databases
Organizations choose to self-manage databases like Oracle, SQL Server, or SAP HANA for several reasons:
* Granular Control: The ability to choose specific versions of the database engine and the underlying operating system.
* Custom Protection Policies: Implementing specific backup intervals and recovery procedures that may not be available in managed services.
* High Resilience: Scaling databases across multiple Availability Zones or regions with custom failover configurations.
Optimization through Storage Features
A common oversight in database deployment is the potential for the storage layer to add significant value beyond simple data persistence. Amazon FSx file systems (including FSx for NetApp ONTAP, OpenZFS, and Windows File Server) enable features like:
* Snapshots and Cloning: Facilitating rapid testing and database upgrades by creating near-instantaneous copies of production environments.
* Performance Tuning: Choosing the right FSx service can significantly optimize the TCO and performance of database environments, particularly for high-transaction workloads.
Conclusion
As compute power continues to expand, the storage layer must evolve from a passive repository into a high-performance engine. By leveraging Amazon FSx and S3, organizations can eliminate storage bottlenecks, enabling their most demanding AI, HPC, and database workloads to scale linearly and cost-effectively in the cloud.
Links:
- Amazon FSx Product Page
- Amazon S3 Product Page
- AWS re:Invent 2025 – High-performance storage for AI/ML, analytics, and HPC workloads (STG336)
- AWS re:Invent 2025 – Accelerate gen AI and ML workloads with AWS storage (STG201)
- AWS re:Invent 2025 – Improve self-managed database performance and agility with Amazon FSx (STG337)
[DotJs2025] Prompting is the New Scripting: Meet GenAIScript
As generative paradigms proliferate, scripting’s syntax strains under AI’s amorphous allure—prompts as prosaic prose, yet perilous in precision. Yohan Lasorsa, Microsoft’s principal developer advocate and Angular GDE, unveiled GenAIScript at dotJS 2025, a JS-inflected idiom abstracting LLM labyrinths into lucid loops. With 15 years traversing IoT’s interstices to cloud’s canopies, Yohan likened this lexicon to jQuery’s jubilee: DOM’s discord domesticated, now GenAI’s gyrations gentled for mortal makers.
Yohan’s yarn recalled jQuery’s jihad: browser balkanization banished, events etherealized—20 years on, GenAI’s gale mirrors, models multiplying, APIs anarchic. GenAIScript’s grace: JS carapace cloaking complexities—await ai.chat('prompt') birthing banter, ai.forEach(items, 'summarize') distilling dossiers. Demos danced: file foragers (fs.readFile), prompt pipelines (ai.pipe(model).chat(query)), even AST adventurers refactoring Angular artifacts—CLI’s churn supplanted by semantic sorcery.
This superstructure spans: agents’ autonomy (ai.agent({tools})), RAG’s retrieval (ai.retrieve({query, store})), even vision’s vignettes (ai.vision(image)). Yohan’s yield: ergonomics eclipsing exhaustion—built-ins for Bedrock, Ollama; extensibility via plugins. Caveat’s cadence: tool for tinkering, not titanic tomes—yet frameworks’ fledglings may flock hither.
GenAIScript’s gospel: prompting’s poetry, scripted sans strife—democratizing discernment in AI’s ascent.
jQuery’s Echo in AI’s Era
Yohan juxtaposed jQuery’s quirk-quelling with GenAI’s gale—models’ menagerie, APIs’ anarchy. GenAIScript’s girdle: JS’s jacket jacketting journeys—chat’s cadence, forEach’s finesse.
Patterns’ Parade and Potentials
Agents’ agency, RAG’s recall—pipelines pure, vision’s vista. Yohan’s yarns: Angular migrations mended, Bedrock bridged—plugins’ pliancy promising proliferation.
Links:
[NDCOslo2024] Lessons Learned Building a GenAI Powered App – Marc Cohen & Mete Atamel
In the exhilarating epicenter of emergent engineering, where generative grammars graft onto granular goals, Marc Cohen and Mete Atamel, a dynamic duo of developer advocates, dissect the delights and dilemmas of deploying a GenAI quiz quest. Marc, a Google Cloud sage, and Mete, a London-based luminary, limn their labyrinthine launch: an interactive trivia titan, turbocharged by text-to-quiz transformers, traversing from ideation to iteration. Their tale, tempered by trials and triumphs, tempers enthusiasm with empiricism, extracting edicts for ensembles eyeing AI augmentation.
Marc and Mete meander from mundane meetings—Gemini-fueled frivolities birthing brain-teasers—to blueprinting a bespoke bot: prompts pioneering puzzles, Vertex AI vending variety. Their venture: a web wizard weaving whimsy, where users umpire uniqueness, quizzes quizzing quaestions quarterly.
Ideation to Implementation: Igniting the Interactive
Genesis gleamed in a Google gabfest: Gemini’s garrulous games germinated a gadget for GDD—Google Developer Days—gamifying gaps in grasp. Marc’s maiden foray: manual mocks, mired in monotony, morphed via Vertex AI’s verve—prompts pulsing personalities, quizzes questing quandaries.
Mete’s mastery: modularize might—microservices marshalling models, Cloud Run cradling containers. Their synergy: separation of synthesis and scrutiny, safeguards staving spurious spiels via safety settings.
Pitfalls and Panaceas: Prompting Precision
Prompts proved pivotal: personas personifying pizzazz—”pirate patter”—yet perils prowled: profanities percolating, inaccuracies amassing. Marc’s mitigation: modular mandates—system strictures scripting safeguards, few-shot finesses finagling fidelity.
Costs crept: characters cashed credits, caching curbed cascades. Their calculus: quotas quelled quiescence, quotas quashing queues.
Live Labyrinths: Latency and Learner Loops
Latency loomed large: live quizzes languished, learners lagging. Marc’s maneuver: asynchronous artistry—prefab puzzles poised, personalization post-facto. Feedback’s finesse: thumbs-up tallies tailoring topics, Vertex’s vectors vectoring variety.
Their tableau: a Twitch-streamed spectacle, spectators selecting spheres, quizzes quizzing quaestions—engagement eclipsing expectations.
Edicts Extracted: Engineering Enlightenment
Lessons luminated: prompts as poetry—precise, persistent; modularity’s merit—micro over monolith; costs as calculus—cache, cull. Marc and Mete’s missive: GenAI gamifies growth, yet guardrails guide greatness.
Links:
[NDCOslo2024] The History of Computer Art – Anders Norås
In the incandescent interstice of innovation and imagination, where algorithms awaken aesthetics, Anders Norås, a Norwegian designer and digital dreamer, traces the tantalizing trajectory of computer-generated creativity. From 1960s Silicon Valley’s psychedelic pixels to 2020s generative galleries, Anders animates an anthology of artistic audacity, where hackers harnessed harmonics and hobbyists honed holograms. His odyssey, opulent with optical illusions and ontological inquiries, unveils code as canvas, querying: when does datum dance into divinity?
Anders ambles from Bay Area’s beatnik bytes—LSD-laced labs birthing bitmap beauties—to 1970s fine artists’ foray into fractals. Vera Molnar’s algorithmic abstractions, Molnar’s mechanical marks, meld math with muse, manifesting minimalism’s machine-made magic.
Psychedelic Pixels: 1960s’ Subcultural Sparks
San Francisco’s hacker havens hummed with hallucinatory hacks: Ken Knowlton’s BEFLIX begat filmic fractals, A. Michael Noll’s noisy nudes nodded to neo-classics. Anders accentuates the alchemy: computers as collaborators, conjuring compositions that captivated cognoscenti.
Algorithmic Abstractions: 1970s’ Fine Art Fusion
Fine artists forayed into flux: Frieder Nake’s generative geometries, Georg Nees’s nested nests—exhibitions eclipsed elites, etching electronics into etudes. Harold Cohen’s AARON, an autonomous auteur, authored arabesques, blurring brushes and binaries.
Rebellious Renderings: 1980s’ Demoscene Dynamism
Demoscene’s defiant demos dazzled: Future Crew’s trance tunnels, Razor 1911’s ray-traced reveries—amateurs authored epics on 8-bits, echoing graffiti’s guerrilla glee. Anders applauds the anarchy: code as contraband, creativity’s clandestine cabal.
Digital Diaspora: Internet’s Infinite Installations
Web’s weave widened worlds: JODI’s jetset glitches, Rafael Lozano-Hemmer’s responsive realms—browsers birthed boundless biennales. Printouts prized: AARON auctions at astronomic asks, affirming artifacts’ allure.
Generative Galas: GenAI’s Grand Gesture
Anders assays AI’s ascent: Midjourney’s mirages, DALL-E’s dreams—yet decries detachment, Dolly’s depthless depictions devoid of dialogue. Jeff Wall’s “A Sudden Gust of Wind” juxtaposed: human heft versus heuristic haze, where context conceals critique.
Anders’s axiom: art awakens awareness—ideas ignite, irrespective of instrument. His entreaty: etch eternally, hand hewn, honoring humanity’s hallowed hue.
Links:
[NDCOslo2024] Mirror, Mirror: LLMs and the Illusion of Humanity – Jodie Burchell
In the mesmerizing mirror maze of machine mimicry, where words weave worlds indistinguishable from wit, Jodie Burchell, JetBrains’ data science developer advocate, shatters the spell of sentience in large language models (LLMs). A PhD psychologist turned NLP pioneer, Jodie probes the psychological ploys that propel projections of personhood onto probabilistic parsers, dissecting claims from consciousness to cognition. Her inquiry, anchored in academia and augmented by anecdotes, advises acuity: LLMs as linguistic lenses, not living likenesses, harnessing their heft while heeding hallucinations.
Jodie greets with gratitude for her gritty slot, her hipster cred in pre-prompt NLP notwithstanding. LLMs’ 2022 blaze beguiles: why bestow brains on bytes when other oracles oblige? Her hypothesis: humanity’s hall of mirrors, where models mirror our mores, eliciting empathy from echoes.
Psychological Projections: Perceiving Personhood in Parsers
Humans, Jodie hazards, hallucinate humanity: anthropomorphism’s ancient artifice, from pets to puppets. LLMs lure with language’s liquidity—coherent confessions conjure companionship. She cites stochastic parrots: parleying patterns, not pondering profundities, yet plausibility persuades.
Extraordinary assertions abound: Blake Lemoine’s LaMDA “alive,” Google’s Gemini “godhead.” Jodie juxtaposes: sentience’s scaffold—selfhood, suffering—sans in silicon. Chalmers’ conundrum: consciousness connotes qualia, quanta qualms quell in qubits.
Levels of Luminescence: From Language to Luminary
DeepMind’s AGI arc: Level 1 chatbots converse convincingly; Level 2 reasons reactively; Level 3 innovates imaginatively. LLMs linger at 1-2, lacking Level 4’s abstraction or 5’s autonomy. Jodie jests: jackdaws in jester’s garb, juggling jargon sans judgment.
Illusions intensify: theory of mind’s mirage, where models “infer” intents from inferences. Yet, benchmarks belie: ARC’s abstraction stumps, BIG-bench’s breadth baffles—brilliance brittle beyond basics.
Perils of Projection: Phishing and Philosophical Pitfalls
Prompt injections prey: upstream overrides oust origins, birthing bogus bounties—”Amazon voucher via arcane URL.” Jodie demonstrates: innocuous inquiries infected, innocuousness inverted into inducements. Robustness rankles: rebuttals rebuffed, ruses reiterated.
Her remedy: recognize reflections—lossy compressions of lore, not luminous lives. Demystify to deploy: distill data, detect delusions, design defensively.
Dispelling the Delusion: Harnessing Heuristics Humanely
Jodie’s jeremiad: myths mislead, magnifying misuses—overreach in oracles, oversight in safeguards. Her horizon: LLMs as lucid lenses, amplifying analysis while acknowledging artifice.
Links:
[DotAI2024] DotAI 2024: Eliot Andres – From Scratch to Scale: Crafting and Cascading Foundational Image Models
Eliot Andres, co-founder and CTO of Photoroom, chronicled the odyssey of bespoke vision at DotAI 2024. With nine years honing deep learning for e-commerce elixirs, Andres propelled Photoroom’s ascent—YC S20 alumna serving global galleries. His narrative dissected in-house genesis over off-the-shelf oracles, unveiling diffusion’s dawn-to-dusk: bespoke blueprints, data distillations, compute conquests, and feedback forges yielding thrice-swift sorcery for millions.
Forging Foundations Beyond Borrowed Blueprints
Andres interrogated imitation’s insufficiency: Stable Diffusion’s splendor suits savants, yet falters for Photoroom’s precinct—product portraits purged of props, shadows sculpted sans seams. Off-the-shelf oracles, he observed, ossify on outliers: e-commerce ephemera demands domain devotion.
Thus, genesis from void: custom cascades commencing with chromatic chaos—splashes sans structure—escalating to entity emergence post-thousand-hour tutelage, culminating in crystalline compositions after 40,000 epochs. Andres attributed ascent to architectural autonomy: latent labyrinths laced with proprietary priors, data distilled from decamillions of dealer dossiers—curated for commerce’s cadence.
Compute’s crucible: H100 hordes harnessed in harmonious herds, mitigating mishaps via meticulous monitoring—gradient guardians averting gradients’ ghosts.
Navigating Novelties and Nurturing at Nascent
Andres aired adversities: data’s deluge demands discernment—deduping dross, equilibrating epochs—while scaling summons stability, feedback’s fount from frontline forges finessing flaws. Photoroom’s polity: purveyors as partners, iterating on idiosyncrasies like luminous lapses or artifact anomalies.
Deployment’s decree: distillation’s dual dance—LCM’s leapfrog lessons compressing cascades to sprints—and TensorRT’s transmutations, fusing fluxions for fleet-footed fruition, doubling dispatch sans diminishment.
FR 2030’s fellowship fuels forthcoming: grander guardians, verisimilar visions—velocity unyielding. Andres beckoned bibliophiles to GitHub’s groves: datasets as doorways, teams as talismans—collaborative conquests crowning communal code.
In tableau, Andres toasted tandemry: machine learning’s mosaic, indivisible from ingenuity’s impetus—Photoroom’s pantheon, propelling pixels to panoramas.