Jonathan Lalou's Blog

Posts Tagged ‘Redis’

In today’s distributed, cloud-native environments, shared caching is no longer an optimization—it’s a necessity. Whether you’re scaling out web servers, deploying stateless containers, or orchestrating microservices in Kubernetes, a centralized, fast-access cache is a cornerstone for performance and resilience.

This post explores why Redis, especially via Amazon ElastiCache, is an exceptional choice for this use case—and how you can use it in production-grade AWS architectures.

🔧 Why Use Redis for Shared Caching?

Redis (REmote DIctionary Server) is an in-memory key-value data store renowned for:

• Lightning-fast performance (sub-millisecond)
• Built-in data structures: Lists, Sets, Hashes, Sorted Sets, Streams
• Atomic operations: Perfect for counters, locks, session control
• TTL and eviction policies: Cache data that expires automatically
• Wide language support: Python, Java, Node.js, Go, and more

☁️ Redis in AWS: Use ElastiCache for Simplicity & Scale

Instead of self-managing Redis on EC2, AWS offers Amazon ElastiCache for Redis:

• Fully managed Redis with patching, backups, monitoring
• Multi-AZ support with automatic failover
• Clustered mode for horizontal scaling
• Encryption, VPC isolation, IAM authentication

ElastiCache enables you to focus on application logic, not infrastructure.

🌐 Real-World Use Cases

📈 Architecture Pattern: Cache-Aside with Redis

Here’s the common cache-aside strategy:

1. App queries Redis for a key.
2. If hit ✅, return cached value.
3. If miss ❌, query DB, store result in Redis.

Python Example with redis and psycopg2:

import redis
import psycopg2
import json

r = redis.Redis(host='my-redis-host', port=6379, db=0)
conn = psycopg2.connect(dsn="...")

def get_user(user_id):
    cached = r.get(f"user:{user_id}")
    if cached:
        return json.loads(cached)

    with conn.cursor() as cur:
        cur.execute("SELECT id, name FROM users WHERE id = %s", (user_id,))
        user = cur.fetchone()
        if user:
            r.setex(f"user:{user_id}", 3600, json.dumps({'id': user[0], 'name': user[1]}))
        return user

🌍 Multi-Tiered Caching

To reduce Redis load and latency further:

• Tier 1: In-process (e.g., Guava, Caffeine)
• Tier 2: Redis (ElastiCache)
• Tier 3: Database (RDS, DynamoDB)

This pattern ensures that most reads are served from memory.

⚠️ Common Pitfalls to Avoid

🌐 Bonus: Redis for Lambda

Lambda is stateless, so Redis is perfect for:

• Shared rate limiting
• Caching computed values
• Centralized coordination

Use redis-py, ioredis, or lettuce in your function code.

🔺 Conclusion

If you’re building modern apps on AWS, ElastiCache with Redis is a must-have for state sharing, performance, and reliability. It plays well with EC2, ECS, Lambda, and everything in between. It’s mature, scalable, and robust.

Whether you’re running a high-scale SaaS or a small internal app, Redis gives you a major performance edge without locking you into complexity.

Alexandre Daubois, lead Symfony developer at Wanadev Digital, delivered a concise yet impactful session at Forum PHP 2023, spotlighting the power of Predis, a PHP client for Redis. Focusing on his team’s work at Wanadev Digital, Alexandre shared how Predis’s stream capabilities resolved critical performance issues in their 3D home modeling tool, Kozikaza. His talk highlighted practical applications of Redis streams, inspiring developers to leverage this underutilized tool for efficient data handling.

The Power of Redis Streams

Alexandre introduced Redis streams as a lightweight, in-memory data structure ideal for handling large datasets. At Wanadev Digital, the Kozikaza platform, which enables users to design 3D home models in browsers, faced challenges with storing and processing large JSON models. Alexandre explained how Predis’s stream functionality allowed his team to write data incrementally to cloud storage, avoiding memory bottlenecks. This approach enabled Kozikaza to handle massive datasets, such as 50GB JSON files, efficiently.

Solving Real-World Challenges

Detailing the implementation, Alexandre described how Predis’s Lazy Stream feature facilitated piecewise data writing to cloud buckets, resolving memory constraints in Kozikaza’s workflow. He shared user behavior insights, noting that long session times (up to six hours) made initial load times less critical, as users kept the application open. This context allowed Alexandre’s team to prioritize functionality over premature optimization, using Predis to deliver a robust solution under tight deadlines.

Links:

• Wanadev Digital website
• Kozikaza website

Chayim Kirshen, a veteran of the startup ecosystem and client manager at Redis, captivated audiences at Devoxx Poland 2022 with a dynamic exploration of real-time data pipeline challenges. Drawing from his experience with high-stakes environments, including a 2010 stock exchange meltdown, Chayim outlined strategies to ensure data integrity and performance in large-scale systems. His talk provided actionable insights for developers, emphasizing the importance of storing raw data, parsing in real time, and leveraging technologies like Redis to address data inconsistencies.

The Perils of Unclean Data

Chayim began with a stark reality: data is rarely clean. Recounting a 2010 incident where hackers compromised a major stock exchange’s API, he highlighted the cascading effects of unreliable data on real-time markets. Data pipelines face issues like inconsistent formats (CSV, JSON, XML), changing sources (e.g., shifting API endpoints), and service reliability, with modern systems often tolerating over a thousand minutes of downtime annually. These challenges disrupt real-time processing, critical for applications like stock exchanges or ad bidding networks requiring sub-100ms responses. Chayim advocated treating data as programmable code, enabling developers to address issues systematically rather than reactively.

Building Robust Data Pipelines

To tackle these issues, Chayim proposed a structured approach to data pipeline design. Storing raw data indefinitely—whether in S3, Redis, or other storage—ensures a fallback for reprocessing. Parsing data in real time, using defined schemas, allows immediate usability while preserving raw inputs. Bulk changes, such as SQL bulk inserts or Redis pipelines, reduce network overhead, critical for high-throughput systems. Chayim emphasized scheduling regular backfills to re-import historical data, ensuring consistency despite source changes. For example, a stock exchange’s ticker symbol updates (e.g., Fitbit to Google) require ongoing reprocessing to maintain accuracy. Horizontal scaling, using disposable nodes, enhances availability and resilience, avoiding single points of failure.

Real-Time Enrichment and Redis Integration

Data enrichment, such as calculating stock bid-ask spreads or market cap changes, should occur post-ingestion to avoid slowing the pipeline. Chayim showcased Redis, particularly its Gears and JSON modules, for real-time data processing. Redis acts as a buffer, storing raw JSON and replicating it to traditional databases like PostgreSQL or MySQL. Using Redis Gears, developers can execute functions within the database, minimizing network costs and enabling rapid enrichment. For instance, calculating a stock’s daily percentage change can run directly in Redis, streamlining analytics. Chayim highlighted Python-based tools like Celery for scheduling backfills and enrichments, allowing asynchronous processing and failure retries without disrupting the main pipeline.

Scaling and Future-Proofing

Chayim stressed horizontal scaling to distribute workloads geographically, placing data closer to users for low-latency access, as seen in ad networks. By using Redis for real-time writes and offloading to workers via Celery, developers can manage millions of daily entries, such as stock ticks, without performance bottlenecks. Scheduled backfills address data gaps, like API schema changes (e.g., integer to string conversions), by reprocessing raw data. This approach, combined with infrastructure-as-code tools like Terraform, ensures scalability and adaptability, allowing organizations to focus on business logic rather than data management overhead.

Links:

• Redis company website