Skip to main content

Redis Cluster Support

This document describes how the Tracker GraphQL application supports Redis Cluster deployments, including AWS MemoryDB.

Overview​

The application supports both standalone Redis instances and Redis Cluster deployments. Redis Cluster is a distributed implementation of Redis that provides:

  • Automatic data sharding across multiple Redis nodes
  • High availability through master-slave replication
  • Automatic failover when a master node is unavailable

AWS MemoryDB is a Redis-compatible, fully managed database service that uses a Redis Cluster architecture and requires TLS/SSL connections.

CROSSSLOT Error​

When using Redis Cluster, you might encounter the following error:

redis.exceptions.ResponseError: CROSSSLOT Keys in request don't hash to the same slot

What Causes This Error?​

In Redis Cluster, keys are distributed across different nodes using a hashing algorithm. Each key is assigned to a specific "slot" (0-16383) based on a hash of the key name. When performing operations that involve multiple keys (like transactions or certain commands), all keys must hash to the same slot.

The CROSSSLOT error occurs when you try to perform an operation on multiple keys that hash to different slots in the cluster.

Our Solution​

To solve this issue, the application uses hash tags in key names. In Redis Cluster, if a key contains a section enclosed in curly braces {}, only that section is used for hash calculation. This allows you to ensure that related keys are assigned to the same slot.

For example, these keys will all hash to the same slot because they share the same hash tag {tracker}:

{tracker}:pending
{tracker}:processing:123
{tracker}:metadata:456
{tracker}:workers

Implementation Details​

Automatic Detection​

The application automatically detects Redis Cluster endpoints by checking for "clustercfg" in the hostname (which is present in AWS MemoryDB endpoints). When a Redis Cluster is detected, it:

  1. Uses the RedisCluster client instead of the standard Redis client
  2. Enables TLS/SSL for secure connections (required by AWS MemoryDB)
  3. Uses hash tags in all key names

Key Naming Convention​

All Redis keys in the application follow this pattern:

Standalone Redis​

prefix:suffix

Redis Cluster​

{prefix}:suffix

The curly braces ensure that all keys with the same prefix hash to the same slot in the cluster.

Code Example​

# Detect Redis Cluster
is_cluster = "clustercfg" in redis_host.lower()

if is_cluster:
    # Use RedisCluster client for Redis Cluster
    redis_client = RedisCluster(
        host=redis_host,
        port=redis_port,
        username=redis_username,
        password=redis_password,
        ssl=True  # TLS is required for AWS MemoryDB
    )

    # Use hash tags in key names
    key = f"{{tracker}}:pending"
else:
    # Use standard Redis client for standalone Redis
    redis_client = Redis(
        host=redis_host,
        port=redis_port,
        username=redis_username,
        password=redis_password,
        ssl=redis_ssl,
        db=redis_db
    )

    # No hash tags needed
    key = "tracker:pending"

Backward Compatibility​

The changes made to support Redis Cluster are fully backward compatible with standalone Redis instances:

  1. Hash tags in key names: In standalone Redis, the curly braces in keys like {tracker}:pending are just treated as regular characters with no special meaning. They don't affect functionality at all.

  2. Automatic client detection: The code detects Redis Cluster endpoints and only uses the RedisCluster client when needed. For regular Redis instances, it continues to use the standard Redis client.

  3. Conditional TLS: The TLS/SSL support is only enabled when explicitly configured, so it won't affect existing deployments.

Configuration​

Standalone Redis​

REDIS_HOST=redis
REDIS_PORT=6379
REDIS_USERNAME=
REDIS_PASSWORD=your_password
REDIS_SSL=false
REDIS_DB=0

Redis Cluster (AWS MemoryDB)​

REDIS_HOST=clustercfg.tracker.xxxxx.memorydb.region.amazonaws.com
REDIS_PORT=6379
REDIS_USERNAME=your_username
REDIS_PASSWORD=your_password
REDIS_SSL=true
# REDIS_DB is ignored for Redis Cluster

Testing Redis Cluster Connections​

You can test your Redis Cluster connection using the provided utility script:

cd fetcher/build
python test_redis_connection.py --host clustercfg.tracker.xxxxx.memorydb.region.amazonaws.com --port 6379 --username your_username --password your_password --ssl

Troubleshooting​

CROSSSLOT Errors​

If you still encounter CROSSSLOT errors:

  1. Check that all related keys use the same hash tag (text inside curly braces)
  2. Ensure that multi-key operations only use keys with the same hash tag
  3. Verify that the Redis client is correctly detecting the cluster configuration

Connection Issues​

For AWS MemoryDB:

  1. Ensure TLS/SSL is enabled (REDIS_SSL=true)
  2. Verify that both username and password are provided
  3. Check that your security groups allow access to port 6379
  4. Use the redis-cli with the --tls flag to test the connection:
redis-cli -h clustercfg.tracker.xxxxx.memorydb.region.amazonaws.com --tls -u your_username -a your_password ping

Best Practices​

  1. Key Design:

    • Always use hash tags for related keys
    • Keep hash tags consistent within functional groups
    • Use descriptive prefixes for better organization

  2. Operations:

    • Avoid multi-key operations across different hash tags
    • Use Lua scripts for complex operations (they execute on a single node)
    • Be aware of cluster topology changes during scaling events

  3. Monitoring:

    • Monitor slot distribution and rebalancing events
    • Watch for node failures and failover events
    • Track connection pool metrics