Spring Data Redis, Expiring and Redis Cluster

I have an app using

Spring Boot 2.2.6.RELEASE (spring-boot-starter-data-redis)
Jedis 3.1.0.

I've have a Redis 5.0.7 Cluster of 6 nodes: 3 masters and 3 slaves with replication 127.0.0.1:7000-7005 (just exemplary values).

I've configured my app this way:

@Configuration
@EnableRedisRepositories(basePackages = "my.package.of.dtos", enableKeyspaceEvents = RedisKeyValueAdapter.EnableKeyspaceEvents.ON_STARTUP)
public class RedisConfiguration {
    @Bean
    JedisConnectionFactory jedisConnectionFactory() {
        return new JedisConnectionFactory(
            new RedisClusterConfiguration(List.of(
                "127.0.0.1:7000",
                "127.0.0.1:7001",
                "127.0.0.1:7002",
                "127.0.0.1:7003",
                "127.0.0.1:7004",
                "127.0.0.1:7005")));
    }

    @Bean
    public RedisTemplate<String, Object> redisTemplate(JedisConnectionFactory jedisConnectionFactory) {
        RedisTemplate<String, Object> template = new RedisTemplate<>();
        template.setConnectionFactory(jedisConnectionFactory);
        return template;
    }
}

and I have a couple of DTOs like:

@AllArgsConstructor
@NoArgsConstructor
@Data
@Builder
@RedisHash(value = "Foo", timeToLive = 300)
public class Foo {
    private String id;
    @Indexed
    private String fooIndexedField;
    private String fooField1;
    private String fooField2;
}

and repositories:

@Repository
public interface FooRepository extends CrudRepository<Foo, String> {
    List<Foo> findByFooIndexedField(String fooIndexedField);
}

My use case: I have a heavy traffic processing app, I write data to Redis and expect to read a list of entities by the indexed field. The data is relevant only for a period of time therefore I'm taking advantage of Redis' expire feature.

Everything seemed to be working until I've noticed that the data in Redis does not expire as expected. When I'm connecting to a Redis Cluster (with RedisClusterConfiguration), once a hash expires the rest of the data associated with it and written by Spring Data Redis remains, phantom expires on its own 5minuts later, but additional sets ex Foo (with all the IDs), Foo:testId1:idx (value Foo:fooIndexedField:testIndex1) and Foo:fooIndexedField:testIndex1 (value testId1) remains there.

I've swapped the redis configuration to RedisStandaloneConfiguration (single node, for testing purposes) and all data was gone when hash expired.

The only thing I've managed to find in Spring documentation so far is: Define and pin keyspaces by using @RedisHash("{yourkeyspace}") to specific slots when you use Redis cluster. This is not something I can do. Some of the hashes need to be distributed across all the nodes since I cannot assume they will fit on one node.

The only thing that keeps my cluster from running out of memory due to orphaned indexes is the setting maxmemory_policy:allkeys-lru that overwrites them. This is troubling since I'm seeing all my nodes as using max memory all the time.

Is there anything that I've missed in my application, Spring Data Redis or Redis cluster setup?

EDIT: Configuration using Redisson redisson-spring-data-22 version 3.13.5:

@Configuration
@EnableRedisRepositories(basePackages = "my.package.of.dtos", enableKeyspaceEvents = RedisKeyValueAdapter.EnableKeyspaceEvents.ON_STARTUP)
public class RedisConfiguration {
    @Bean
    public RedissonConnectionFactory redissonConnectionFactory() {
        Config config = new Config();
        config.useClusterServers().addNodeAddress("redis://127.0.0.1:7000", "redis://127.0.0.1:7001", "redis://127.0.0.1:7002", "redis://127.0.0.1:7003", "redis://127.0.0.1:7004", "redis://127.0.0.1:7005");
        return new RedissonConnectionFactory(config);
    }


    @Bean
    public RedisTemplate<String, Object> redisTemplate(RedissonConnectionFactory redissonConnectionFactory) {
        RedisTemplate<String, Object> template = new RedisTemplate<>();
        template.setConnectionFactory(redissonConnectionFactory);
        return template;
    }
}

gave the same results, unfortunately.

Solution

Turns out Spring Data Redis is not the way to go for me. My experience with Spring Data Redis:

delegates a portion of data expiring process from Redis to the application using it
works only if data from one hash is on the same cluster node
it doesn't work correctly when the application is scaled horizontally
the peak memory usage is doubled compared to the actual data size, due to it saving data and phantom
primary and secondary indexes can become quite costly

I solved all those problems after a thorough read-though of Redis documentation. As it turns out, Redis devs have a very clear vision of how they want Redis to be used. Any deviation from that path leads to strange problems. On the other hand staying on 'the right path' means that you will get all Redis benefits without any effort.

What I did:

changed Redis library to Jedis, implemented my own very simle client based on it
limited saved data to a bare minimum
manually controlled TTL of all saved data
manually controlled entity slotting with usage of {PART_OF_KEY} (ex. for exists on multiple keys which requires them all to be on one slot)

Results I got:

reduced single data size from ~60KB to ~60B, eliminated all indexing, duplicates, etc which in turn allowed me to momentarily save a couple of orders of magnitude more data than before
took advantage of Redis optimized data expiring, no data is saved without a TTL therefore momentary memory usage is always accurate
due to selective slotting only when required I still take advantage of all the nodes that I have in the cluster but at the same time I can take advantage of all the perfomance-focused Redis calls - I never loop redis calls over multiple keys since all of them can be done in single call with multiple parameters

Disclaimer: When I started working on this I only knew the buzzwords about Redis and I did not realize it's true potential nor it's devs-envisioned usage. It all seemed like going against the tide at first, but the more I adjusted my code to the features offered by Redis the more rewaring working with it all felt. I consider Spring Data Redis an awsome tool for fast prototyping but I feel like it's ORM-like approach is like going against averything Redis offers.