I have a Spring Cloud based application with multiple backend Spring-Boot servers. The servers are all sharing their session with a Redis server via @EnableRedisHttpSession.
I'm trying to configure the application with master-slave configuration and Redis Sentinel. My setup is 1 master, two slaves, and 3 sentinels.
I keep getting issues with the environment. At first, I was getting an error saying that my slave it READONLY. So I had to set all slaves to READWRITE. Then, I still got this error. Tus out that when a slave was promoted to master, the old master was tued into slave, with the default READONLY setting. So I had to set the master to READWRITE as well. These are my commands to boot the redis servers (I know it makes no sense to use the same machine, it's for testing purposes at this stage):
Boot master:
redis-server --slave-read-only no --port 6379
Boot slaves:
redis-server --slaveof 127.0.0.1 6379 --slave-read-only no --port 6380
redis-server --slaveof 127.0.0.1 6379 --slave-read-only no --port 6381
application.properties/yml config per spring-boot server participating in the session:
#Redis config
spring.redis.sentinel.master: mymaster
spring.redis.sentinel.nodes: localhost:5000,localhost:5001,localhost:5002
Redis sentinel config:
redis-sentinel sentinel-0.conf
redis-sentinel sentinel-1.conf
redis-sentinel sentinel-2.conf
The conf files are:
port 5000
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 60000
sentinel config-epoch mymaster 6
For the other two sentinels it's port 5001 and 5002
Now, as long as everything is normal and there are no issues, everything works fine.
However, once I kill the master redis server, things stop working. First, I get the following error:
There was an unexpected error (type=Inteal Server Error, status=500).
Caot get Jedis coection; nested exception is redis.clients.jedis.exceptions.JedisCoectionException: Could not get a resource from the pool
What's worse, when I restart my application, things stop working as well. There are no real errors per say, but the application simply won't work. For example, instead of serving me content, I get a page with just some text (no javascript/css/images). This is usually an indication that it caot find the session id it needs to use. And this happens even after logout.
I noticed that when I kill the master server, redis sentinel rewrites my config files. For example, after killing the master redis server, the sentinel.conf now appears like this:
port 5000
sentinel monitor mymaster 127.0.0.1 6381 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 60000
sentinel config-epoch mymaster 6
# Generated by CONFIG REWRITE
dir "/user/odedia/.jenkins/workspace/boot_redis_sentinel_0"
sentinel leader-epoch mymaster 6
sentinel known-slave mymaster 127.0.0.1 6380
sentinel known-slave mymaster 127.0.0.1 6379
sentinel known-sentinel mymaster 127.0.0.1 5002 28d5fdf14fcc023045f00b00fcdbd4d4f4554611
sentinel known-sentinel mymaster 127.0.0.1 5001 a492f1f0e270fa67c22e057fbd12e981811a79fe
sentinel current-epoch 6
And now these settings remain as is even after restarting all servers/redis nodes. I can only get the application working again by manually fixing the config files.
Clearly I can't use this solution at the moment. Any help would be greatly appreciated.
Thanks.