I have a rails activerecord project that has been scaled out to serve approximately 60-100k requests per minute. We use AWS and it takes about 5 xlarge c4 ec2 instances to serve this many requests. We have optimized the system to serve 99.99% of those requests off of a redis cache rendering our mysql DB barely used.
This is great and all but we keep ruing into coection limits for mysql. Amazon RDS apparently limits the number of coections we can have and it seems silly for upping our RDS instance size just so that we can have a larger number of sleeping coections. We literally profiled the RDS server and it maybe gets 10-50 queries a day depending on how many times we update the system.
Is there any way to keep the activerecord coection pool from reserving coections?
We tried simply lowering the coection pool and it helped in lowering coections, but then we started getting:
(ActiveRecord::CoectionTimeoutError) "could not obtain a database coection within 5 seconds
What I would like to achieve is for the rails project to stop trying to pre-allocate coections and only open then up when they are necessary. I'm not well-versed enough in the rails framework and activerecord to understand how the system is reserving coections and why we are getting CoectionTimeoutErrors even though the application isn't even making any DB calls.