I’m working on a webapp-recommendation project.
The idea is that an admin can load a dataset, and a model will be generated accordingly.
When a user logs in, his data will be loaded, since I need to know which objects he engaged, to prevent recommending them.
The data set is stored on S3, and it’s mostly around 30MB, but may be big – as 1GB.
to serve a user (give him recommendations) I need to load his engaged objects, which are not many, but I need to load the entire dataset for this. So currently to serve a user I need to download the entire dataset from s3 into memory – and clearly this is not efficient.
The options I thought of are as following:
-
Store the entire dataset (there maybe more than one data set of the size stated above) in the web database – this doesnt seem efficient to me.
-
Keep storing the dataset on S3, and load it when a user logs in. query the relevant user data from the file, and save it on the product database – this way only active user data will be saved (e.g, only users that logged in in the last 3 months)
-
When an admin will upload the data, I will split it to many files - a file for each user – this maybe effective, but will create many files on S3, and lots of overhead.
What do you think about these options? Are there any other options?