I'm looking to run several similar sparsely-related jobs in parallel using parallel package in R.
The nature of the jobs is that some of them are dependant upon others where dependency is an acyclic directed graph (not a tree). Also each individual job performance can vary in time from run to run.
I would like to clusterApplyLB to load-balance the jobs between work nodes because of varying execution time and I can't determine optimal sequence of jobs beforehand due to the same reason.
Hence, given list of jobs {1,2,3,..} and adjacency matrix M:
I'd like to throw job i into the 'pool' as soon as all of its dependencies M[i] have confirmed to master their successful completion.
Is it even possible, to build up work pool on the fly using parallel? If yes, how can I achieve that?