I have 3 machines: 1x Master with 4x CPU, 8G RAM ; 2x executors with 4x CPU and 16G RAM.
Even if it is not a huge infrastructure I would still expect some perf out of it. When ruing a reduce operation:
tfsent = tfsent.reduce(lambda x,y: Row(tf=spvecadd(x.tf, y.tf), sentiment=spvecadd(x.sentiment, y.sentiment)))
where tfsent has tf and sentiment which are SparseVector, and spvecadd is an home made function to add SparseVector
Doing this, on the 3x 4CPU, only one, on an executor, is ruing 100%. Others are 0%, memory is around 5G/16G. I don't get: * Why is this so long * Why is only 1x CPU working.
Should I partition my data myself? (I mean explicitely distribute the data on both executors? even if to my mind thats Spark's job).
Thanks for any help, ideas or tips pltrdy