I want to save from spark-streaming to couple of elastic-search indices. I create pairs of <key(index), value>, when I execute groupByKey the result is Tuple of <key(index), Iterable<value>> but in order to save to elasticsearch using elasticsearch-spark plugin I need the values as JavaRDD<value>.
I know that there is an option of sparkContext.parallelize(list) which creates JavaRDD from list, but this can be executed only on the driver.
Is there another option to create JavaRDD that can be executed on the executor? Or another way that I can achieve Tuple2<key(index), JavaRDD<value>> which works on executor? If not how can I make only the switch of Iterator to JavaRDD on driver and the plugin writing to elasticsearch at the executor?
Thanks,
Daniela