Spark and huge shared objects

ساخت وبلاگ

Vote count: 0

Can I have a shared object between executors on the same worker?

For example, I have some data in files. This data should be compiled before I can use it. After compilation it is not-serializable Java object. But unfortunately compilation takes a lot of time. I want to compile it once on each worker and use this object in all tasks on the worker.

Could you give me some advice how I can achieve this?

Thanks.

asked 1 min ago

back soft...
ما را در سایت back soft دنبال می کنید

برچسب : spark huge,huge plumper spark, نویسنده : استخدام کار backsoft بازدید : 212 تاريخ : دوشنبه 15 شهريور 1395 ساعت: 22:17