I was getting below error for one of my spark job -
Task not serializable: java.io.NotSerializableException: java.lang.reflect.Field
I realised that I had a class in one of the closures which was using keeping a java.lang.reflect.Field array in the memory. As java.lang.reflect.Field does not implement java.io.Serializable it cant be serialised by Spark and hence the error. How can I solve this error? It is not possible to not use Field.
- Create a new instance of class in each executor so it is not serialized. I used https://www.nicolaferraro.me/2016/02/22/using-non-serializable-objects-in-apache-spark/ which seemed to work for one object type but not for two.
- Use Kryo serializer for my class which is using Field - Will this work? Does Kryo works with classes that don't implement java.io.Serializable.
- Use broadcast variable - I am not sure about this. Does broadcast variable has to be serializable?
What are the other ways to solve this NotSerializableException when an object is truly cant be serialized for whatever reason.