apache / carbondata / 1574
76%

DEFAULT BRANCH: master
Ran
Jobs 1
Files 1031
Run time 57s
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
1574

push

jenkins

ravipesala 
[CARBONDATA-3300] Fixed ClassNotFoundException when using UDF in spark-shell

Analysis:
When a spark-shell is run a scala interpreter session is started which is the main thread for that shell. This session uses TranslatingClassLoader, therefore the UDF(  in the stacktrace) that is defined would be loaded into TranslatingClassLoader.

When deserialization happens an ObjectInputStream is create and the application tries to read the object, the ObjectInputStream uses a native method(sun.misc.VM.latestUserDefinedLoader() ) call to determine the ClassLoader that will be used to load the class. This native method returns URLClassLoader which is the parent of TranslatingClassLoader where the class was loaded.
Because of this ClassNotFoundException is thrown.

Class Loader Hierarchy

ExtClassLoader(head) -> AppClassLoader -> URLClassLoader -> TranslatingClassLoader

This looks like a bug in the java ObjectInputStream implementation as suggested by the following post
https://stackoverflow.com/questions/1771679/difference-between-threads-context-class-loader-and-normal-classloader

Operation	Thread	Thread ClassLoader	ClassLoader
Register	Main	Translating	Translating
Serialize	Main	Translating	Translating
Deserialize	Thread-1	Translating	URLClassLoader
Solution:
Use ClassLoaderObjectInputStream to specify the class loader that should be used to load the class.

This closes #3132

61880 of 79279 relevant lines covered (78.05%)

1.04 hits per line

Jobs
ID Job ID Ran Files Coverage
1 1574.1 0
78.05
Source Files on build 1574
Detailed source file information is not available for this build.