581
76%

Ran 27 Jun 2018 08:21AM UTC

Jobs 1

Files 1035

Run time 49s

Badge

Embed ▾

pending completion

Build # 581

Build Type

push

jenkins

Committed by ravipesala

Commit Message

[CARBONDATA-2608] SDK Support JSON data loading directly (without AVRO conversion)

What changes were proposed in this pull request?
Currently SDK Support JSON data loading only with AVRO.
So, converting json to avro record and avro to carbon object is a two step process. Hence there is a need for a new carbonWriter that works with Json without AVRO.
This PR implents that.

Highlights:
#Works with just the json data and carbon schema.
#Implements hadoop's FileInputFormat to create JsonInputFormat.
#supports reading multiple json files in a folder.
#supports reading json data in multiline with record identifier.
#supports single row json read and write.
#Handles bad records when loading json data
#Supports n-level nesting of complex types as josn data

This closes #2384

Run Details

261 of 261 new or added lines in 11 files covered. (100.0%)

51378 of 71306 relevant lines covered (72.05%)

0.97 hits per line

New Missed Lines in Diff

Lines	Coverage	∆	File
1	95.65		processing/src/main/java/org/apache/carbondata/processing/loading/steps/JsonInputProcessorStepImpl.java
1	80.0	0.36%	processing/src/main/java/org/apache/carbondata/processing/util/CarbonDataProcessorUtil.java
2	83.69	0.0%	processing/src/main/java/org/apache/carbondata/processing/loading/steps/InputProcessorStepWithNoConverterImpl.java
3	89.29		store/sdk/src/main/java/org/apache/carbondata/sdk/file/JsonCarbonWriter.java
4	93.92	-1.98%	processing/src/main/java/org/apache/carbondata/processing/loading/DataLoadProcessBuilder.java
4	94.12		processing/src/main/java/org/apache/carbondata/processing/loading/parser/impl/JsonRowParser.java
31	0.0		processing/src/main/java/org/apache/carbondata/processing/loading/jsoninput/JsonStreamReader.java
74	0.0		processing/src/main/java/org/apache/carbondata/processing/loading/jsoninput/JsonInputFormat.java

Uncovered Existing Lines

Lines	Coverage	∆	File
1	82.66	1.02%	core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java
1	79.07	-0.93%	processing/src/main/java/org/apache/carbondata/processing/loading/steps/InputProcessorStepImpl.java
1	78.57	-1.79%	processing/src/main/java/org/apache/carbondata/processing/loading/iterator/CarbonOutputIteratorWrapper.java
1	85.0	-1.25%	integration/spark2/src/main/scala/org/apache/carbondata/stream/StreamJobManager.scala
2	87.04	-3.7%	processing/src/main/java/org/apache/carbondata/processing/loading/TableProcessingOperations.java
3	57.14	-42.86%	core/src/main/java/org/apache/carbondata/core/scan/model/ProjectionDimension.java
3	72.46	0.14%	integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
5	82.86	-1.99%	core/src/main/java/org/apache/carbondata/core/scan/complextypes/ArrayQueryType.java
6	82.93	-3.56%	core/src/main/java/org/apache/carbondata/core/metadata/schema/table/column/CarbonDimension.java
6	84.78	-0.84%	core/src/main/java/org/apache/carbondata/core/scan/executor/impl/AbstractQueryExecutor.java
7	64.38	-9.59%	integration/spark-common/src/main/scala/org/apache/spark/sql/test/util/QueryTest.scala
15	68.42	0.34%	core/src/main/java/org/apache/carbondata/core/scan/executor/util/QueryUtil.java
20	74.7	-8.97%	integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDatasourceHadoopRelation.scala
20	64.62	-10.86%	core/src/main/java/org/apache/carbondata/core/scan/complextypes/PrimitiveQueryType.java
26	55.56	-31.62%	core/src/main/java/org/apache/carbondata/core/scan/complextypes/StructQueryType.java
26	87.74	-8.54%	core/src/main/java/org/apache/carbondata/core/scan/executor/util/RestructureUtil.java
32	0.0	0.0%	integration/presto/src/main/java/org/apache/carbondata/presto/PrestoFilterUtil.java
36	89.6	-2.71%	core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java
52	66.01	-27.17%	core/src/main/java/org/apache/carbondata/core/scan/collector/impl/DictionaryBasedResultCollector.java
63	66.35	0.58%	core/src/main/java/org/apache/carbondata/core/util/DataTypeUtil.java
82	47.1	-48.82%	core/src/main/java/org/apache/carbondata/core/scan/model/QueryModelBuilder.java

Jobs

ID	Job ID	Ran	Files	Coverage
1	581.1	27 Jun 2018 08:21AM UTC	1035	72.05

apache / carbondata / 581
76%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

New Missed Lines in Diff

Uncovered Existing Lines

Jobs

Source Files on build 581

apache / carbondata / 581 76%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

New Missed Lines in Diff

Uncovered Existing Lines

Jobs

Source Files on build 581

apache / carbondata / 581
76%

README BADGES
x