apache / carbondata / 1698
76%

DEFAULT BRANCH: master
Ran
Jobs 1
Files 1054
Run time 55s
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
1698

push

jenkins

ravipesala 
[CARBONDATA-3309] MV datamap supports Spark 2.1

[Problem]
MV datamap doesn't support Spark 2.1 version, so we need to support it

[Solution]
The following is the modification point and all MV test cases are passed on spark 2.1 version

The Class we cann’t access in Spark 2.1 version
(1). org.apache.spark.internal.Logging
(2). org.apache.spark.sql.internal.SQLConf
Solution:Create class extends above classed

The Class that Spark 2.1 version doesn’t have
(1). org.apache.spark.sql.catalyst.plans.logical.Subquery
(2). org.apache.spark.sql.catalyst.catalog.interface.HiveTableRelation
Solution: Use CatalogRelation instead and don’t use (in LogicalPlanSignatureGenerator) Mv the Subquery code to carbon project

The method that we can’t access in Spark 2.1 version
(1). sparkSession.sessionState.catalog.lookupRelation
Solution: Solution:Add this method of SparkSQLUtil

The changes of some class
(1). org.apache.spark.sql.catalyst.expressions.SortOrder
(2). org.apache.spark.sql.catalyst.expressions.Cast
(3). org.apache.spark.sql.catalyst.plans.Statistics
Solution: Adapt the new interface

The method that Spark 2.1 version doesn’t have
(1). normalizeExprId,canonicalized of org.apache.spark.sql.catalyst.plans.QueryPlan
(2). CASE_SENSITIVE of SQLConf
(3). STARSCHEMA_DETECTION of SQLConf
Solution:Don’t use normalize , canonicalize and the CASE_SENSITIVE, STARSCHEMA_DETECTION

Some logicplan optimization rules that Spark 2.1 version doesn’t have
(1). SimplifyCreateMapOps
(2). SimplifyCreateArrayOps
(3). SimplifyCreateStructOps
(4). RemoveRedundantProject
(5). RemoveRedundantAliases
(6). PullupCorrelatedPredicates
(7). ReplaceDeduplicateWithAggregate
(8). EliminateView
Solution: Delete or move the code to carbon project

Generate the instance in SparkSQLUtil to adapt Spark 2.1 version

Query SQL pass the MV check in Spark 2.1 version(CarbonSessionState)

This closes #3150

64010 of 81575 relevant lines covered (78.47%)

1.04 hits per line

Jobs
ID Job ID Ran Files Coverage
1 1698.1 0
78.47
Source Files on build 1698
Detailed source file information is not available for this build.