apache / carbondata

80%

master: 76%

LAST BUILD ON BRANCH master_perf_vector

branch: master_perf_vector

CHANGE BRANCH

x

Reset

• master_perf_vector
• 0810_support_zstd_compressor_final_store
• 0813_read_compressor_from_datafiles
• 0814_bloom_sdv
• 0815_bug_bloom_dict_write
• 0912_customize_compressor
• 0914_refactor_compressor
• 0921_bug_compressor_adaptiveDeltaIntegralCodec
• 0921_zstd_doc
• 0924_bloom_compatibility
• 0928_bug_drop_bloom_cache
• 0928_fix_bloom_jvm_crash
• 1
• 101101_opt_debug_log
• 181012_longstring_df_append_seq
• 181016_opt_merge_index_method
• 181016_opt_multiple_dir_4_compaction
• 181018_bug_remove_property
• 181019_bug_spark_ds_windows_error
• 181019_bug_write_longstring_sparkfileformat
• 181022_bug_doc
• 181023_opt_unsafe_memory
• 181026_opt_property_template
• 181029_bug_close_stream
• 181029_bug_error_cli_windows_test
• 181102_bug_compaction_cores
• 181105_opt_prefetch_compaction
• 181125_bug_minmax_dm
• 21dec
• 23-nov
• 2823
• 2863
• Adaptive_Encoding
• AllCol
• BugFix
• CARBONDATA-1890
• CARBONDATA-2059
• CARBONDATA-2243
• CARBONDATA-2595
• CARBONDATA-2755
• CARBONDATA-2818
• CARBONDATA-2854
• CARBONDATA-2885
• CARBONDATA-2896_TestCases
• CARBONDATA-2912_twoInsert1.3.2
• CARBONDATA-2925
• CARBONDATA-2932-CarbonReaderProjection
• CARBONDATA-2939-compatibilitySDK
• CARBONDATA-2946
• CARBONDATA-2947
• CARBONDATA-2952_CarbonReaderCppAPI
• CARBONDATA-2956_CarbonReaderSupportConfiguration
• CARBONDATA-2975
• CARBONDATA-2981_primitiveDataType
• CARBONDATA-2982_supportArrayStringInSchema
• CARBONDATA-2986
• CARBONDATA-2991
• CARBONDATA-2996_SchemaSupportFolder
• CARBONDATA-2997_supportReadSchema
• CARBONDATA-2998
• CARBONDATA-2999_readSchemaFromS3
• CARBONDATA-3000_writeData
• CARBONDATA-3002_FixSpellError2
• CARBONDATA-3002_FixSpellError3
• CARBONDATA-3002_fixSpellError
• CARBONDATA-3003_supportBatchRow
• CARBONDATA-3019
• CARBONDATA-3023
• CARBONDATA-3027MemoryException
• CARBONDATA-3030_removeParameter
• CARBONDATA-3032_removeBlockletSize
• CARBONDATA-3038_RefactorConfiguration
• CARBONDATA-3063_setProperty
• CARBONDATA-3073_TableProperties
• CARBONDATA-3079_mergeColumn
• CARBONDATA-3093_RefactorConfiguration
• CARBONDATA-3094_cocurrentRead
• CARBONDATA-3095_OptimizeDoc
• CARBONDATA-3096
• CARBONDATA-3096_Branch
• CARBONDATA-3097_getVersionDetails
• CARBONDATA-3102_fixBeelineThriftServerError
• CARBONDATA-3108_CarbonRowJvmCrash
• CARBONDATA-3109_getLengthInCarbonROw
• CARBONDATA-3116_queryOnDataMap
• CARBONDATA-3127
• CARBONDATA-3128
• CARBONDATA-3131
• CARBONDATA-3145
• CARBONDATA-3154_FixSpark2_1_0TestError
• CARBONDATA-3160
• CARBONDATA-3165
• CARBONDATA-3167_DataFrameSDKS3
• CARBONDATA-3190_OpenExampleStyleCheck
• CARBONDATA-3199
• CARBONDATA-3213
• CARBONDATA-3215_OptimizeDoc
• CARBONDATA-3216_FixBugsOfCSDK
• CARBONDATA-3221_SDKSupportMultipleFiles
• CARBONDATA-3224_ValidateImproperValue
• CARBONDATA-3229
• CARBONDATA-3232_AddTestCaseForAlluxio
• CARBONDATA-3232_OptimizeSupportAlluxio
• CARBONDATA-3241
• CARBONDATA-3250_OptimizeHive
• CARBONDATA-3251_UT2_1Error
• CARBONDATA-3255_supportBinary
• CARBONDATA-3256
• CARBONDATA-3258
• CARBONDATA-3271_pythonSDK
• CARBONDATA-3275_fixCIErrors
• CARBONDATA-3280_batchFailed
• CARBONDATA-3293
• CARBONDATA-3293_1
• CARBONDATA-3294
• CARBONDATA-3297
• CARBONDATA-3313
• CARBONDATA-3317
• CARBONDATA-3336_Binary
• CARBONDATA-3342_filterNon
• CARBONDATA-3342_fixFilterNonIssue
• CARBONDATA-3356_DataSourceReadSDKVarchar
• CARBONDATA-3358_Binary_supportConfigurableDecode
• CARBONDATA-3363_SDKReadFilelist
• CARBONDATA-3370
• CARBONDATA-3374_optimizeDocFixSpellError
• CARBONDATA-3380
• CARBONDATA-3383
• CARBONDATA-3386
• CARBONDATA-3393
• CARBONDATA-3397
• CARBONDATA-3400
• CARBONDATA-3406
• CARBONDATA-3408_carbonSessionPartitionSupportBinary
• CARBONDATA-3410_supportUDFSQLFunctionForBinary
• CARBONDATA-3415
• CARBONDATA-3423_validateDictionary
• CARBONDATA-3424_properExceptionForAVGSubstrBinary
• CARBONDATA-3441
• CARBONDATA-3443
• CARBONDATA-3455
• CARBONDATA-3461_supportFilterSet
• CARBONDATA-3477
• CARBONDATA-3483
• CARBONDATA-3485
• CARBONDATA-3488
• CarbonDataCARBONDATA-3446_supportReadComplexSchema
• CarbonPropertiesLog
• CompactionCompatibilityFix
• ComplexCompactionIssue
• ComplexDelimiters
• ConfigDoc
• DTS
• DataMismatchFix
• DataTypeSerlznIssue
• Data_not_supported
• Datasize0Issue
• DescFormRangeCol
• DocChangeForRangeCol
• DocOfComplexDelim
• DocUpdate
• FloatInfiniteFix
• GlobalDictForMap
• GroupColToProjection
• HOTFIX-20190505
• HOTFIX-20190506
• HiveRead
• INClauseIssue
• ISPerf
• JITBasedOPtimization
• JVMCrashForLoadAndQuery
• LRU
• ListenersRDD
• MVAllColSupport
• MVCoalesce
• MVCoalesceGroupLess
• MVFiltersAlias
• MVRewriteBug
• MVSpark2.1
• MVStackException
• MVTwoJoinTabs
• MapDDL5Dec
• MapDDLSupport16Oct
• MapDataLoadFailure
• MapImplIssue
• NPE_fix
• NoSuchTable
• NullPointerInRange
• PreAgg
• PrestoVarchar
• ProceededRowCount
• RangeColNewRDDImpl
• RangeColProp
• RangeFilterWithBetween
• RangeGCIssue
• RangeSortCoresModify
• RebieKong-README-UPDATE-2
• Refactor_ColumnPageWrapper
• RefreshAndCacheColumnsFix
• SDVBooleanTestCases
• SortScopeCompatibilityIssue
• StrictCountStar
• SyncInit
• Test
• TestCaseFix
• Threadpool
• WangQingNa
• adaptive_encoding_fix
• addDebugInfo
• add_dynamic_config_load_cores
• addcolumn
• aftermycommit
• alluxio-support
• alluxio_fs
• alter
• alter_doc
• alter_rename
• alter_table_set_sort_scope
• alterfix
• alterfixnoSort
• auditlog
• avro_support_sdk
• avro_support_sdk1
• b010
• basic_improvements
• batchSize_fix
• batch_sizeFix
• binary
• block_stream_mv
• blocklet_size
• bloom-jvm-crash
• bloom_bits_access
• bloom_getLiteralString
• bloom_searchmode
• bloom_shard_op
• broadcast_issue
• bug/CARBONDATA-3123
• bug/CARBONDATA-3134
• bug/CARBONDATA-3147
• bug/CARBONDATA-3300
• bug/CARBONDATA-3332
• bug/CARBONDATA-3334
• bug/CARBONDATA-3341
• bug/CARBONDATA-3460
• bugfix/CARBONDATA-2132
• build_check
• build_improv
• byte_sortcol_support
• byteutil
• cache_commands_refactoring
• cache_fix
• cache_fix2
• cache_fix3
• cache_issue
• carboncli-NPE
• carbondata-3209
• carbondata-3401
• carbondata-3422
• cast_issue
• cdh-support
• change_loglevel
• citest
• cleanfiles
• cleanfiles_mergeindex
• cli-test-fix
• cli-wrong-last-col
• cli1.7
• cli_fix
• codegen
• column_comp
• columnmeta
• comp
• comp_datamap
• compaction_fix
• compatability_checks
• compatibility_fix
• compatibility_fix_shivam
• compatibilty_fix
• complex-issue
• complex_bue
• complex_compaction
• complex_issue
• complexsdv
• compressor_desc_formatted_fix
• compresssize
• concurrent
• concurrent_datamap_issue
• correct-carbon-table-creation
• count
• count_single
• count_start_optimization
• coverity_check_temp
• create_table_no_column
• createtablefix
• csdk_test
• customfssupport
• dag_less_text
• data_mismatch_decimal
• dataloadfailure
• datamap-choose-fix
• datamaploaderfix
• datasourceAddColumn
• date
• decimal_fix
• delete_issue
• deletetemp
• desc
• desc_issue
• dev
• disable_inverted_index
• distribute
• distributed_index_server
• dmChooser
• dmRebuild
• dm_showQuery
• doc
• doc_avro
• doc_fix
• doc_sept
• doc_update
• drop_cache_command
• drop_col
• drop_table_slow
• dropcache
• dropissue
• droppreagg
• droppreaggsimple
• eat_exception
• ef_index_dm_framework
• ef_index_dm_framework_query
• ef_index_dm_minmax
• enable-local-dictionary
• ensureArray
• error
• error_exception
• error_spelling
• exception
• exceptionFix
• executor_id_distribution
• explain_block_level
• explain_block_skip
• explain_countstar
• explain_fix
• explain_profiler_disable_default
• external-format-ddl
• fallback
• fileformat-back-compat
• fileformat-date-issue
• fileformat-doc
• fileformat-test
• filter-push-doc
• filterfix
• fix
• fix-between-query
• fix-code-style
• fix-json
• fixQueryPerf
• fix_PR3001_issue
• fix_blockletsize
• fix_carbonfile_impl
• fix_external_table
• fix_load_min
• fix_load_min_size_bug
• fix_map_type_issues
• fix_multi_bloom
• fix_package_paths
• fix_streaming_issue
• fixcitest2.3
• fixsdv
• float
• float_byte_bug
• float_support
• formattingFix_S3
• global_sort
• hdfs_quota_fix
• hdfs_quota_fix3
• hdfs_quota_fix_improv
• hive-exec-depend-remove
• hive-stats-issue
• horizontal
• hotfix
• hotfix-1
• hotfix-20190515
• hotfix-fix-error-logger
• hotfix-fix-error-path
• hotfix-fix-none-default-value
• hotfix-optimize-carbondatathriftserver
• hotfix-optimize-module-name
• hotfix_change_doc
• hotfix_compile_pr2705
• hotfix_compile_pr2749
• hotfix_remove_unuse_servlet
• idcarbon
• implicit_column_filter_serialization
• improve
• improve-index-read
• index
• indexServerTest
• index_server_doc
• index_server_legacy_distribution
• index_server_update_fix
• inferschema
• insert-crash-issue
• integrate1
• integrate2
• invertedindex
• is_test
• issue_fix
• issues
• java_version_18
• javacarbonsession
• join_expression
• json
• jvmcrash
• kafka
• killapp
• latestb05
• like
• like_mv
• load_filepath
• load_sort_scope
• load_sort_scope_fix
• local
• local-1.5
• local-master
• local_dict_sdv
• log
• log4j
• log_issue
• logissuefix
• lru
• lru_cache_NPE
• lru_cache_bug
• lru_index_server_constraint
• lsc_preagg
• lucene_doc
• lz4
• make_doc_example_simple
• map_spark_carbon_file_support
• mas_mig_spark2.3_carbon_latest
• master
• master_102019
• master_17
• master_22
• master_doc
• master_explin
• master_new
• master_optimize
• memory_leak_driver
• mergeBloomIndexEvent
• merge_bloomindex
• merge_index_failure
• mergesmallfilesfix
• metalock_deadlock
• metrics-fix
• min
• minmax
• minmax_oldstore
• modifiedMDT_fix
• move-mv-to-assembly
• mv-count-fix
• mvPlan
• mv_2.1
• mv_assembly
• mv_auto_compact
• mv_char
• mv_col_literal
• mv_complex
• mv_conc
• mv_doc
• mv_incrementload
• mv_invert
• mv_issues
• mv_part_show
• mv_samecol
• mv_select
• mv_stable
• mvbugs
• mvexample
• negative_index_exception
• nested
• new_b
• no_sort_issue
• nonsense_warn_messages
• nontransactionalTableFix
• nosort_compaction
• nosort_compaction_imporv
• nosort_getpartition
• npe_fix
• numcores
• oom_fix
• optimize
• optimize-quick-start-guide-documentation
• optimize_cleanfiles
• page_compaction
• parameter_comb
• part
• partition
• partition-fileformat-issues
• partition-slow-issue
• partition_alter_bugfix
• patch-1
• patch-16
• patch-17
• patch-8
• patch-9
• perf-config-prop
• perf-decompress-optimize1
• perf-filter-scan1
• perf-full-scan
• perf-inverted-index
• perf-lazy-load
• perf-poc-2
• pertest
• planfix
• pr
• pr/2642/head
• pr3209
• pr3209-test
• preagg_query_fix
• preferredlocationfix
• presto
• presto-coverage
• presto-ditionary-fix
• presto-documentation
• presto-hive-metastore
• presto-lazy-load
• presto-s3-update
• presto-test
• presto-update-doc
• presto_210
• presto_streaming
• prestosql_migration
• project_cols
• python_support
• query_local
• query_model
• query_statistics_issue
• rand
• range_column_bug1
• range_column_doc
• range_column_fix_kyro
• range_column_table_level
• range_data_skew
• range_sort
• rangepartitionbug
• reader_fix
• reader_mem_leak
• reader_perf_improvement
• rebuild_blockletid
• reduce-blocklet-min-size
• refactorPreaggregateUT-1
• refector
• remove-scala-dep
• remove_btree_related
• remove_mdk
• remove_search
• remove_tblproperty
• remove_unused
• remove_warning
• rename_compaction
• rewrite_documents
• runzhliu
• s3
• s3-metrics
• s3Fix
• scan
• scan-test
• scan_comp
• sdk
• sdk-multi-folder-support
• sdk_doc
• sdk_local_dic_default
• sdk_map_support
• sdk_multi_user
• sdk_reader
• sdk_sort
• sdvSegmentReading
• segmentwrapperfix
• session
• set_stream_property
• shivam1
• showCacheDisplayFix
• show_cache_command_fix
• show_metacache_output_change
• show_segments_sort_columns
• show_size
• showcache
• sk_ak_fix
• snappyoffheapfailure
• sort_columns_alter_doc
• sort_columns_alter_table
• sort_columns_compaction
• sort_columns_compaction_issue
• sort_columns_enhance_custom_compaction
• sorttempDefault
• spark-fileformat
• spark2.3.2
• stale_carbon_table
• stats-issue
• stats-refresh-new
• storeimprovement
• stream_minmax
• streamingSdvTest
• streaming_ci_issue
• streaming_delete_lock
• streaming_index_null
• streaming_issue
• streaming_lock
• streaming_measure_datatype
• streaming_minmax_log
• streamsql_doc
• string_min_max_decision
• summary
• support_gen_cpp_code
• tabledetail
• task/CARBONDATA-3337
• test
• test_case_fix
• test_fix
• testcase-remove
• testcasefix
• testload
• testmaster
• timestamp_di
• unify_bad_record_path_prop
• unmanaged_table
• unsafememoryissue
• unsafesortstoragefix
• upadte_mv
• update_data_mismatch_fix
• update_doc_1.5.0
• update_doc_links
• update_documentation
• update_fail_dynamic_exec_static_val
• upgrade_1.6
• uuid-task-change
• varchar_fix
• viewfs
• wangqingna01
• wip_upgrade_to_spark2.3.2
• working_backup
• write_is_sort_in_footer
• written
• wrongcount
• xt-spell-error
• zcarbon

[CARBONDATA-3090][Integration] optimizing the columnar vector filling flow by removing unwanted filter check.

## What changes were proposed in this pull request?
optimizing the columnar vector filling flow by removing unwanted filter check, currently while filling the column vectors as part of vector reading flow in the carbon query,  while processing the batch of rows there is a filter condition check which will validate each row whether its part of filter, since the batch already has filtered rows while filling the columnvector , the filter check is not useful, and even in the current flow the boolean filteredRows[] is never been set, always the values will be false.
As part of the optimization this redundant validation of filter check is been removed.

## How was this patch tested?
Existing Unit test.