supabase / etl

75%

main: 74%

LAST BUILD ON BRANCH rework-table-copy

branch: rework-table-copy

CHANGE BRANCH

x

Reset

• rework-table-copy
• aalexandrov/local_dev_commands
• abhi/ensure-pg-14-support
• add-backtrace-staging
• add-bytes-metric
• add-cargo-sort
• add-clickhouse-timeouts
• add-ddl-deletion
• add-ddl-metrics
• add-destination-metric
• add-destination-tag-sentry
• add-endpoint
• add-flag
• add-force
• add-ipv6-support
• add-label
• add-latency-endpoint
• add-log-for-not-copying
• add-metrics
• add-new-lag-metrics
• add-new-tag
• add-oriole
• add-publication-endpoints
• add-read-replica-support
• add-sentry-errors
• add-shutdown-method
• add-support-for-multiple-encryption
• add-test
• add-test-metric
• add-topology-constraint
• arm
• artyom/co-change
• asynchronous-progress-tracking
• attempt-fix-flaky
• attempt-fix-lost-data
• attempt-size-batching
• attempt-solve-tests-failures
• attempting-sqlx-bump
• backoff-bq
• benjamincoenen/etl-607-remove-logs
• benjamincoenen/etl-690-ducklake-pause-trusts-unbounded-cr-status-expiry
• benjamincoenen/etl-789-build-the-supabase-postgres-fdw-bridge-for-supabase
• benjamincoenen/etl-832-add-validation-to-check-if-schema-and-tables-for-ducklake-in
• benjamincoenen/etl-850-use-source-schemas-in-ducklake-and-add-schema-aware
• benjamincoenen/etl-869-do-not-return-secrets-from-the-etl-api
• better-defaults
• better-timer
• bnjjj/add_inlined_data_bytes_metrics
• bnjjj/add_pause_metrics
• bnjjj/chaos_testing_xtask
• bnjjj/clean_up_all
• bnjjj/ducklake
• bnjjj/ducklake_better_cleanup
• bnjjj/ducklake_catalog_maintenance
• bnjjj/ducklake_cleanup_ddl
• bnjjj/ducklake_ddl
• bnjjj/ducklake_ext_maintenance_metrics
• bnjjj/ducklake_fixes
• bnjjj/ducklake_looogs
• bnjjj/ducklake_optim
• bnjjj/enable_expire_snapshot_duck
• bnjjj/etl_api_replicator_resource
• bnjjj/ext_maintenance_generic
• bnjjj/external_maintenances
• bnjjj/fix_573
• bnjjj/fix_622
• bnjjj/fix_670
• bnjjj/fix_ducklake_maintenance_runner_libpq
• bnjjj/fix_libpq_to_host_ipv6
• bnjjj/improve_ducklake_maintenances
• bnjjj/improve_ducklake_shutdown
• bnjjj/improve_expire_snapshot_maintenance
• bnjjj/inline_flush_maintenance
• bnjjj/migrate_actix
• bnjjj/more_context_on_interrupts
• bnjjj/move_metrics_connections_to_pg
• bnjjj/pipeline_id_metrics_label
• bnjjj/prod_defaults_ducklake
• bnjjj/remove_batch_flush
• bnjjj/remove_useless_clone
• bnjjj/simplify_fix_ducklake_flush_maintenance
• bnjjj/sync_rust_toolchain_and_msrv
• bnjjj/upgrade_duckdb
• bnjjj/warning_reuse_ducklake_schema
• bump-bq-client
• bump-vector
• change-concurrency-bq
• change-default
• change-default-memory
• change-limit
• chore/match_same_arms
• chore/unused-deps
• chore/update-github-actions-node24
• ci-fix
• ci-improve
• cleanuo
• cleanup-a-few-things
• cleanup-deps
• cleanup-releases
• codex/change-specs
• codex/implement-fail-rs-for-fault-injection
• codex/json-number-precision
• concurrency
• connection-retry
• deps/bump-rust-sqlx
• etienne/sec-666-pin-all-github-actions-to-full-commit-sha
• experimental-iceberg-impl
• feat/api-destination-features
• feat/etl-643-snowflake-ci
• feat/etl-identity-labels
• feat/extend-xtask-cmds
• feat/fork-ci-protection
• feat/gated-examples
• feat/realtime-destination
• feat/replicator-destination-features
• feat/snowflake
• feat/snowflake-api-config
• feat/snowflake-benchmarks
• feat/snowflake-k8s
• feat/snowflake-validator
• feat/test-snowflake
• feat/xtask-destination-presets
• figure-out-memory-leak
• fix
• fix-advisory
• fix-alerts
• fix-bq
• fix-bq-order
• fix-bq-retries
• fix-ci
• fix-ci-docker
• fix-ci-not-working
• fix-connection
• fix-connection-error
• fix-core
• fix-ddl
• fix-deduped
• fix-deletion-behavior
• fix-depedency
• fix-docker
• fix-docker-ci
• fix-docker-image
• fix-docs
• fix-docs-design
• fix-ducklake
• fix-endpoint
• fix-endpoint-error
• fix-error
• fix-errors-returned
• fix-etl-telemetry-error-message-casing
• fix-flaky
• fix-lag
• fix-log-level
• fix-many
• fix-null-parse
• fix-only
• fix-permission-denied
• fix-pipeline-update
• fix-query
• fix-quoting
• fix-script
• fix-security
• fix-security-advisory
• fix-security-warning
• fix-sentry
• fix-slow
• fix-stall
• fix-stalling
• fix-stalling-test
• fix-startup
• fix-state
• fix-stuck
• fix-table-mapping
• fix-tasks
• fix-test-flaky
• fix-tests
• fix-ui-docs
• fix-url
• fix/add-cargo-lock
• fix/allow-empty-publication
• fix/bigquery-test-skip
• fix/clickhouse-native-port-9001
• fix/fork-pr-notice
• fix/graceful-k8s-failure
• fix/image-update-missing-target
• fix/pipeline-api-error-mapping
• fix/quote-xtask-identifiers
• fix/replicator-rustflags
• fix/schema-masks-restart-test-flake
• fix/security-audit
• fix/snowflake-default-literals
• fix/ssrf-id-validation
• fix/vendored-swagger-ui
• general-refactor
• implement-better-cleanup
• implement-better-shutdown
• implement-paralle-table-copy
• improve-allocations
• improve-backpressure
• improve-backpressure-aggressiveness
• improve-backpressure-calculation
• improve-benchmarking
• improve-billing
• improve-bq-connection
• improve-bq-errors
• improve-bq-files
• improve-bq-handling
• improve-bq-handling-not-found
• improve-catchup
• improve-ci-bench
• improve-cleanaup
• improve-code
• improve-comments
• improve-concurrency
• improve-config
• improve-connection-handling
• improve-ddl
• improve-ddl-defaults
• improve-ddl-handling
• improve-defaults
• improve-docs
• improve-docs-ddl
• improve-docs-mobile
• improve-durable-behavior
• improve-error-handling
• improve-error-messages
• improve-idle-timeouts
• improve-ipv6
• improve-lagh
• improve-logging-streams
• improve-metric-tag
• improve-metrics-code
• improve-name
• improve-notify-handling
• improve-ordering
• improve-panioc
• improve-pub-docs
• improve-public-docs
• improve-query-performance
• improve-replica-identity
• improve-reset
• improve-script
• improve-script-xtask
• improve-script-xtask-test
• improve-scripts
• improve-sequencing
• improve-shutdown
• improve-source-error
• improve-tag
• improve-tests
• improve-truncation
• improve-validation
• improve-validation-slot
• improve-vector-size
• improve-verifcation
• jm/avoid-deser-clones
• jm/ci-security-hardening
• jm/clickhouse
• jm/clickhouse-engine-shared-variant
• jm/clickhouse-large-row-tests-v2
• jm/clickhouse-loadtest
• jm/clickhouse-metrics
• jm/clickhouse-replacing-merge-tree
• jm/clickhouse-validation-hygiene
• jm/clippy-lints
• jm/consolidate-integration-tests
• jm/ducklake-test-fixes
• jm/etl-core
• jm/events-count-wait
• jm/fuzz-bytea-panic
• jm/nextest
• jm/nix
• jm/notify-before-start
• jm/retest-flaky-under-load
• jm/retries-flaky-test-setup
• jm/sql-like-slots
• jm/when-then-comment-clarifications
• kafaichoi/ensure-unique-per-source-and-destination-pair
• main
• make-logs-less-verbose
• make-sentry-more-secure
• mh/ecr-action
• migrations
• move-arc-clone
• move-crates
• move-docs
• move-to-multi-encryption
• new-label
• new-metric
• new-metrics
• new-reset-endpoint
• new-test-metric
• or/iceberg
• partial-configs
• perf/xtask-build-isolation
• raminder/etl-148-process_syncing_tables-in-apply-loop-should-be-called-on
• raminder/etl-150-new-table-sync-workers-keep-launching
• ref/destination-kind
• ref/fmt-xtask
• ref/improve-minor-things
• ref/remove-end-batch
• ref/snowflake-auth-pem
• ref/snowflake-example
• ref/split-validator-tests
• ref/split-validators
• refactor/clippy-lints
• refactor/group-destination-tests
• refactor/stores-cloning
• refactor/test-db-default-values
• refine-behavior
• release/v0.1.1
• remove-arc-table-descriptor
• remove-errored
• remove-pipeline-id
• remove-rollback
• remove-taint
• remove-waiting
• reproduce-failure
• response-time-slo
• rework-api
• rework-apply-loop
• rework-etl-core
• riccardo/add-endpoint
• riccardo/feat/add-replication-client
• riccardo/feat/add-taint
• riccardo/feat/ddl-support
• riccardo/feat/ddl-support-2
• riccardo/feat/ddl-support-3
• riccardo/feat/implement-apply-loop
• riccardo/feat/workers-structure
• riccardo/fix-data-conversion
• riccardo/fix/deadlock
• riccardo/fix/do-not-remove-mappings
• riccardo/fix/update-flush-lsn
• riccardo/pr-503-error-handling-only
• riccardobusetti/etl-100-allow-multiple-api-keys-from-use-by-different-clients
• riccardobusetti/etl-104-handle-values-outside-the-range-accepted-by-bigquery
• riccardobusetti/etl-129-set-sane-options-when-connecting-to-postgres
• riccardobusetti/etl-141-investigate-the-possibility-to-reduce-the-number-of-error
• riccardobusetti/etl-146-properly-handle-arraycell-option-values
• riccardobusetti/etl-15-rename-pgdatabaseoptions-to-pgdatabaseconfig-for-sqlx
• riccardobusetti/etl-156-find-a-solution-for-inability-to-truncate-bigquery-table
• riccardobusetti/etl-167-implement-cascade-delete-of-postgres-replication-state
• riccardobusetti/etl-17-create-new-types-for-oid
• riccardobusetti/etl-174-add-tests-for-conversions
• riccardobusetti/etl-19-implement-mechanism-to-cleanup-unused-slots-and-origins
• riccardobusetti/etl-207-the-pipelineshutdown_and_wait-function-sometimes-returns
• riccardobusetti/etl-210-extend-the-state-stored-in-postgres
• riccardobusetti/etl-212-handle-failed-table-sync-workers
• riccardobusetti/etl-214-simplify-how-a-failure-of-the-pipeline-is-exposed-to-the
• riccardobusetti/etl-215-its-neighborly-to-re-export-types-used-in-your-public-api
• riccardobusetti/etl-218-cleanup-unused-dependencies
• riccardobusetti/etl-225-move-table-schemas-into-the-state-store
• riccardobusetti/etl-232-store-table-creation-state-in-bigquery-destination
• riccardobusetti/etl-234-write-from-scratch-the-initial-migrations-before-the-alpha
• riccardobusetti/etl-236-add-support-for-more-types-in-the-table-schema-of-the
• riccardobusetti/etl-240-cleanup-replication-slots-on-pipeline-deletion
• riccardobusetti/etl-252-handle-row-errors-in-fallback-during-streaming-in-bigquery
• riccardobusetti/etl-254-implement-mechanism-for-defaults
• riccardobusetti/etl-257-use-gcp-bigquery-client-instead-of-our-fork
• riccardobusetti/etl-259-force-restart-pipeline-after-starting-it
• riccardobusetti/etl-262-handle-truncate-event-when-table-was-not-loaded
• riccardobusetti/etl-265-handle-state-reset-properly
• riccardobusetti/etl-266-handle-table-schema-and-table-mappings-override
• riccardobusetti/etl-268-partitioned-tables-do-not-work-directly-due-to-lack-of-pk
• riccardobusetti/etl-272-detect-tables-being-removed
• riccardobusetti/etl-273-allow-filtering-logs-by-pipeline-id
• riccardobusetti/etl-274-avoid-buildup-of-the-replication-states
• riccardobusetti/etl-277-implement-the-backend-endpoints-for-returning-image
• riccardobusetti/etl-283-figure-out-test-failure
• riccardobusetti/etl-285-implement-mechanism-to-version-etl
• riccardobusetti/etl-287-implement-batch-timeout-with-timer
• riccardobusetti/etl-288-implement-mechanism-to-defer-shutdown-when-in-a-transaction
• riccardobusetti/etl-289-add-taints-to-etl-resources
• riccardobusetti/etl-290-implement-image-update-endpoint
• riccardobusetti/etl-309-add-support-for-renaming-publication
• riccardobusetti/etl-319-change-backoff-of-k8s
• riccardobusetti/etl-326-improve-error-messages
• riccardobusetti/etl-336-sample-verbose-endpoints-at-a-smaller-rate
• riccardobusetti/etl-352-add-support-for-publication-with-subset-of-columns
• riccardobusetti/etl-368-allow-creating-destination-tables-when-source-has-no-data
• riccardobusetti/etl-413-add-error-notification-for-errors-in-table-sync-workers
• riccardobusetti/etl-416-postgresql-18-support
• riccardobusetti/etl-433-validate-correctness-before-starting-etl
• riccardobusetti/etl-467-for-the-postgres-connection-to-the-main-api-database-do-not
• riccardobusetti/etl-468-implement-validation-endpoint
• riccardobusetti/etl-473-skip-to-initial-copy-data
• riccardobusetti/etl-482-inconsistencies-in-tests-between-a-fork-and-upstream
• riccardobusetti/etl-494-implement-graceful-handling-of-invalidated-slot
• riccardobusetti/etl-514-create-metrics-for-the-tokio-runtime-and-expose-them
• riccardobusetti/etl-525-implement-memory-aware-processing
• riccardobusetti/etl-56-ensure-theres-always-exactly-one-default-image
• riccardobusetti/etl-563-add-oriole-to-etl-ci-matrix-testing
• riccardobusetti/etl-596-investigate-missing-keep-alives-during-shutdown
• riccardobusetti/etl-61-handle-schema-changes-in-the-source
• riccardobusetti/etl-636-recreate-bigquery-view-when-schema-is-updated
• riccardobusetti/etl-66-ensure-unique-table-names-in-destinations
• riccardobusetti/feat/add-agents-file
• riccardobusetti/feat/add-configcat
• riccardobusetti/feat/add-diagram
• riccardobusetti/feat/add-error-notifications
• riccardobusetti/feat/add-function
• riccardobusetti/feat/add-labels
• riccardobusetti/feat/add-memory-profiling-script
• riccardobusetti/feat/add-more-data-to-logs
• riccardobusetti/feat/add-more-metrics
• riccardobusetti/feat/add-rust-docs
• riccardobusetti/feat/add-sentry-replicator
• riccardobusetti/feat/add-trusted
• riccardobusetti/feat/add-update-endpoint
• riccardobusetti/feat/arm
• riccardobusetti/feat/better-resource-provisioning
• riccardobusetti/feat/default-to-manual
• riccardobusetti/feat/do-not-return-password
• riccardobusetti/feat/emit-project-logs
• riccardobusetti/feat/etl-docs
• riccardobusetti/feat/expose-errors
• riccardobusetti/feat/expose-lag
• riccardobusetti/feat/fail-rs-setup
• riccardobusetti/feat/implement-state-for-tables
• riccardobusetti/feat/improve-application-name
• riccardobusetti/feat/improve-big-query
• riccardobusetti/feat/improve-bigquery
• riccardobusetti/feat/improve-deletio
• riccardobusetti/feat/improve-endpoints
• riccardobusetti/feat/improve-errors
• riccardobusetti/feat/improve-exports
• riccardobusetti/feat/improve-github
• riccardobusetti/feat/improve-k8s
• riccardobusetti/feat/improve-logs
• riccardobusetti/feat/improve-pipeline
• riccardobusetti/feat/improve-readme
• riccardobusetti/feat/improve-refresh
• riccardobusetti/feat/improve-rx
• riccardobusetti/feat/improve-tests
• riccardobusetti/feat/improve-vector-flushing
• riccardobusetti/feat/improve-version-handling
• riccardobusetti/feat/integrate-sentry
• riccardobusetti/feat/limit-pipelines
• riccardobusetti/feat/log-error
• riccardobusetti/feat/new-endpoint
• riccardobusetti/feat/new-endpooint
• riccardobusetti/feat/new-logo
• riccardobusetti/feat/pre-stop-hook
• riccardobusetti/feat/publish-rust-docs
• riccardobusetti/feat/remove-bg-emulator
• riccardobusetti/feat/remove-context-path
• riccardobusetti/feat/return-key
• riccardobusetti/feat/return-table-id
• riccardobusetti/feat/rework-logs
• riccardobusetti/feat/status-endpoint
• riccardobusetti/feat/swap-image
• riccardobusetti/feat/update-dependencies
• riccardobusetti/feat/update-docs
• riccardobusetti/feat/update-migration
• riccardobusetti/feat/use-mutex
• riccardobusetti/feat/use-secret
• riccardobusetti/feat/use-sentry-secret
• riccardobusetti/feat/write-docs-mkdocs
• riccardobusetti/fix/bigquery-conversion
• riccardobusetti/fix/ci
• riccardobusetti/fix/config
• riccardobusetti/fix/docker-build
• riccardobusetti/fix/formating
• riccardobusetti/fix/improve-ci
• riccardobusetti/fix/improve-output
• riccardobusetti/fix/optional-skip
• riccardobusetti/fix/panics
• riccardobusetti/fix/run-action
• riccardobusetti/fix/send-more-keep-alive
• riccardobusetti/fix/sentry-setup
• riccardobusetti/fix/shutdown-signal
• riccardobusetti/fix/tests-stuck
• riccardobusetti/fix/unwrap
• riccardobusetti/fix/workflows
• riccardobusetti/improve
• riccardobusetti/inject-version-env-variable
• riccardobusetti/psql-551-fill-schema-cache-from-destination
• riccardobusetti/psql-553-implement-bigquery-destination-v2
• riccardobusetti/psql-557-hook-the-v2-pipeline-in-the-replicator
• riccardobusetti/ref/change-application-name
• riccardobusetti/ref/change-defaults
• riccardobusetti/ref/change-endpoint
• riccardobusetti/ref/improve
• riccardobusetti/ref/improve-actions
• riccardobusetti/ref/improve-exports
• riccardobusetti/ref/improve-messages
• riccardobusetti/ref/increase-timeout
• riccardobusetti/ref/move-notify-state
• riccardobusetti/ref/remove-readme-section
• riccardobusetti/ref/rename-crates
• riccardobusetti/ref/simplify-schema
• rs/abort-on-panic
• rs/add-array-support
• rs/add-k8s-json-tests
• rs/add-metrics-port
• rs/add-missing-project-ref-in-logs
• rs/add-pipeline-id-to-metrics
• rs/add-postgres-team-as-codeowners
• rs/add-rest-catalog-params
• rs/add-scalar-support
• rs/add-stopping-status
• rs/add-update-config-endpoint
• rs/allow-enabling-debug-logs
• rs/api-metrics
• rs/benchmarking
• rs/build-mac-image
• rs/bump-editions
• rs/bump-iceberg-rust
• rs/bump-local-image
• rs/bump-max-replication-slots
• rs/clippy-fixes
• rs/clippy-no-deps
• rs/db-sinks-to-destinations
• rs/debug-stuck-tests
• rs/docs
• rs/egress-metric
• rs/emit-iceberg-egress-metrics
• rs/encrypt-s3-access-key-id
• rs/expect-clippy
• rs/feat/improve-logs
• rs/fix-abort-on-panic
• rs/fix-api-binary-name
• rs/fix-app-env
• rs/fix-assets
• rs/fix-batch-sent-metric
• rs/fix-clippy-errors
• rs/fix-compiler-error
• rs/fix-docker-build
• rs/fix-failing-test
• rs/fix-links-in-readme
• rs/fix-panic
• rs/fix-secrets-serialization
• rs/fix-sentry-config
• rs/fix-test-tracing
• rs/fix-tests
• rs/fix-wrong-null-values-encoding
• rs/fix/sync-done-bug
• rs/fix/update-config
• rs/gh-disk-space
• rs/iceberg-client
• rs/iceberg-destination
• rs/iceberg_scripts
• rs/ignore-stuck-tests
• rs/improve-config
• rs/improve-connections
• rs/improve-etl-error-macro
• rs/improve-logging
• rs/improve-metrics
• rs/improve-replication-client
• rs/init-crypto-provider
• rs/limit-max-connections
• rs/listen-on-all-interfaces
• rs/make-tests-run-in-parallel-with-clippy-and-fmt
• rs/mock-k8s-client
• rs/no-primary-key
• rs/one-ns-for-one-schema
• rs/reduce-logging-in-api
• rs/remove-array-null
• rs/remove-duplicate-code
• rs/remove-duplicate-config
• rs/remove-expect
• rs/remove-old-advisories
• rs/remove-old-code
• rs/remove-rustls-pemfile
• rs/remove-unknown-types-to-bytes-feature
• rs/remove-unused-method
• rs/remove_double_underscores
• rs/rename-namespaces
• rs/rename-pg_replicate-to-supabase_etl
• rs/rename-supabase_etl-to-etl
• rs/renames-and-other-cleanup
• rs/replicator-metrics
• rs/return-conflict-error
• rs/run-iceberg-replicator
• rs/show-all-tables
• rs/sinks-to-destinations
• rs/state-store
• rs/state-store-new
• rs/tls-config
• rs/toast
• rs/tracing-in-tests
• rs/update-api-for-iceberg-config
• rs/update-app-labels
• rs/update-benchmark-scripts
• rs/update-codeowners
• rs/use-blacksmith-docker
• rs/use-blacksmith-runners
• rs/use-consts
• rs/use-orbstack-for-dev
• scale
• scripts/improve-k8s-check
• sentry-debug-symbols
• swap-notification-endpoint
• table-id-integer
• test/improve-streaming-test
• tls-in-tests
• trail-whitespaces
• update-allocator
• update-docker
• update-params
• update-runner
• upload-source-code-coveralls
• use-new-version
• vividata-pr/serialise-first-table-creation

Build # 28444591316

Build Type

Pull #861

github

Committed by web-flow

Commit Message

Merge 1da6e24c2 into 72df905ce

Pull Request Pull Request #861: ref(core): Rework table copy parallelization algorithm

Coverage Stats

437 of 494 new or added lines in 8 files covered. (88.46%)

6 existing lines in 4 files now uncovered.

39565 of 52936 relevant lines covered (74.74%)

35229.52 hits per line