Logflare / logflare

71%

main: 71%

LAST BUILD ON BRANCH adammokan/anl-1180-more-robust-clickhouse-ingest

branch: adammokan/anl-1180-more-robust-clickhouse-ingest

CHANGE BRANCH

x

Reset

• adammokan/anl-1180-more-robust-clickhouse-ingest
• abhi-improve-metadata-parsing
• abhi-serialize-instead-of-inspect
• adammokan/anl-1137-logflare-needs-to-add-support-for-fsandboxed_table_or_source
• adammokan/anl-1152-bug-alerts-route-raising-500-error
• adammokan/anl-1169-fix-clickhouse-connection-pool-wake-behavior
• adammokan/anl-1175-modify-ch-ingest-table-schema
• adammokan/anl-1175-modify-ch-ingest-table-schema-ii
• adammokan/anl-1216-add-logging-and-lock-in-settings-for-clickhouse-ingestion
• adammokan/anl-1217-configure-retries-for-clickhouse-ingest-pipeline
• adammokan/anl-1219-use-tesla-client-and-tesla-retry-middleware-for-clickhouse
• adammokan/anl-1221-update-clickhouse-docs-with-ingest-flags
• adammokan/anl-1231-crash-dump-when-postgres_backend_url-contains-special
• adammokan/anl-1233-chore-follow-up-adjustments-for-tesla-client-in-clickhouse
• adammokan/anl-1234-chore-normalize-clickhouse-casing-for-consistency-in-logging
• adammokan/anl-1235-change-clickhouse-ingest-table-naming-convention
• adammokan/anl-1236-bug-fix-uuid-encoding-in-clickhouse-ingestor
• adammokan/anl-1237-chore-adjust-clickhouse-connection-timeouts-and-retries
• adammokan/anl-1243-filter-old-events-before-inserting-into-clickhouse
• adammokan/anl-1244-adjust-clickhouse-pipeline-batch-settings
• adammokan/anl-1245-tighten-up-clickhouse-logevent-iterations-when-checking
• adammokan/anl-1248-shorten-min-interval-for-alerts-from-15-5m
• adammokan/anl-1249-fix-bug-with-transform_config1-callback-in-s3adaptor
• add-clickhouse-docs
• amokan-add-json-error-handlers
• amokan-ch-retries-follow-up
• amokan-clickhouse-query-fiesta
• amokan-clickhouse-sandbox-queries-follow-up
• amokan-clickhouse-sandboxed-queries
• amokan-fix-view-link-on-search-results
• amokan-increase-ch-conn-timeout
• amokan-lql-no-select
• amokan-lql-sandbox-queries
• amokan-sql-parser-ast-fix
• amokan-sql-parser-ast-fix-part-ii
• amokan/ch-wait-for-async-insert
• amokan/clickhouse-pipeline-ack-issue
• amokan/fix-ci-issue
• amokan/more-clickhouse-ingest-pipeline-logging
• amokan/more-clickhouse-logging-for-retries
• amokan/otel-exporter-batch-size-config
• amokan/postgres-lql
• amokan/remove-id-from-bq-cluster-keys
• amokan/schema-unlock-fix
• avoid-atom-conversions-during-query-building
• backpressure-issue
• bb/adaptor-refactor
• bb/axiom-adapter
• bb/last9-adaptor
• bb/otlp-http-adaptor
• bb/rules-cache
• bq-json-type
• bug-schema-utils
• ch-connection-handling
• ch-ingest-batch-size-change
• ch-read-pool-init-bug
• ch_dynamic_table_naming
• chore/add-blacksmith
• chore/blacksmith-only-for-image-buidls
• chore/change-ets-metrics-to-memory-usage
• chore/ci-on-every-branch
• chore/comment-out-dialyxer-check
• chore/fix-flaky-test-attempt
• chore/flaky-test
• chore/flaky-test-sourcesup-shutdown
• chore/improve-dev-env
• chore/improve-search-lv-test
• chore/libcluster-pg-fixes
• chore/pr-comments-2461
• chore/reduce-rpc-flakiness
• chore/refactor-recent-logs-server
• chore/remove-debugging-statement
• chore/remove-event-context-flag
• chore/remove-gce-strategy
• chore/remove-libcluster-pg-ssl
• chore/remove-unused-logging
• chore/single-tenant-seeding-bug
• chore/typing-log-event-cache
• chore/update-carg-lockfile
• ci/use-bitnamilegacy
• clickhouse-adaptor-sql-improvements
• clickhouse-adaptor-tiny-log-removal
• clickhouse-ci-fixes
• clickhouse-connection-pools
• clickhouse-docker
• clickhouse-log-drain-phase1
• clickhouse-lql
• clickhouse-reads
• clickhouse-reads-implementation
• clickhouse-schema-changes
• clickhouse-test-setup-fix
• clickhouse_ch_client
• clickhouse_cloud_http_pool
• default-backend-require-source
• default-ingest-backend-alt
• default-ingest-improvements
• default_backend_rework
• dev-logflare-ui
• doc/fix-openapi
• docs-source-creation-guide
• docs/contributing-backends
• docs/db-ssl
• docs/endpoints-subquery-expansion
• docs/fix-markdown-heading
• docs/logpush-documentation
• docs/trace-ingestion
• docs/update-accesss-tokens-querying-docs
• dynamic-tracing
• endpoint-backend-abstraction
• endpoint-backend-abstractions
• endpoint-live-ui-test-improvement
• endpoint-query-bug
• endpoint-query-ui-updates
• etienne/admin-api-updates
• feat/account-level-default-ingestion
• feat/add-cluster-information-to-otel-metrics-exporter
• feat/adjust-chart-period
• feat/alerts-live-monaco-editor
• feat/all-metrics
• feat/backends-cache-refactoring
• feat/cache-clear-button
• feat/disable-tailing-source
• feat/erlsysmon-net-kernel-logging
• feat/event-context
• feat/export-metrics-through-otel
• feat/format-query
• feat/gen-singleton
• feat/gensgingleton-transient
• feat/improve-local-timezone-viewing
• feat/ingest-query-scopes
• feat/ingestion-metrics-system-monitoring-refactor
• feat/interlinking
• feat/lql-select-alias
• feat/modal-custom-commands
• feat/monaco_editor_for_endpoints
• feat/move-delete-source-button
• feat/open-api-improvements
• feat/open-api-spex-improvements
• feat/optional-datadog-pools
• feat/otlp-logs-ingestion
• feat/query-cost-ui-feedback
• feat/recommended-query-fields
• feat/search-page-log-levels
• feat/search-results-select-fields
• feat/st-ch-ingestion
• feat/storage-write-api-improvements
• feat/storage-write-api-improvements-v2
• feat/storage-write-api-initial-v2
• feat/storage-writer-arrow-rs
• feat/support-ap2-datadog
• feat/team-id-param
• feat/team-switcher
• feat/team_link_component
• feat/time-partition-bq-hour
• feat/top-processes-and-ets-tables-metrics
• feat/user-system-monitoring
• fix-startup
• fix/add-loki-backend-username-password
• fix/adjustments-from-last-pr
• fix/admin-accounts-list
• fix/alerting-improvements
• fix/alerts-scheduler-termination
• fix/artifact-repository-grafana
• fix/as-for-postgres-alias
• fix/avoid-creating-log-events-from-cache
• fix/backend-database-field
• fix/backends-defaults
• fix/bandit-version-bump-metrics
• fix/bq-sql-extra-quotes
• fix/broken-documentation-links
• fix/build-message-as-string
• fix/ce-role
• fix/changed-endoint-labels-clears-cache
• fix/chart-timezone
• fix/cleaner-query-lables
• fix/clickhouse-array-select-handling
• fix/clickhouse-parameters-mapping
• fix/cluster-nodename-suffix
• fix/compile-time-env-value
• fix/correctly-handle-no-home-team
• fix/credo-diff-against-main
• fix/dashboard-fields-bug
• fix/dashboard-metrics-poller
• fix/dashboard-navigation-team
• fix/datadog-http1
• fix/debug-modals-sql
• fix/delete-source-link
• fix/deps-branch-logger
• fix/duplicate-id-warnings
• fix/endpoint-query-wrap
• fix/endpoint-run-query
• fix/endpoints-form-changes
• fix/endpoints-shutdown-bug
• fix/error-response
• fix/event-modal-permalink
• fix/faster-backend-syncing
• fix/fetch-from-paritiontime-table
• fix/field-in-array
• fix/flaky-alerting-test
• fix/global-broadcast
• fix/goth-partitioning-sync
• fix/grpc-server-gzip
• fix/gzip-cached-query-results
• fix/handle-mutliple-union-all
• fix/handle-syn-resolve-kill
• fix/http1-for-datadog
• fix/legacy-artifacts-explorer
• fix/libcluster-pg
• fix/log-event-whitespace
• fix/lql-aggregates
• fix/lql-negations
• fix/managed-sa-accounts-when-disabled
• fix/metadata-queries
• fix/misc-bugs
• fix/monaco-editor-web-workers
• fix/monaco-eidtor-scrolling
• fix/monaco-font-load-errors
• fix/move-ensure-source-sup-started
• fix/named-parameter-mode
• fix/navbar-session-fetching
• fix/nil-attributes
• fix/nil-suggested-fields
• fix/node-22-jquery-fix
• fix/otel-ingestion
• fix/pii-redaction-testing
• fix/pino-logflare-link
• fix/pseudo-streaming-buffer
• fix/query-only-backends-selection
• fix/recent-events-touch
• fix/reduce-conflict-logs-emitted
• fix/reduce-node-connection-attempts
• fix/refactor-log-event-live-and-component
• fix/regression-query-error
• fix/reject-ets-info-undefined-returns
• fix/remove-compile-env
• fix/remove-error-string
• fix/remove-grpc-logger
• fix/remove-touch-team-user
• fix/resource-checks
• fix/restrict-tesla-env-structs-stringifying
• fix/revert-gzip-results
• fix/run-endpoint-query-with-labels
• fix/save-incidentio-backend-metadata-error
• fix/search-view-not-handling-user-context
• fix/set-verify-source
• fix/source-routing-without-sink
• fix/special-char-support
• fix/sql-postgres-translation
• fix/sql-translation-nested-cte-regexp-contains
• fix/ssl-ecto-repo
• fix/stale-clickhouse-config
• fix/syn-conflict-stop-proc
• fix/syn-silencing
• fix/syn-stop-supervisor
• fix/sync-rules
• fix/table-name-format
• fix/team-link-in-eex-template
• fix/team-param-everywhere
• fix/team-switching
• fix/test-ce-role
• fix/test-query-returned-error
• fix/typespecs
• fix/uncontrolled-endpoints-form
• fix/unnesting-metadata
• fix/update-hash-for-otel-exporter-dep
• fix/user-bq-reset-values
• google-namespace-refactor
• hide-backend-creds-from-view
• improving-tracing
• investigate-sql-conversion-issue
• la/api-gateway-logs
• lql-filter-rule-bug
• lql-improvements
• lql-mimic-test-mock-issues
• lql-rule-serialization-bug
• lql-select-improvement
• main
• more-test-improvements
• multi-default-backends
• naive-ch-pr-additions
• ne/activate-dialyzer
• ne/admin-controller-tests
• ne/anl-1194-connection-pools-single-mode
• ne/backed
• ne/backend-adaptors
• ne/backend-form
• ne/backends
• ne/backends-postgres
• ne/bigquery-refactor
• ne/bigquery-schema-refactor
• ne/billing
• ne/cargo
• ne/cluster-live
• ne/container-logs
• ne/coveralls
• ne/credo
• ne/credo-nesting-logflare-context_cache)
• ne/credo-nesting-logflare-logs-source_parsers-syslog_parser)
• ne/credo-nesting-logflare-logs-vercel-nimble_vercel_parser)
• ne/credo-nesting-logflare-logs-vercel-vercel_parser)
• ne/credo-nesting-logflare-lql-parser)
• ne/credo-nesting-logflare-source-text_notification_server)
• ne/credo-nesting-logflare-sql-dialect_translation)
• ne/credo-nesting-logflare-users-users)
• ne/credo-nesting-logflare_web-controllers-plugs-require_auth)
• ne/credo-nesting-logflare_web-controllers-plugs-set_team_user)
• ne/credo-nesting-logflare_web-live-vercellogdrains_live-index)
• ne/dbg
• ne/deprecated-cache-local
• ne/dialect-translation-refactor
• ne/dialyzer
• ne/display-helpers
• ne/e2e-url
• ne/ecto-atom-type
• ne/ecto-clickhouse
• ne/ecto-regex-tests
• ne/ecto-term
• ne/edit-billing-tests
• ne/elixir-logger-view
• ne/enable-typing
• ne/endpoint
• ne/endpoint-controller-api
• ne/endpoints
• ne/endpoints-resolver
• ne/errors
• ne/ets
• ne/ex-coveralls
• ne/ex-machina
• ne/execute
• ne/fix-text-notification-server
• ne/fix-typo
• ne/flaky
• ne/flaky-backends
• ne/flaky-test
• ne/flaky-tests
• ne/gen_utils
• ne/grpc-tests
• ne/integration-tests-supabase
• ne/lib/logflare/alerting
• ne/lib/logflare/backends/adaptor/postgres_adaptor/pg_repo
• ne/lib/logflare/billing
• ne/lib/logflare/context_cache
• ne/lib/logflare/logs/log_event
• ne/lib/logflare/rules
• ne/lib/logflare/source/email_notification_server
• ne/lib/logflare_web/live/backends/backends_live
• ne/lib/logflare_web/live/source_backends_live
• ne/live-rules
• ne/logflare-utils-parens
• ne/logflare/backends-parens
• ne/logflare/contexts-parens
• ne/logflare/google-parens
• ne/logflare/logs-parens
• ne/logflare/lql-parens
• ne/logflare/pubsub-parens
• ne/logflare/source-parens
• ne/logflare_web/controllers-parens
• ne/logflare_web/live-parens
• ne/logs-logflare-container
• ne/logs-search-live-refactor
• ne/logs-search-operations-nesting
• ne/logs-types
• ne/long-lines
• ne/mimic
• ne/mimic-upgrade
• ne/mix-test
• ne/nesting
• ne/open-telemetry-sampler
• ne/open_api_spex
• ne/parens
• ne/partner-live
• ne/phoenix-live-view
• ne/phoenix-test
• ne/playwright
• ne/plug-set-team-if-nil
• ne/pw-screenshots
• ne/refactor-old-templates
• ne/refactor-rpc
• ne/refactor-utils
• ne/revert-missing-catch-all
• ne/rules
• ne/rules-followup
• ne/silent_logs
• ne/sobelow
• ne/source
• ne/source-bigquery-nesting
• ne/source-controller
• ne/source-data-nesting
• ne/source-routing-nested
• ne/source-schemas
• ne/source_controller
• ne/sources-controller
• ne/sql-typos
• ne/stream-data
• ne/system-metrics-parens
• ne/team-controller-tests
• ne/team-user-controller
• ne/teams
• ne/tesla
• ne/test
• ne/test-parens
• ne/test-util
• ne/test-warnings
• ne/testing-unsubscribing
• ne/tests-clickhouse
• ne/tests-coverage-context
• ne/trys
• ne/tuple-warnings
• ne/types
• ne/types-gensingleton
• ne/types/alerting
• ne/types/backends
• ne/types/billing
• ne/types/endpoints
• ne/types/fixes
• ne/types/ingests
• ne/types/mismatch
• ne/types/pattern-match
• ne/types/patterns
• ne/types/unkown-types
• ne/typing-unknows
• ne/typings-ex-unit
• ne/unicode
• ne/user-controller-tests
• ne/user-module-offenses
• ne/user-prefs
• ne/users
• ne/util-tupple
• ne/vercel-tests
• ne/warning
• ne/warnings
• ne/warnings-migrations
• ne/web-utils
• non-lv-auth-checks
• order-ch-table
• otel-improvements
• otlp-metrics
• otlp-metrics-grpc
• patch-1
• perf/add-job-timeout
• perf/auto-sourcesup-shutdown
• perf/boot-init-removal
• perf/bq-clustering
• perf/broadcast-scaling
• perf/ch-tweaks
• perf/client-connections
• perf/endpoint-cache-tweaks
• perf/filtering-for-events
• perf/improve-pooling
• perf/increase-tail-time
• perf/ingesteventqueue-fix-buffer-lens-caching
• perf/ipv6
• perf/length-gt-zero
• perf/log-event-make
• perf/max-mem
• perf/memory-tweaks
• perf/nanosecond-for-recent-events-touch
• perf/only-kill-local-nodes
• perf/optional-job-creation-mode
• perf/otel-metrics-exporter-max-generations-optimizations
• perf/partition-supervisor-endpoints
• perf/partition-syn-endpoints
• perf/pool-sizes
• perf/reduce-acceptors
• perf/reduce-dd-connection-counts
• perf/reduce-ingestion-broadcast
• perf/reduce-schema-module-calls
• perf/refactor-search-query-executor
• perf/remove-buffer-producer-telemetry
• perf/remove-info-level-logging
• perf/remove-task-nesting
• perf/schema-update-error
• perf/shift-syncing-to-workers
• perf/split-socket-loading
• perf/syn-gen-scope
• perf/temp-increase-erlsysmon-limits
• perf/truncate-non-default-backend-queues
• perf/tweak-acceptors
• perf/tweak-bandit-connection-handling
• perf/tweak-ingest-event-queue-length
• perf/tweak-otel-metric-exporter-opts
• perf/tweak-sourcesup-shutdown
• perf/tweak-thousand-island
• query-backend-relationship
• rd/tcp-backend
• reduce-dry-run-queries
• refactor-bq-adaptor-execute-query
• refactor-ch-sup-init
• refactor-ecto-to-sql-transformations
• refactor/dashboard-delete-team-member
• refactor/dashboard-saved-searches
• refactor/dashboard_live
• refactor/move-humanize-bytes-to-logflareweb-utils
• refactor/remove-case-conditions
• refactor/remove-unused-sql-query
• refactor/remove_unused_code
• refactor/user-metrics-poller
• refactor/verified-routes-warning
• refs/pull/2402/merge
• remove_ch_web_from_backend_list
• revert/libcluster-postgres
• ruslandoga/ch-uuid
• s3-backend
• sentry-backend
• sql-module-refactoring
• sql-module-test-improvements
• style/formatting
• test/deterministic-assertions
• test/login-user-helper
• test/process-and-ets-metrics
• test/reduce_use_of_sleep
• test/saved_search_factory_lql
• test/telemetry-correctly-skips-delete-ets-tables
• treq/fix-add-error-logging-for-ui-routes
• treq/multiple-team-users
• treq/system-sources-partition-date-fix
• ui/remove-dashboard-member-delete
• updated-ingest-telemetry
• wake-clickhouse-connections-after-spindown
• ziinc/add-documentation-for-cloudflare-integration
• ziinc/anl-1024-logflare-deprecate-all-v1-code
• ziinc/anl-1027-remove-global-broadcasting-for-buffer-rate-and-recent
• ziinc/anl-1030-disable-pii-like-ip-address-from-logs
• ziinc/anl-1049-add-dynamicpipeline-broadway-autoscaling-to-clickhouse
• ziinc/anl-1062-lf-endpoint-redact-pii-to-redact-pii-optionally
• ziinc/anl-1096-user-is-able-to-route-account-wide-bq-queries-for
• ziinc/anl-1112-logflare-reduce-schema-sampling-to-at-most-1-event-per
• ziinc/anl-1177-expose-created_at-and-updated_at-in-log-drains-api
• ziinc/anl-31-logflare-sql-parser-adjust-parser-to-allow-union-all-queries
• ziinc/anl-643-logflare-add-ram-85-of-max-to-health-check
• ziinc/anl-645-logflare-sql-parser-fix-struct-parsing-error
• ziinc/anl-654-logflare-perf-tenant-upgraded-index
• ziinc/anl-658-add-coveralls-to-logflarelogflare
• ziinc/anl-661-create-cf-example-otel-tracing-worker-app-in-logflare-repo
• ziinc/anl-663-add-http_protobuf-ingestion-to-grpc-server
• ziinc/anl-674-logflare-grpc-otel-bugfixes-storage-team-ingestion
• ziinc/anl-675-logflare-grpc-otel-identity-encoding
• ziinc/anl-691-slackhookserver-fixes-customer-reporting-delayed
• ziinc/anl-697-fix-bad-cookie-definition-for-mnesia-on-logflare-instance
• ziinc/anl-708-logflare-search-automatically-change-chart-group
• ziinc/anl-712-logflare-perf-try-using-compressed-memory-option-for-local
• ziinc/anl-714-logflare-single-tenant-bugfix-stored-plan-data-may-change
• ziinc/anl-716-logflare-otel-traces-query-params-api-key-redaction
• ziinc/anl-717-logflare-access-tokens-ingestion-scopes-bug-for-grpc
• ziinc/anl-719-logflare-source-routing-bug-when-sink-is-nil
• ziinc/anl-720-logflare-performance-task-fixing-message-queue-backup-for
• ziinc/anl-721-logflare-perf-broadway-garbage-collection-for-batcher
• ziinc/anl-722-logflare-ingestion-orphaned-queues-get-cleaned-up-correctly
• ziinc/anl-723-recentlogsserver-high-memory-footprint-at-runtime
• ziinc/anl-724-logflare-account-deletion-bug-not-able-to-delete-acocunt-due
• ziinc/anl-725-logflare-alerts-quantum-job-failure-due-to-remote-node
• ziinc/anl-726-logflare-ingestion-routing-process-not-alive-for
• ziinc/anl-728-logflare-bandit-closed-errors
• ziinc/anl-730-logflare-remove-instance-external-address
• ziinc/anl-731-logflare-log-list-highlight-log-event-on-hover
• ziinc/anl-732-logflare-add-cluster-metadata-to-opentelemetry-traces
• ziinc/anl-794-logflare-alerts-incidentio-integration
• ziinc/anl-800-add-channel_name-option-to-postgres-libcluster-clustering
• ziinc/anl-818-o11y-tag-logflare-endpoint-queries-with-the-endpoint-uuid
• ziinc/anl-857-logflare-ingestion-improve-dynamicpipeline-resolving-to
• ziinc/anl-867-logflare-endpoints-does-not-register-nested-query-parameters
• ziinc/anl-879-logflare-queries-queries-get-truncated-due-to-query-params
• ziinc/anl-882-logflare-endpoints-support-large-sql-queries
• ziinc/anl-884-compress-logflare-endpoints-memory-footprint
• ziinc/anl-890-logflare-endpoints-post-support-for-body-params
• ziinc/anl-929-logflare-backends-should-not-show-secrets-in-config-for-json
• ziinc/anl-931-logflare-service-account-provisioning
• ziinc/anl-933-logflare-auth-google-auth-causing-issues-with-long-strings
• ziinc/anl-942-handle-deleted-google-accounts-bug
• ziinc/anl-944-revert-endpoints-partition-supervisor-change
• ziinc/anl-946-add-traces-for-endpoints-execution-and-endpointsresultscache
• ziinc/anl-947-add-in-metrics-for-logflare-endpoints
• ziinc/anl-948-logflare-byob-managed-service-accounts-handling
• ziinc/anl-953-endpoints-add-in-bq-billing-tagging-by-project
• ziinc/fix-flaky-test-by-adding-retry-logic

Build # 738f37c480c77d6fd1849214e648bb6b7f22700a-PR-2893

Build Type

Pull #2893

github

Committed by amokan

Commit Message

increase pool size for clickhouse finch pool

Pull Request Pull Request #2893: chore: rework of clickhouse ingestion pipeline to use a simplified flow for our fixed schema

Run Details

54 of 61 new or added lines in 3 files covered. (88.52%)

8537 of 12015 relevant lines covered (71.05%)

3744.93 hits per line