Logflare / logflare

60%

main: 65%

LAST BUILD ON BRANCH perf/tweak-ingest-event-queue-length

branch: perf/tweak-ingest-event-queue-length

CHANGE BRANCH

x

Reset

• perf/tweak-ingest-event-queue-length
• add-clickhouse-docs
• avoid-atom-conversions-during-query-building
• backpressure-issue
• ch-connection-handling
• ch-ingest-batch-size-change
• ch-read-pool-init-bug
• ch_dynamic_table_naming
• chore/add-blacksmith
• chore/blacksmith-only-for-image-buidls
• chore/change-ets-metrics-to-memory-usage
• chore/ci-on-every-branch
• chore/comment-out-dialyxer-check
• chore/fix-flaky-test-attempt
• chore/flaky-test
• chore/improve-dev-env
• chore/improve-search-lv-test
• chore/libcluster-pg-fixes
• chore/pr-comments-2461
• chore/refactor-recent-logs-server
• chore/remove-debugging-statement
• chore/remove-gce-strategy
• chore/remove-libcluster-pg-ssl
• chore/remove-unused-logging
• chore/single-tenant-seeding-bug
• chore/update-carg-lockfile
• ci/use-bitnamilegacy
• clickhouse-adaptor-sql-improvements
• clickhouse-adaptor-tiny-log-removal
• clickhouse-ci-fixes
• clickhouse-connection-pools
• clickhouse-docker
• clickhouse-log-drain-phase1
• clickhouse-lql
• clickhouse-reads
• clickhouse-reads-implementation
• clickhouse-schema-changes
• clickhouse-test-setup-fix
• clickhouse_ch_client
• clickhouse_cloud_http_pool
• default-backend-require-source
• default-ingest-backend-alt
• default-ingest-improvements
• default_backend_rework
• dev-logflare-ui
• doc/fix-openapi
• docs/contributing-backends
• docs/endpoints-subquery-expansion
• docs/fix-markdown-heading
• docs/logpush-documentation
• docs/trace-ingestion
• docs/update-accesss-tokens-querying-docs
• dynamic-tracing
• endpoint-backend-abstraction
• endpoint-backend-abstractions
• endpoint-live-ui-test-improvement
• endpoint-query-bug
• endpoint-query-ui-updates
• feat/add-cluster-information-to-otel-metrics-exporter
• feat/adjust-chart-period
• feat/alerts-live-monaco-editor
• feat/all-metrics
• feat/backends-cache-refactoring
• feat/cache-clear-button
• feat/disable-tailing-source
• feat/erlsysmon-net-kernel-logging
• feat/event-context
• feat/export-metrics-through-otel
• feat/gen-singleton
• feat/gensgingleton-transient
• feat/improve-local-timezone-viewing
• feat/ingest-query-scopes
• feat/modal-custom-commands
• feat/monaco_editor_for_endpoints
• feat/move-delete-source-button
• feat/open-api-improvements
• feat/optional-datadog-pools
• feat/otlp-logs-ingestion
• feat/query-cost-ui-feedback
• feat/search-page-log-levels
• feat/storage-write-api-improvements
• feat/storage-write-api-improvements-v2
• feat/storage-write-api-initial-v2
• feat/support-ap2-datadog
• feat/time-partition-bq-hour
• feat/top-processes-and-ets-tables-metrics
• feat/user-system-monitoring
• fix-startup
• fix/add-loki-backend-username-password
• fix/adjustments-from-last-pr
• fix/admin-accounts-list
• fix/alerting-improvements
• fix/alerts-scheduler-termination
• fix/artifact-repository-grafana
• fix/avoid-creating-log-events-from-cache
• fix/bandit-version-bump-metrics
• fix/bq-sql-extra-quotes
• fix/broken-documentation-links
• fix/ce-role
• fix/changed-endoint-labels-clears-cache
• fix/chart-timezone
• fix/cleaner-query-lables
• fix/compile-time-env-value
• fix/credo-diff-against-main
• fix/datadog-http1
• fix/delete-source-link
• fix/endpoint-run-query
• fix/endpoints-form-changes
• fix/endpoints-shutdown-bug
• fix/event-modal-permalink
• fix/fetch-from-paritiontime-table
• fix/field-in-array
• fix/global-broadcast
• fix/goth-partitioning-sync
• fix/grpc-server-gzip
• fix/gzip-cached-query-results
• fix/handle-mutliple-union-all
• fix/handle-syn-resolve-kill
• fix/libcluster-pg
• fix/log-event-whitespace
• fix/lql-aggregates
• fix/lql-negations
• fix/managed-sa-accounts-when-disabled
• fix/metadata-queries
• fix/misc-bugs
• fix/move-ensure-source-sup-started
• fix/nil-attributes
• fix/node-22-jquery-fix
• fix/otel-ingestion
• fix/pii-redaction-testing
• fix/pino-logflare-link
• fix/pseudo-streaming-buffer
• fix/reduce-node-connection-attempts
• fix/refactor-log-event-live-and-component
• fix/regression-query-error
• fix/reject-ets-info-undefined-returns
• fix/remove-compile-env
• fix/remove-error-string
• fix/remove-grpc-logger
• fix/restrict-tesla-env-structs-stringifying
• fix/revert-gzip-results
• fix/run-endpoint-query-with-labels
• fix/save-incidentio-backend-metadata-error
• fix/source-routing-without-sink
• fix/sql-postgres-translation
• fix/ssl-ecto-repo
• fix/syn-conflict-stop-proc
• fix/syn-silencing
• fix/syn-stop-supervisor
• fix/sync-rules
• fix/test-ce-role
• fix/test-query-returned-error
• fix/typespecs
• fix/uncontrolled-endpoints-form
• fix/unnesting-metadata
• google-namespace-refactor
• hide-backend-creds-from-view
• improving-tracing
• lql-filter-rule-bug
• lql-improvements
• lql-mimic-test-mock-issues
• lql-rule-serialization-bug
• main
• more-test-improvements
• multi-default-backends
• naive-ch-pr-additions
• ne/admin-controller-tests
• ne/backed
• ne/backend-adaptors
• ne/backend-form
• ne/backends
• ne/backends-postgres
• ne/bigquery-refactor
• ne/bigquery-schema-refactor
• ne/billing
• ne/cargo
• ne/cluster-live
• ne/coveralls
• ne/credo
• ne/credo-nesting-logflare-context_cache)
• ne/credo-nesting-logflare-logs-source_parsers-syslog_parser)
• ne/credo-nesting-logflare-logs-vercel-nimble_vercel_parser)
• ne/credo-nesting-logflare-logs-vercel-vercel_parser)
• ne/credo-nesting-logflare-lql-parser)
• ne/credo-nesting-logflare-source-text_notification_server)
• ne/credo-nesting-logflare-sql-dialect_translation)
• ne/credo-nesting-logflare-users-users)
• ne/credo-nesting-logflare_web-controllers-plugs-require_auth)
• ne/credo-nesting-logflare_web-controllers-plugs-set_team_user)
• ne/credo-nesting-logflare_web-live-vercellogdrains_live-index)
• ne/dbg
• ne/deprecated-cache-local
• ne/dialect-translation-refactor
• ne/dialyzer
• ne/display-helpers
• ne/ecto-atom-type
• ne/ecto-regex-tests
• ne/ecto-term
• ne/edit-billing-tests
• ne/elixir-logger-view
• ne/enable-typing
• ne/endpoint
• ne/endpoint-controller-api
• ne/endpoints
• ne/endpoints-resolver
• ne/errors
• ne/ets
• ne/ex-coveralls
• ne/fix-text-notification-server
• ne/flaky
• ne/flaky-backends
• ne/flaky-test
• ne/flaky-tests
• ne/gen_utils
• ne/grpc-tests
• ne/lib/logflare/alerting
• ne/lib/logflare/backends/adaptor/postgres_adaptor/pg_repo
• ne/lib/logflare/billing
• ne/lib/logflare/context_cache
• ne/lib/logflare/logs/log_event
• ne/lib/logflare/rules
• ne/lib/logflare/source/email_notification_server
• ne/lib/logflare_web/live/backends/backends_live
• ne/lib/logflare_web/live/source_backends_live
• ne/live-rules
• ne/logflare-utils-parens
• ne/logflare/backends-parens
• ne/logflare/contexts-parens
• ne/logflare/google-parens
• ne/logflare/logs-parens
• ne/logflare/lql-parens
• ne/logflare/pubsub-parens
• ne/logflare/source-parens
• ne/logflare_web/controllers-parens
• ne/logflare_web/live-parens
• ne/logs-search-live-refactor
• ne/logs-search-operations-nesting
• ne/logs-types
• ne/long-lines
• ne/mimic
• ne/mimic-upgrade
• ne/mix-test
• ne/nesting
• ne/open-telemetry-sampler
• ne/open_api_spex
• ne/parens
• ne/partner-live
• ne/phoenix-test
• ne/plug-set-team-if-nil
• ne/refactor-old-templates
• ne/refactor-rpc
• ne/refactor-utils
• ne/rules
• ne/rules-followup
• ne/silent_logs
• ne/sobelow
• ne/source
• ne/source-bigquery-nesting
• ne/source-controller
• ne/source-data-nesting
• ne/source-routing-nested
• ne/source-schemas
• ne/sources-controller
• ne/sql-typos
• ne/stream-data
• ne/system-metrics-parens
• ne/team-controller-tests
• ne/team-user-controller
• ne/teams
• ne/tesla
• ne/test
• ne/test-parens
• ne/test-util
• ne/test-warnings
• ne/testing-unsubscribing
• ne/tests-clickhouse
• ne/tests-coverage-context
• ne/trys
• ne/types
• ne/types/alerting
• ne/types/backends
• ne/types/billing
• ne/types/endpoints
• ne/types/fixes
• ne/types/ingests
• ne/types/mismatch
• ne/types/pattern-match
• ne/types/patterns
• ne/types/unkown-types
• ne/typing-unknows
• ne/typings-ex-unit
• ne/unicode
• ne/user-controller-tests
• ne/user-prefs
• ne/users
• ne/vercel-tests
• ne/warning
• ne/warnings
• ne/warnings-migrations
• ne/web-utils
• order-ch-table
• otel-improvements
• otlp-metrics
• otlp-metrics-grpc
• patch-1
• perf/add-job-timeout
• perf/boot-init-removal
• perf/bq-clustering
• perf/broadcast-scaling
• perf/endpoint-cache-tweaks
• perf/filtering-for-events
• perf/improve-pooling
• perf/ipv6
• perf/memory-tweaks
• perf/nanosecond-for-recent-events-touch
• perf/only-kill-local-nodes
• perf/optional-job-creation-mode
• perf/partition-supervisor-endpoints
• perf/partition-syn-endpoints
• perf/pool-sizes
• perf/reduce-dd-connection-counts
• perf/reduce-schema-module-calls
• perf/refactor-search-query-executor
• perf/remove-info-level-logging
• perf/remove-task-nesting
• perf/schema-update-error
• perf/shift-syncing-to-workers
• perf/temp-increase-erlsysmon-limits
• perf/tweak-bandit-connection-handling
• query-backend-relationship
• reduce-dry-run-queries
• refactor-bq-adaptor-execute-query
• refactor-ch-sup-init
• refactor-ecto-to-sql-transformations
• refactor/dashboard-delete-team-member
• refactor/dashboard-saved-searches
• refactor/dashboard_live
• refactor/move-humanize-bytes-to-logflareweb-utils
• refactor/remove-unused-sql-query
• refactor/remove_unused_code
• refactor/user-metrics-poller
• refactor/verified-routes-warning
• refs/pull/2402/merge
• remove_ch_web_from_backend_list
• revert/libcluster-postgres
• s3-backend
• sentry-backend
• sql-module-refactoring
• sql-module-test-improvements
• test/process-and-ets-metrics
• test/reduce_use_of_sleep
• test/saved_search_factory_lql
• test/telemetry-correctly-skips-delete-ets-tables
• ui/remove-dashboard-member-delete
• updated-ingest-telemetry
• wake-clickhouse-connections-after-spindown
• ziinc/anl-1030-disable-pii-like-ip-address-from-logs
• ziinc/anl-1049-add-dynamicpipeline-broadway-autoscaling-to-clickhouse
• ziinc/anl-1062-lf-endpoint-redact-pii-to-redact-pii-optionally
• ziinc/anl-1096-user-is-able-to-route-account-wide-bq-queries-for
• ziinc/anl-31-logflare-sql-parser-adjust-parser-to-allow-union-all-queries
• ziinc/anl-643-logflare-add-ram-85-of-max-to-health-check
• ziinc/anl-645-logflare-sql-parser-fix-struct-parsing-error
• ziinc/anl-654-logflare-perf-tenant-upgraded-index
• ziinc/anl-658-add-coveralls-to-logflarelogflare
• ziinc/anl-661-create-cf-example-otel-tracing-worker-app-in-logflare-repo
• ziinc/anl-663-add-http_protobuf-ingestion-to-grpc-server
• ziinc/anl-674-logflare-grpc-otel-bugfixes-storage-team-ingestion
• ziinc/anl-675-logflare-grpc-otel-identity-encoding
• ziinc/anl-691-slackhookserver-fixes-customer-reporting-delayed
• ziinc/anl-697-fix-bad-cookie-definition-for-mnesia-on-logflare-instance
• ziinc/anl-708-logflare-search-automatically-change-chart-group
• ziinc/anl-712-logflare-perf-try-using-compressed-memory-option-for-local
• ziinc/anl-714-logflare-single-tenant-bugfix-stored-plan-data-may-change
• ziinc/anl-716-logflare-otel-traces-query-params-api-key-redaction
• ziinc/anl-717-logflare-access-tokens-ingestion-scopes-bug-for-grpc
• ziinc/anl-719-logflare-source-routing-bug-when-sink-is-nil
• ziinc/anl-720-logflare-performance-task-fixing-message-queue-backup-for
• ziinc/anl-721-logflare-perf-broadway-garbage-collection-for-batcher
• ziinc/anl-722-logflare-ingestion-orphaned-queues-get-cleaned-up-correctly
• ziinc/anl-723-recentlogsserver-high-memory-footprint-at-runtime
• ziinc/anl-724-logflare-account-deletion-bug-not-able-to-delete-acocunt-due
• ziinc/anl-725-logflare-alerts-quantum-job-failure-due-to-remote-node
• ziinc/anl-726-logflare-ingestion-routing-process-not-alive-for
• ziinc/anl-728-logflare-bandit-closed-errors
• ziinc/anl-730-logflare-remove-instance-external-address
• ziinc/anl-731-logflare-log-list-highlight-log-event-on-hover
• ziinc/anl-732-logflare-add-cluster-metadata-to-opentelemetry-traces
• ziinc/anl-794-logflare-alerts-incidentio-integration
• ziinc/anl-800-add-channel_name-option-to-postgres-libcluster-clustering
• ziinc/anl-818-o11y-tag-logflare-endpoint-queries-with-the-endpoint-uuid
• ziinc/anl-857-logflare-ingestion-improve-dynamicpipeline-resolving-to
• ziinc/anl-867-logflare-endpoints-does-not-register-nested-query-parameters
• ziinc/anl-879-logflare-queries-queries-get-truncated-due-to-query-params
• ziinc/anl-882-logflare-endpoints-support-large-sql-queries
• ziinc/anl-884-compress-logflare-endpoints-memory-footprint
• ziinc/anl-890-logflare-endpoints-post-support-for-body-params
• ziinc/anl-931-logflare-service-account-provisioning
• ziinc/anl-933-logflare-auth-google-auth-causing-issues-with-long-strings
• ziinc/anl-942-handle-deleted-google-accounts-bug
• ziinc/anl-944-revert-endpoints-partition-supervisor-change
• ziinc/anl-946-add-traces-for-endpoints-execution-and-endpointsresultscache
• ziinc/anl-947-add-in-metrics-for-logflare-endpoints
• ziinc/anl-948-logflare-byob-managed-service-accounts-handling
• ziinc/anl-953-endpoints-add-in-bq-billing-tagging-by-project

Build # af5dac053171fa3e089b3a0135a3a4203d980efe-PR-2369

Build Type

Pull #2369

github

Committed by Ziinc

Commit Message

perf: tweak ingestion throughput and batch sizes

Pull Request Pull Request #2369: perf: ingestion tweaking for bigquery

Run Details

4 of 9 new or added lines in 3 files covered. (44.44%)

5515 of 9163 relevant lines covered (60.19%)

1624.88 hits per line