24841644892

Committed 23 Apr 2026 02:44PM UTC coverage: 85.279% (-4.0%) from 89.306%

Build # 24841644892

Build Type

push

gh-ci

Committed by

web-flow

Commit Message

feat: v1.11.0 PostgreSQL compatibility and SQL feature expansion (#2090)

* Add structured audit logging with immutable audit trail

Introduces a new --audit-log flag that records all gRPC operations as
structured JSON events in immudb's tamper-proof KV store. Events are
stored under the audit: key prefix in systemdb, queryable via Scan and
verifiable via VerifiableGet. An async buffered writer ensures minimal
latency impact. Configurable event filtering (all/write/admin) via
--audit-log-events flag.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Add PostgreSQL ORM compatibility layer and verification functions

Extend the pgsql wire protocol with immudb verification functions
(immudb_state, immudb_verify_row, immudb_verify_tx, immudb_history,
immudb_tx) accessible via standard SQL SELECT statements.

Add pg_catalog resolvers (pg_attribute, pg_index, pg_constraint,
pg_type, pg_settings, pg_description) and information_schema
resolvers (tables, columns, schemata, key_column_usage) to support
ORM introspection from Django, SQLAlchemy, GORM, and ActiveRecord.

Add PostgreSQL compatibility functions: current_database,
current_schema, current_user, format_type, pg_encoding_to_char,
pg_get_expr, pg_get_constraintdef, obj_description, col_description,
has_table_privilege, has_schema_privilege, and others.

Add SHOW statement emulation for common ORM config queries and
schema-qualified name stripping for information_schema and public
schema references.

* Implement EXISTS and IN subquery support in SQL engine

Replace the previously stubbed ExistsBoolExp and InSubQueryExp
implementations with working non-correlated subquery execution.

EXISTS subqueries resolve the inner SELECT and check if any rows
are returned. IN subqueries resolve the inner SELECT, iterate the
result set, and compare each value against the outer expression.
Both support NOT variants (NOT EXISTS, NOT IN).

Correlated subqueries (referencing outer query columns) ar... (continued)

Coverage Stats

7254 of 10471 new or added lines in 124 files covered. (69.28%)

115 existing lines in 18 files now uncovered.

44599 of 52298 relevant lines covered (85.28%)

127676.6 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

84.38

/embedded/sql/options.go

/*
Copyright 2026 Codenotary Inc. All rights reserved.

SPDX-License-Identifier: BUSL-1.1
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    https://mariadb.com/bsl11/

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package sql

import (
        "fmt"

        "github.com/codenotary/immudb/embedded/store"
)

const (
        defaultDistinctLimit  = 1 << 20 // ~ 1mi rows
        defaultSortBufferSize = 1024
)

type Options struct {
        prefix                        []byte
        sortBufferSize                int
        distinctLimit                 int
        distinctSpillThreshold        int // 0 = spill disabled, hard distinctLimit cap applies
        maxWindowRows                 int // 0 = unlimited
        maxKeyLen                     int // 0 = leave package default in place
        autocommit                    bool
        lazyIndexConstraintValidation bool
        parseTxMetadata               func([]byte) (map[string]interface{}, error)

        multidbHandler MultiDBHandler
        tableResolvers []TableResolver
}

func DefaultOptions() *Options {
        return &Options{
                sortBufferSize: defaultSortBufferSize,
                distinctLimit:  defaultDistinctLimit,
        }
}

func (opts *Options) Validate() error {
        if opts == nil {
                return fmt.Errorf("%w: nil options", store.ErrInvalidOptions)
        }

        if opts.distinctLimit <= 0 {
                return fmt.Errorf("%w: invalid DistinctLimit value", store.ErrInvalidOptions)
        }

        if opts.sortBufferSize <= 0 {
                return fmt.Errorf("%w: invalid SortBufferSize value", store.ErrInvalidOptions)
        }

        // 0 means "leave the package default" (no override). Anything explicit
        // must fit a uint16 length-prefix (the on-disk encoding ceiling) and
        // be at least wide enough for the small system PKs.
        if opts.maxKeyLen != 0 && (opts.maxKeyLen < 64 || opts.maxKeyLen > 65535) {
                return fmt.Errorf("%w: MaxKeyLen must be in [64, 65535]", store.ErrInvalidOptions)
        }

        return nil
}

func (opts *Options) WithPrefix(prefix []byte) *Options {
        opts.prefix = prefix
        return opts
}

func (opts *Options) WithDistinctLimit(distinctLimit int) *Options {
        opts.distinctLimit = distinctLimit
        return opts
}

// WithDistinctSpillThreshold enables D5 spill-to-disk for SELECT DISTINCT.
// When threshold > 0, the in-memory dedup set spills to a temp file once
// it accumulates threshold distinct digests, allowing DISTINCT queries
// over arbitrarily large result sets without OOMing the process. The
// distinctLimit hard cap is then ignored (effectively unlimited rows).
//
// Default 0 preserves the legacy behaviour: ErrTooManyRows when the
// in-memory dedup set hits distinctLimit, no temp-file spill.
//
// Pick a threshold around (RAM budget / sha256.Size / 4): each digest
// is 32 bytes plus map overhead, so 100k → ~10MB working set. Larger
// thresholds reduce spill churn at the cost of memory headroom.
func (opts *Options) WithDistinctSpillThreshold(n int) *Options {
        opts.distinctSpillThreshold = n
        return opts
}

func (opts *Options) WithAutocommit(autocommit bool) *Options {
        opts.autocommit = autocommit
        return opts
}

func (opts *Options) WithLazyIndexConstraintValidation(lazyIndexConstraintValidation bool) *Options {
        opts.lazyIndexConstraintValidation = lazyIndexConstraintValidation
        return opts
}

func (opts *Options) WithMultiDBHandler(multidbHandler MultiDBHandler) *Options {
        opts.multidbHandler = multidbHandler
        return opts
}

// WithSortBufferSize specifies the size of the buffer used to sort rows in-memory
// when executing queries containing an ORDER BY clause. The default value is 1024.
// Increasing this value improves sorting speed at the expense of higher memory usage.
func (opts *Options) WithSortBufferSize(size int) *Options {
        opts.sortBufferSize = size
        return opts
}

func (opts *Options) WithMaxWindowRows(maxRows int) *Options {
        opts.maxWindowRows = maxRows
        return opts
}

func (opts *Options) WithParseTxMetadataFunc(parseFunc func([]byte) (map[string]interface{}, error)) *Options {
        opts.parseTxMetadata = parseFunc
        return opts
}

func (opts *Options) WithTableResolvers(resolvers ...TableResolver) *Options {
        opts.tableResolvers = append(opts.tableResolvers, resolvers...)
        return opts
}

// WithMaxKeyLen overrides the engine-side maximum length (in bytes) for
// indexed VARCHAR columns. The value is clamped to [64, 65535] by Validate.
// When set, NewEngine assigns it to the package-level MaxKeyLen so that
// EncodeRawValueAsKey (and every other site that consults MaxKeyLen) sees
// the new ceiling. Note that the underlying store layer still enforces
// its own composite-key limit (embedded/store.MaxKeyLen, default 1024 B),
// which is the practical ceiling at insert time. Pass 0 (or omit the
// call) to leave the package default in place — the value is unchanged.
func (opts *Options) WithMaxKeyLen(maxKeyLen int) *Options {
        opts.maxKeyLen = maxKeyLen
        return opts
}

1	/*
2	Copyright 2026 Codenotary Inc. All rights reserved.
3
4	SPDX-License-Identifier: BUSL-1.1
5	you may not use this file except in compliance with the License.
6	You may obtain a copy of the License at
7
8	https://mariadb.com/bsl11/
9
10	Unless required by applicable law or agreed to in writing, software
11	distributed under the License is distributed on an "AS IS" BASIS,
12	WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13	See the License for the specific language governing permissions and
14	limitations under the License.
15	*/
16
17	package sql
18
19	import (
20	"fmt"
21
22	"github.com/codenotary/immudb/embedded/store"
23	)
24
25	const (
26	defaultDistinctLimit = 1 << 20 // ~ 1mi rows
27	defaultSortBufferSize = 1024
28	)
29
30	type Options struct {
31	prefix []byte
32	sortBufferSize int
33	distinctLimit int
34	distinctSpillThreshold int // 0 = spill disabled, hard distinctLimit cap applies
35	maxWindowRows int // 0 = unlimited
36	maxKeyLen int // 0 = leave package default in place
37	autocommit bool
38	lazyIndexConstraintValidation bool
39	parseTxMetadata func([]byte) (map[string]interface{}, error)
40
41	multidbHandler MultiDBHandler
42	tableResolvers []TableResolver
43	}
44
45	func DefaultOptions() *Options {	1,914✔
46	return &Options{	1,914✔
47	sortBufferSize: defaultSortBufferSize,	1,914✔
48	distinctLimit: defaultDistinctLimit,	1,914✔
49	}	1,914✔
50	}	1,914✔
51
52	func (opts *Options) Validate() error {	1,917✔
53	if opts == nil {	1,918✔
54	return fmt.Errorf("%w: nil options", store.ErrInvalidOptions)	1✔
55	}	1✔
56
57	if opts.distinctLimit <= 0 {	1,918✔
58	return fmt.Errorf("%w: invalid DistinctLimit value", store.ErrInvalidOptions)	2✔
59	}	2✔
60
61	if opts.sortBufferSize <= 0 {	1,915✔
62	return fmt.Errorf("%w: invalid SortBufferSize value", store.ErrInvalidOptions)	1✔
63	}	1✔
64
65	// 0 means "leave the package default" (no override). Anything explicit
66	// must fit a uint16 length-prefix (the on-disk encoding ceiling) and
67	// be at least wide enough for the small system PKs.
68	if opts.maxKeyLen != 0 && (opts.maxKeyLen < 64 \|\| opts.maxKeyLen > 65535) {	1,913✔
NEW 69	return fmt.Errorf("%w: MaxKeyLen must be in [64, 65535]", store.ErrInvalidOptions)	×
NEW 70	}	×
71
72	return nil	1,913✔
73	}
74
75	func (opts Options) WithPrefix(prefix []byte) Options {	1,914✔
76	opts.prefix = prefix	1,914✔
77	return opts	1,914✔
78	}	1,914✔
79
80	func (opts Options) WithDistinctLimit(distinctLimit int) Options {	3✔
81	opts.distinctLimit = distinctLimit	3✔
82	return opts	3✔
83	}	3✔
84
85	// WithDistinctSpillThreshold enables D5 spill-to-disk for SELECT DISTINCT.
86	// When threshold > 0, the in-memory dedup set spills to a temp file once
87	// it accumulates threshold distinct digests, allowing DISTINCT queries
88	// over arbitrarily large result sets without OOMing the process. The
89	// distinctLimit hard cap is then ignored (effectively unlimited rows).
90	//
91	// Default 0 preserves the legacy behaviour: ErrTooManyRows when the
92	// in-memory dedup set hits distinctLimit, no temp-file spill.
93	//
94	// Pick a threshold around (RAM budget / sha256.Size / 4): each digest
95	// is 32 bytes plus map overhead, so 100k → ~10MB working set. Larger
96	// thresholds reduce spill churn at the cost of memory headroom.
97	func (opts Options) WithDistinctSpillThreshold(n int) Options {	2✔
98	opts.distinctSpillThreshold = n	2✔
99	return opts	2✔
100	}	2✔
101
102	func (opts Options) WithAutocommit(autocommit bool) Options {	3✔
103	opts.autocommit = autocommit	3✔
104	return opts	3✔
105	}	3✔
106
107	func (opts Options) WithLazyIndexConstraintValidation(lazyIndexConstraintValidation bool) Options {	851✔
108	opts.lazyIndexConstraintValidation = lazyIndexConstraintValidation	851✔
109	return opts	851✔
110	}	851✔
111
112	func (opts Options) WithMultiDBHandler(multidbHandler MultiDBHandler) Options {	837✔
113	opts.multidbHandler = multidbHandler	837✔
114	return opts	837✔
115	}	837✔
116
117	// WithSortBufferSize specifies the size of the buffer used to sort rows in-memory
118	// when executing queries containing an ORDER BY clause. The default value is 1024.
119	// Increasing this value improves sorting speed at the expense of higher memory usage.
120	func (opts Options) WithSortBufferSize(size int) Options {	3✔
121	opts.sortBufferSize = size	3✔
122	return opts	3✔
123	}	3✔
124
NEW 125	func (opts Options) WithMaxWindowRows(maxRows int) Options {	×
NEW 126	opts.maxWindowRows = maxRows	×
NEW 127	return opts	×
NEW 128	}	×
129
130	func (opts Options) WithParseTxMetadataFunc(parseFunc func([]byte) (map[string]interface{}, error)) Options {	836✔
131	opts.parseTxMetadata = parseFunc	836✔
132	return opts	836✔
133	}	836✔
134
135	func (opts Options) WithTableResolvers(resolvers ...TableResolver) Options {	1✔
136	opts.tableResolvers = append(opts.tableResolvers, resolvers...)	1✔
137	return opts	1✔
138	}	1✔
139
140	// WithMaxKeyLen overrides the engine-side maximum length (in bytes) for
141	// indexed VARCHAR columns. The value is clamped to [64, 65535] by Validate.
142	// When set, NewEngine assigns it to the package-level MaxKeyLen so that
143	// EncodeRawValueAsKey (and every other site that consults MaxKeyLen) sees
144	// the new ceiling. Note that the underlying store layer still enforces
145	// its own composite-key limit (embedded/store.MaxKeyLen, default 1024 B),
146	// which is the practical ceiling at insert time. Pass 0 (or omit the
147	// call) to leave the package default in place — the value is unchanged.
NEW 148	func (opts Options) WithMaxKeyLen(maxKeyLen int) Options {	×
NEW 149	opts.maxKeyLen = maxKeyLen	×
NEW 150	return opts	×
NEW 151	}	×

codenotary / immudb / 24841644892

Source File Press 'n' to go to next uncovered line, 'b' for previous

Source File
Press 'n' to go to next uncovered line, 'b' for previous