dgraph-io / dgraph

67%

main: 66%

LAST BUILD ON BRANCH aman/makefile

branch: aman/makefile

CHANGE BRANCH

x

Reset

• aman/makefile
• ahsan/check-deleteNS
• aman/changelog
• aman/checkout
• aman/ci
• aman/cleanup
• aman/default
• aman/dgo
• aman/dgraphtest
• aman/ff
• aman/flaky
• aman/fuzz
• aman/go-git
• aman/mac
• aman/matrix
• aman/minor
• aman/multitenancy-upgrade
• aman/normalize
• aman/oss
• aman/restore-upgrade
• aman/revert-normalize
• aman/simple
• aman/snap
• aman/test
• aman/unit
• aman/upgrade
• aman/upgrade-testing
• aman/v23.0.1
• aman/wait
• anurag/cp-rdf-over-http
• cifuzz-int
• harshil/drop_op
• jassi/upgrade-multi-tenancy
• jassi/zero_health_issue_5517
• joshua-goldstein-patch-1
• joshua/add-workflow
• joshua/coverage-fix
• joshua/coverage-security
• joshua/cves
• joshua/cves2
• joshua/dgraph-ci-fix
• joshua/pr-template
• joshua/upgrade-tests
• main
• oscs_fix_ci00k58au51qt1nt0is0
• raphael/local-test-lambda-acl
• release/v23
• sanjayk/vscode-launch-json-fix
• shiva/backup
• shiva/nfs
• shiva/upgrade2
• siddesh/failed-download
• skrdgraph/ci-frequency
• u_testing_info

Build # 5117788574

Build Type

push

Committed by web-flow

Commit Message

fix(bulk): removed buffer max size (#8841)

## Problem

On huge datasets, bulk import crashes with this stack trace:

```
panic: z.Buffer max size exceeded: 68719476736 offset: 68719476718 grow: 51

goroutine 421 [running]:
github.com/dgraph-io/ristretto/z.(*Buffer).Grow(0xc050d17c00, 0x33)
        /home/runner/go/pkg/mod/github.com/dgraph-io/ristretto@v0.1.1/z/buffer.go:180 +0x55b
github.com/dgraph-io/ristretto/z.(*Buffer).SliceAllocate(0xc050d17c00, 0x2f)
        /home/runner/go/pkg/mod/github.com/dgraph-io/ristretto@v0.1.1/z/buffer.go:266 +0x2e
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*mapIterator).Next(0xc00e56f410, 0x3?, {0xc034740c40, 0x1d, 0x20})
        /home/runner/work/dgraph/dgraph/dgraph/cmd/bulk/reduce.go:206 +0x126
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*reducer).reduce.func2()
        /home/runner/work/dgraph/dgraph/dgraph/cmd/bulk/reduce.go:486 +0x5aa
created by github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*reducer).reduce
        /home/runner/work/dgraph/dgraph/dgraph/cmd/bulk/reduce.go:476 +0x452
```

## Solution

Removing hard limit of 64GB to buffer sizes avoided panic and
successfully completed the import.

Run Details

1 of 1 new or added line in 1 file covered. (100.0%)

58165 of 86823 relevant lines covered (66.99%)

2232922.88 hits per line