thanos / ExZarr

66%

main: 76%

LAST BUILD ON BRANCH S3_storage_backend

branch: S3_storage_backend

CHANGE BRANCH

x

Reset

Sync Branches

• S3_storage_backend
• Additional_Data_Types
• Array_Operations
• Chunking_and_Group
• Complete_Zarr_v3_Metadata_Handling
• Concurrent_RW_Support
• Documentation_and_Examples
• Google_Cloud_Storage_backend
• doc_cleanup
• docs_fix
• github_setup
• main
• new_documentaion
• new_notebooks
• nx_integration
• performance_and_benchmarks
• release_v1.0.0
• v0.3.0
• v2_and_v3_formats_conversions
• zarr3_extras
• zarr_v3
• zarr_v3_metadata_support

Build # 0cb9e445397b6dd8dca57514be7e37dd2cb58041-PR-134

Build Type

Pull #134

github

Committed by thanos

Commit Message

closed #133 - S3 Storage Backend Implementation Complete

  Summary

  I've successfully completed the S3 storage backend implementation for ExZarr. While the S3 backend code already existed in the codebase, it was missing comprehensive tests and examples. I've added:

  What Was Added

  1. S3 Mock Tests (test/ex_zarr/storage/s3_mock_test.exs)

  - 45 comprehensive tests covering all S3 backend operations
  - Uses Mox to mock ExAws and ExAws.S3 modules for fast, reliable testing
  - No AWS credentials required - runs in CI/CD pipelines
  - Coverage includes:
    - Initialization and configuration
    - Chunk read/write operations (1D, 2D, 3D, ND indices)
    - Metadata operations
    - List and delete operations
    - Error handling (missing chunks, invalid buckets, S3 failures)
    - Chunk key encoding for all dimensions
    - Concurrent operations
    - Integration scenarios

  2. S3 Integration Tests (test/ex_zarr/storage/s3_test.exs)

  - Real S3 testing with localstack or minio
  - Tagged with @tag :s3 (excluded by default)
  - Tests include:
    - Backend registration
    - Real metadata and chunk operations
    - Concurrent read/write operations
    - Prefix isolation
    - Storage module integration
    - Full workflow tests
  - Complete setup instructions for localstack/minio in comments

  3. S3 Usage Example (examples/s3_storage.exs)

  - Comprehensive, executable example demonstrating S3 usage
  - Six detailed examples:
    a. Basic array creation in S3
    b. Writing and reading data with Nx integration
    c. Direct chunk operations
    d. Multiple arrays with different prefixes
    e. Error handling patterns
    f. Performance tips and best practices
  - Includes setup instructions for AWS and minio
  - Production-ready code with proper error handling

Pull Request Pull Request #134: closed #133 - S3 Storage Backend Implementation Complete

Run Details

4 of 21 new or added lines in 1 file covered. (19.05%)

1938 of 2928 relevant lines covered (66.19%)

1058.0 hits per line