cshum / imagor / 21428639185

92%

push

github

feat: add S3 loader bucket routing with pattern-based multi-region support (#706)

* feat: add S3 loader bucket routing with multi-region and fallback support

Add prefix-based bucket routing for S3 loader to support multi-tenant and
multi-region setups. Images can be routed to different S3 buckets based on
path prefix, with each bucket supporting its own region, endpoint, and
credentials.

Key features:
- Longest-prefix-first matching for routing rules
- Per-bucket configuration: region, endpoint, credentials
- Up to 2 fallback buckets when object not found in primary
- Efficient S3 client caching per unique region/endpoint/credentials
- YAML configuration file support

Configuration example:
  default_bucket:
    name: imagor-default
    region: us-east-1
  fallback_buckets:
    - name: imagor-archive
      region: us-west-2
  rules:
    - prefix: users
      bucket:
        name: imagor-users
        region: eu-west-1
    - prefix: products
      bucket:
        name: imagor-products
        region: ap-southeast-1

Usage:
  S3_LOADER_BUCKET_ROUTER_CONFIG=/path/to/config.yaml

Implementation notes:
- Uses GetObject directly for fallback checks (single API call per bucket)
- Lazy blob loading preserved for non-fallback paths
- Thread-safe client caching with RWMutex
- Full test coverage for routing and YAML parsing

* refactor: switch S3 bucket routing from prefix to pattern-based matching

- Replace prefix-based rules with regex pattern extraction using named capture group
- routing_pattern with (?P<bucket>...) extracts bucket identifier from object key
- Supports S3 partition optimization via random prefixes (e.g., f7a3-B1-image.jpg)
- Update README with new YAML config format and pattern examples

* refactor: move bucket routing to separate S3RouterLoader

Per maintainer feedback, routing logic is now in a separate loader
(S3RouterLoader) instead of being embedded in S3Storage.

Architecture:
- S3RouterLoader implements imagor.Loader interface
- R... (continued)

131 of 177 new or added lines in 4 files covered. (74.01%)

5583 of 6077 relevant lines covered (91.87%)

1.1 hits per line