astronomer / astro-cli / b9177ddb-bcd0-4b87-9b75-2b0fe82cb535

39%

main: 45%

Build Type

Pull #2078

circleci

Committed by jlaneve

Commit Message

Batch import Airflow settings to speed up `astro dev start`

Previously, `AddVariables`, `AddConnections`, and `AddPools` each ran a
separate `docker exec` (or `bash -c` for standalone mode) per object.
Every call paid the full Airflow CLI startup cost (~2-5s), so a project
with 10 connections, 5 variables, and 2 pools would spend 1-2+ minutes
just on settings injection.

Replace the per-item loops with Airflow's native batch import commands
(`airflow variables import`, `airflow connections import --overwrite`,
`airflow pools import`) for Airflow 2+. A single JSON file is written
into the container/venv via a heredoc and imported in one CLI invocation,
reducing O(n) exec calls to O(1) per object type.

Key changes:
- Variables: marshal to `{"name":"value",...}` JSON, import in one call
- Connections: marshal to `{"conn_id":{...}}` JSON (or URI string),
  import with `--overwrite` (eliminates the list+delete+add cycle)
- Pools: marshal to `{"pool_name":{"slots":N,"description":"..."}}` JSON,
  import in one call
- Airflow 1 codepaths are preserved unchanged as legacy fallbacks
- Exit code from the import command is captured and propagated even
  after temp file cleanup

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Pull Request Pull Request #2078: Batch import Airflow settings to speed up dev start

Coverage Stats

152 of 179 new or added lines in 1 file covered. (84.92%)

33 existing lines in 2 files now uncovered.

24937 of 63291 relevant lines covered (39.4%)

9.49 hits per line