Change Log: 0.2.2

Released: June 3, 2026

Focus

Operational visibility for long-lived workers, safer partial-batch delivery, and a more production-shaped Kafka worker example.

Highlights

Live worker observability for long-lived pipelines

WorkerPool now registers pipelines with the metrics collector before the first run completes, so /health and /metrics can show long-lived consumers while they are still running.

Live health payloads now expose:

  • registered pipeline metadata and schedule
  • running lifecycle state and active run id
  • live throughput and elapsed runtime
  • live records_consumed, records_written, records_dropped, records_errored
  • records_pending for in-flight records that have not reached a terminal outcome yet

This makes the health surface useful for continuous workers that may not finish a run for hours or days.

Timed batch flushes for long-lived sources

DeliveryConfig now supports batch_flush_interval_ms for batched delivery. This allows long-lived sources to flush partial sink batches after a bounded wait, instead of holding records indefinitely until the batch fills or the run ends.

The implementation uses a single-owner flush task for pending_writes, which avoids duplicate delivery when timeout-based flushing and new arrivals happen close together.

Live metrics no longer under-report buffered source counters

Live /health snapshots now include hot-path source counters that have not yet been flushed into the main PipelineMetrics object. This fixes confusing observability states where records_written could temporarily appear higher than records_consumed during a live run.

Kafka worker example updated for long-lived operation

The etl-order-reliability example now models long-lived Kafka consumers more closely:

  • consumer pipelines stay attached to their Kafka group instead of exiting after short chunked runs
  • the worker no longer auto-runs the sample producer
  • Kafka UI is included in the local Docker stack for topic, group, and lag inspection
  • the example uses timed batch flushing to avoid stuck partial batches at low traffic

Tests

Release validation includes:

  • worker observability and health response coverage for running pipelines
  • regression coverage for timed partial-batch flushes
  • example worker tests for the Kafka reliability example
  • full package CI from the source tree

Packaging

  • Bumped agora-etl to 0.2.2.
  • Root make ci now delegates to package-level CI, and package test targets run with PYTHONPATH=src so CI validates the checked-out source tree rather than a stale installed wheel.