Agora ETL Framework

When to read this: you want the quickest map of what Agora does and where to go next in the documentation.

agora-etl is a Python async ETL framework built around a Source → Middleware chain → Sink(s) model. It handles checkpointing, dead-letter queues, retries, and long-running workers so you can focus on the transformation logic.

The runtime stays intentionally smaller than the whole ecosystem:

  • agora-etl owns runtime semantics and extension contracts
  • agora-etl-plugins owns official backend integrations and backend-specific production guidance
  • agora-etl-rs stays optional and accelerates selected hot paths without changing runtime semantics
  • operator-facing product workflows should build on those contracts instead of redefining them

Start here

Guides

Guide What it covers
Quickstart Build and run a first pipeline
Pipelines Compose sources, middlewares, sinks, fan-out, and routing
Lifecycle Learn startup, run, shutdown, worker, and replay order
Running with uvloop Opt into uvloop at the application entrypoint without changing core runtime semantics
Runtime Guarantees What the runtime promises under success, failure, and restart
Failure Handling DLQ, retry, and sink failure policies
Checkpointing Resume long-running and file-based pipelines
Scheduling ScheduledPipeline, WorkerPool, and graceful shutdown
Testing Test sources, middlewares, and whole pipelines
Observability Run summaries, health endpoints, metrics, and tracing
Configuration AgoraSettings and agora/v1 TOML configs
Plugins Official plugin families and plugin authoring

Reference

Reference What it covers
Sources Built-in and custom source types
Sinks Built-in and custom sink types
Middlewares Built-in, AI, and custom middlewares
Schema Schema inference, contracts, and persistence
State Shared key-value backends and helper stores
Architecture Execution lanes, state, plugin loading, and runtime structure
CLI Command reference
Change Log Release history