LoginSign Up
← Back to all sparks
A

Apache DolphinScheduler

INFRA · APIS
Velocity2.5

Distributed visual workflow scheduler for big data pipelines

dolphinscheduler.apache.org ↗

Apache DolphinScheduler adds missed fire policy and AWS EMR Serverless integration in its 3.4.x patch series.

workflow-schedulerdata-engineeringapacheaws-integrationbatch-processing
◆Current state
DolphinScheduler 3.4.x is a mature data workflow scheduler on a quarterly patch cadence. Recent releases address real operational gaps: schedule missed fire policy handling (what happens when a job doesn't fire at its scheduled time), Amazon EMR Serverless as a new task plugin, configurable maximum runtime for workflow and task instances, and complement data dependency support. Security hygiene also landed — HTTP TRACE disabled, plaintext passwords removed from worker logs.
◆Where it's heading
The 3.4.x series is consolidating scheduler reliability (missed fire, dispatch timeout for missing worker groups, configurable max runtime) while expanding cloud integrations (AWS EMR Serverless, with prior support for SageMaker and Kubernetes from the 3.3.0 connection center work). The pattern is incremental operational hardening rather than architectural change. The 3.3.0-alpha connection center abstraction (Zeppelin, SageMaker, K8s) is the most notable structural addition in the visible history.
◆Prediction
Additional AWS or cloud provider task plugins are the most likely near-term additions given the EMR Serverless landing. Scheduler reliability work will continue as the missed fire policy and timeout logic get extended to more edge cases.

◆Recent moves

  1. 8d ago

    Release 3.4.3

    DSIP-107 adds a schedule missed fire policy, defining what the scheduler does when a job fails to fire at its intended time — a genuine operational gap for long-running data pipelines where missed fires have downstream consequences. Also in this release: plaintext passwords scrubbed from worker logs, closing a credentials-in-logs exposure that affects any team running distributed DolphinScheduler workers.

    View source ↗
  2. 3mo ago

    Release 3.4.2

    Amazon EMR Serverless joins as a new task plugin, enabling DolphinScheduler to orchestrate serverless Spark/Hive jobs on AWS without managing EMR cluster instances. Complement data dependency support (DSIP-95) fills a scheduling logic gap for backfill workflows. Real operational additions for AWS-heavy data teams.

    View source ↗
  3. 6mo ago

    Release 3.4.1

    Configurable maximum runtime for workflow and task instances prevents runaway jobs from tying up resources indefinitely — a missing operational control for production data pipelines. Dispatch timeout checking for missing or empty worker groups handles the edge case where a scheduled job can't be placed, rather than hanging silently.

    View source ↗
  4. 1y ago

    3.3.0 alpha: connection center for Zeppelin, SageMaker, and K8s

    3.3.0-alpha introduced a connection center abstraction supporting Zeppelin, SageMaker, and Kubernetes external connections — a structural addition that standardizes how DolphinScheduler integrates with external compute targets. Remote log handler abstraction (AbsRemoteLogHandler) also lands here, enabling pluggable log storage backends for distributed setups.

    View source ↗