LoginSign Up
← Back to all sparks
W

Windmill

INFRA · APIS
Velocity7.5

Open-source developer platform to build internal tools and workflows from scripts

www.windmill.dev ↗

Windmill is quietly turning its orchestrator into a DuckLake-native data platform.

ducklakedata-pipelineslakehousedata-environmentsobservabilityorchestration
◆Current state
Windmill remains a script-and-workflow orchestrator, but its recent output has narrowed almost entirely to the data layer: DuckLake materialization, partitioned assets, freshness, and now schema contracts. The changelog reads less like a general automation tool and more like a managed lakehouse being assembled on top of the existing job engine, with most of the heavier machinery gated to Enterprise Edition.
◆Where it's heading
The direction is a governed, self-maintaining data platform: fork-scoped data environments for isolated iteration, freshness SLAs with an auto-healing watchdog, scheduled compaction and retention, range backfills, and save-time schema validation between producers and consumers. Each release hardens a different operational corner of the DuckLake stack rather than adding breadth, so Windmill is deepening into data-ops rather than widening its surface.
◆Prediction
Schema contracts shipped as non-blocking warnings, so the next likely move is to tighten that loop — enforceable contracts or richer lineage on the DuckLake graph. Continued Enterprise gating of the data-ops features (maintenance, freshness watchdog, range backfill) also points to a productized data-platform tier.

◆Recent moves

  1. 4d ago

    Fork data environments for DuckLake materialization

    ⚡ SPARK

    Merges Windmill's existing workspace-fork concept with the DuckLake data layer, giving each fork an isolated, copy-on-write data environment that defers unmaterialized tables back to the parent. This is the release that turns forks from a code-branching feature into a full data-branching one, and it anchors the platform's iteration story.

    View source ↗
  2. 4d ago

    Schema contracts for pipeline consumers

    Extends the DuckLake materialization arc into data governance: consumer scripts are now validated against captured producer schemas at save time, surfacing dropped columns and lineage breaks as editor warnings with type-aware autocomplete. It is a contracts-and-lineage layer on top of the pipeline graph, deliberately advisory rather than enforcing.

    View source ↗
  3. 5d ago

    Reasoning effort for AI agent steps

    Broadens the existing AI-agent steps with a model-adaptive reasoning-effort control spanning Anthropic, OpenAI, Azure, Gemini, Bedrock, OpenRouter, DeepSeek, and Mistral. It is a configuration and provider-coverage expansion of a capability Windmill already shipped, not a new direction — but it keeps the agent story current with extended-thinking models.

    View source ↗
  4. 5d ago

    Asset freshness with fresh/stale badge and automatic watchdog

    Adds asset-freshness monitoring to the pipeline graph — a live fresh/stale verdict from a `// freshness` annotation — plus an Enterprise watchdog that auto-reruns stale scripts with capped backoff. Fits the trajectory of turning DuckLake assets into monitored, self-healing data products rather than one-shot jobs.

    View source ↗
  5. 5d ago

    Scheduled Ducklake maintenance

    Adds scheduled maintenance for DuckLakes — snapshot expiry, small-file compaction, and orphaned-file cleanup on a cron cadence, run as observable jobs. A characteristic entry in the current arc: operational plumbing that makes the lakehouse sustainable at scale, gated to Enterprise.

    View source ↗
  6. 7d ago

    10GiB workspace storage quota replaces 50MB upload cap on Community Edition

    View source ↗