Skip to content

Advertisement

DevOps Society

Your GitOps Fleet Needs Release Promotion, Not Faster Sync

Syncing a merge to 200 clusters in an hour is easy. Knowing it is safe is not. A practical GitOps release promotion pattern for platform teams.

Published your local timeupdated

Your GitOps Fleet Needs Release Promotion, Not Faster Sync

A merge to main lands in 200 production clusters within the hour. That is GitOps working as designed, and for a large fleet it is also the problem. Faster sync does not tell you whether a release is safe to keep spreading. GitOps release promotion does.

In a piece published on 20 October, Argo CD co-creator Jesse Suen argued that fleets need a promotion layer on top of sync, with staged waves of clusters, verification gates between them and a clear record of which version runs where. Platform teams running dozens or hundreds of clusters can build this today with Argo CD Progressive Syncs, Kargo or Flux, without replacing their sync tooling.

Why does sync stop being enough?

Suen, who is also co-founder and CTO of Akuity, the company behind Kargo, described a point around 50 clusters where engineers start hesitating before they merge, even when they trust the change. His case study was a platform engineer with more than 200 single-tenant production clusters across several clouds. Every merge synced everywhere at once. Validation happened in one replica environment and took weeks, while propagation took about an hour.

Weeks to check, an hour to spread. You can see why people hesitate.

The core point is that sync and promotion are different jobs. Sync moves desired state from Git into clusters. Promotion decides whether a specific release has earned the right to reach the next group, and remembers why.

What does a fleet promotion pattern look like?

  1. Group clusters into waves. Internal or low-risk clusters first, then a small slice of production, then a larger slice, then the rest. Label clusters by wave so your tools can select them.

  2. Gate each wave on the one before. The next wave starts only when the previous one is healthy and has passed its checks.

  3. Verify with real signals. Kubernetes health is the floor. Add error rates, latency and business metrics from your monitoring, plus smoke tests.

  4. Add soak time. Some failures take hours to show. A release that looks fine after five minutes has not proven much.

  5. Keep the evidence. Store which checks a release passed in each wave, so promotion decisions rest on records rather than memory.

  6. Contain failure. When a wave fails, everything downstream stays on the previous version while you investigate.

Suen also advised testing that a failed check really does halt the release. Plenty of teams find out their gate was decorative in the middle of an incident.

Which tools handle promotion today?

Argo CD Progressive Syncs. ApplicationSets can roll out with a RollingSync strategy, made of ordered steps that select Applications by label, with maxUpdate capping how many update at once. The controller waits for every Application in a step to be Healthy before it moves on. According to the Argo CD docs, the feature has been beta since v3.3.0. It is off by default, turned on with --enable-progressive-syncs, and it disables auto-sync for generated Applications. It only watches health, so pair it with Argo Rollouts analysis if you need metric checks.

Kargo. Kargo models environments as Stages and releases as Freight. After a promotion, a Stage can run verification through AnalysisTemplates that query monitoring tools, run Jobs or call HTTP endpoints. Freight that fails verification is blocked from going further downstream, and requiredSoakTime sets a minimum stay in one Stage before the next. Manual approval skips verification, so decide up front who is allowed to use it.

Flux. Flux has no promotion controller of its own. Its documented pattern uses the notification controller to send a GitHub dispatch event when a staging HelmRelease succeeds, and a workflow then opens a pull request that bumps the production version. Change the event filter to match test success, so promotion follows passing Helm tests rather than just a finished upgrade.

Suen pointed to Rancher Fleet rollout partitions and progressive strategies in Open Cluster Management as other options that cover part of the same ground.

How do you get release visibility?

The question on-call asks during an incident is simple. Which clusters run which version, and why did the rollout stop? If answering means searching five repos and three dashboards, you do not have visibility.

At a minimum, keep one view per release: version, waves completed, checks passed and the current hold reason. Kargo shows this per Stage. With Argo CD or Flux you may need to assemble it from Application status and pull request history. Either way it should be one place a person can look at 3 a.m.

Start small. Put one early wave in front of the rest of the fleet this quarter, add a single metric gate and prove that it blocks. The other pieces can follow once people stop hesitating before they merge.

Quick answers

What is GitOps release promotion?

It means moving a specific release through ordered groups of clusters and advancing only when the previous group passes health and verification checks. Git still holds the desired state.

Does Argo CD support staged rollouts across clusters?

Yes. ApplicationSet Progressive Syncs with the RollingSync strategy roll out in ordered steps and wait for each step to be Healthy. The feature is beta since v3.3.0 and off by default.

How does Kargo stop a bad release?

A Stage runs verification after each promotion. Freight that fails is blocked from downstream Stages, and requiredSoakTime can force a minimum wait before the next promotion.

Advertisement

Follow DevOps Society on LinkedIn

Practical infrastructure engineering in your feed.

Follow

Written by

DevOpsSociety Editorial Team

Editorial Team

The DevOpsSociety Editorial Team covers DevOps, cloud infrastructure, Kubernetes, AI infrastructure, platform engineering, cybersecurity, FinOps, and modern engineering practices. We publish practical insights, technical guides, architecture analysis, and research for engineers and technology leaders.

More from DevOpsSociety →
The Infrastructure Briefing

Get the infrastructure briefing.

Practical DevOps, cloud, AI infrastructure and engineering insights, delivered weekly. Read by engineers and engineering leaders.

No spam. Unsubscribe anytime.

Your GitOps Fleet Needs Release Promotion, Not Faster Sync, DevOps Society