Most observability migrations stall for the same reason. The platform team picks a new standard, then asks every product team to re-instrument their services. The product teams have roadmaps. The migration drifts for two years and ends with both systems running forever.
Atlassian avoided that trap on a very large system. Its metrics pipeline covers roughly 100,000 hosts across 14 regions, runs to a 99.95 percent service level objective, and takes in about 4.8 billion data points a minute. For most of the past decade it ran on gostatsd, the open source StatsD server Atlassian itself maintains. In September three of its engineers wrote up how they moved it onto OpenTelemetry, the open standard for collecting metrics, traces and logs, on the CNCF blog. InfoQ covered it on 29 September.
The detail worth your attention is not the size. It is the order they did things in.
Why they moved at all
gostatsd did one job well. It received StatsD metrics over UDP, the simple fire-and-forget network protocol, and passed them on. It could not carry traces or logs, and more of Atlassian's services were already producing OpenTelemetry data. Every new feature the OpenTelemetry Collector community shipped was something Atlassian would have had to rebuild by hand to keep up.
That is the honest trigger for most teams too. Not that the old tool broke, but that the team maintaining it was quietly becoming a small vendor with one customer.
Keep the front door, rebuild the house
The central decision was to leave the interface alone. Services kept sending StatsD packets to the same address. Behind that address, Atlassian replaced everything. As the engineers put it in InfoQ's write-up, "We kept the interface and rebuilt everything behind it." That turned a company-wide migration into a platform-team migration. Thousands of services did not have to change a line on day one.
They then rebuilt the pipeline in four stages, each with its own cut-down build of the OpenTelemetry Collector:
Collection on each host, accepting both the old StatsD format and native OpenTelemetry, and folding the metrics agent into the tracing agent that was already there.
Ingest, which routes each time series to an aggregation server.
Aggregation, which rolls raw points up before storage. Atlassian wrote its own processor here because the upstream parts did not aggregate metrics the way it needed, and published it as open source.
Forwarding, a stateless gateway that sends data on to SignalFx, S3 and other destinations. Adding a new destination is now a settings change rather than a project.
Serverless functions got the same treatment. An OpenTelemetry extension for AWS Lambda replaced the old one, with the same address and settings and no code changes.
Running old and new side by side
The engineers say plainly that a migration like this runs for months, even years, with both systems live for most of it. They did not publish a start or end date, and gostatsd's aggregation layer was still due to be removed when they wrote.
What they did publish is how they kept the overlap safe. They started with development and staging workloads, and with the teams who felt the most pain from the old setup, because those people gave fast feedback and forgave rough edges. Traffic moved in steps of 1, 10, 50 and 100 percent. Less critical services went before the most critical paths. And they kept the day-to-day workflow for operators the same on both systems, so nobody had to learn two ways of doing things during the overlap.
Cost and volume control
The article contains no money figures, so any claim about dollars saved would be invented. It does contain resource numbers. Before the change, the old aggregators plus a routing layer called nomad took about 38 percent of CPU requests in the metrics clusters, with nomad alone at about 13 percent of resources.
Three changes cut that down. Merging the metrics agent into the tracing agent saved about 3.9 percent CPU per service on average across their most expensive services, which they put at roughly a 30 percent cut in agent cost across the fleet. The old router assigned work by service, so big services piled onto a few overloaded servers. The new one assigns work by individual time series, which spread the load evenly. With that and the removal of a parsing step, the aggregators used about half the CPU for the same traffic.
The aggregation stage does the heavy lifting on volume. About 4.8 billion points a minute come in and about 220 million are kept. That is roughly one point kept for every 22 received. Atlassian did not publish figures on cardinality, meaning the number of distinct time series, so we will not guess at them.
What transfers to 200 hosts
Almost none of the machinery. A team with 200 hosts does not need a dedicated ingest tier, custom aggregation code or sharding by time series. One or two collectors will likely do.
The method transfers well. Keep whatever your applications send today accepted at the same address, swap the pipe behind it, and move app instrumentation later as a separate piece of work. Atlassian is only now starting that second step.
Agent consolidation transfers too. As an illustrative example, a 200-host estate running a separate metrics agent and tracing agent on every host is running 400 agents. Merging them halves the count of things to patch, monitor and upgrade, whatever the CPU saving turns out to be.
What not to copy
Do not write your own aggregation processor because Atlassian did. They hit a gap at a volume you will probably never see. Check the standard Collector components first.
Do not plan a multi-year overlap. At 200 hosts, a quarter of side-by-side running should be plenty, and the longer two systems run, the more likely both become permanent.
Do not trust lab benchmarks alone. Atlassian's sharpest lesson was that small tests missed the real costs, and only continuous profiling in production showed where to optimise. That applies at any size.
The honest caveat
This is a team describing its own project on a foundation blog. It has no failure stories, no timeline and no spending figures. The CPU numbers are Atlassian's own measurements. Treat it as a well-argued playbook, not an audited result.
What to do this quarter
List every agent and client library that sends telemetry from your hosts. Most teams find more than they expected.
Decide which interface you will keep stable during the change, and tell product teams in writing that they do not need to act yet.
Pick two willing teams with real pain from the current setup and start there, in staging.
Turn on continuous profiling for your collectors before you trust any cost estimate.
