Skip to content

Advertisement

DevOps Society

Internal Developer Platform: Architecture, Tools, and Best Practices

Internal developer platform architecture: the control plane, service catalog, golden paths, and CI/CD glue, plus the team model that keeps it running.

Share

Published your local timeupdated

Internal Developer Platform: Architecture, Tools, and Best Practices

The decision has already been made. Someone senior read the Gartner line, the ops queue is four weeks deep, and you have been handed a mandate and two engineers to build an internal developer platform. Now comes the part nobody writes about: which layers you actually build, which you buy, what the control plane should be, and how to keep the abstraction from becoming a wall your best teams have to climb over. This is the architecture companion to the broader case for treating the platform as a product — less about why, entirely about how.

Most platforms that fail do not fail on technology choice. They fail because the team built the portal first, modelled exactly one service shape, and discovered in month seven that the abstraction has no exit.

The internal developer platform reference architecture

Strip the vendor diagrams down and every working platform has the same four pieces. Three are stacked; one runs sideways through all of them.

The developer interface is whatever a team touches to express intent: a Git commit, a CLI invocation, a template form, an IDE plugin. It carries no logic of its own. Its only job is to turn a human wish into a validated, versioned declaration of desired state.

The orchestration layer, or control plane, is where that declaration meets reality. It resolves the request against environment context, applies defaults, expands one short manifest into the forty resources it implies, and reconciles the result. This is the layer that makes a platform a platform rather than a wiki page with good intentions.

The infrastructure plane is the actual substrate: clusters, managed databases, queues, DNS, load balancers, IAM. You are almost certainly not building this. You are consuming your cloud provider's version of it and deciding how much of its surface to expose.

Observability and policy run across all three. They are not a layer you bolt on at the end; they are properties the other layers either have from day one or never acquire.

Four-layer platform architecture with observability and policy spanning every layer

The architectural mistake that costs the most is putting logic in the interface layer. Once the portal knows how to build a service, the CLI and the Git path either duplicate that knowledge or stop working. Every interface must be a thin client over the same control plane.

Build or buy, layer by layer

Layer

Recommendation

Common tools

What goes wrong

Developer interface (portal)

Buy, or defer entirely

Backstage, Port, Cortex, OpsLevel

Built first, becomes a read-only dashboard nobody opens; logic leaks into the UI

Service catalogue & templates

Buy the engine, own the templates

Backstage scaffolder, Port blueprints, cookiecutter

Catalogue rots because it is maintained by hand rather than derived from Git

Control plane / orchestration

Build on an existing engine — never from scratch

Crossplane, Terraform/OpenTofu + CI, Argo CD, Humanitec

Bespoke API becomes a permanently staffed product with two users

Infrastructure plane

Buy

EKS/GKE/AKS, RDS, managed Kafka, Cloud SQL

Wrapping every managed service in a house abstraction that lags the provider

CI/CD

Buy, standardise on one

GitHub Actions, GitLab CI, Argo Workflows, Jenkins

Per-team pipeline sprawl; platform ends up owning 200 bespoke YAML files

Policy

Buy the engine, own the policies

Kyverno, OPA Gatekeeper, Conftest

Enforced only in CI, so anything applied directly to the cluster bypasses it

Secrets

Buy

Vault, AWS/GCP secret managers, External Secrets Operator

Developers handed raw secret-store access, which undoes the abstraction

Observability defaults

Build the wiring, buy the backend

OpenTelemetry, Prometheus, Grafana, vendor APM

Dashboards exist but nothing generates them per service automatically

The pattern: build the glue and the opinions, buy the engines. A platform team's durable value is the set of decisions encoded in templates, compositions and policies, not the reconciliation loop underneath them.

The control plane choice

This is the one decision that is genuinely expensive to reverse, because everything else hangs off it. Three credible options.

Crossplane and the Kubernetes Resource Model

Crossplane extends the Kubernetes API so that cloud infrastructure becomes custom resources reconciled by controllers in a cluster. You define a Composite Resource Definition — your own API, in your own vocabulary — and a Composition that expands it into the underlying managed resources. It graduated in the CNCF in November 2025, which matters mostly as a signal about governance and maintenance load rather than about the code.

What you get is continuous reconciliation rather than periodic apply, drift correction without a pipeline run, and a published API developers can discover with kubectl explain. Because it is built on the Kubernetes Resource Model, everything else that speaks that model (Argo CD, admission policy, RBAC, your existing GitOps flow) works against your infrastructure APIs for free.

Check the version before copying any older example. Crossplane v2 made composite resources namespaced and removed claims entirely, so the v1 pattern of a claim creating a cluster-scoped XR is gone; developers now write the XR directly, in their own namespace.

# Committed to the service repo; reconciled by Crossplane in the team's namespace.
apiVersion: platform.example.com/v1alpha1
kind: PostgresInstance
metadata:
  name: checkout-db
  namespace: team-checkout
spec:
  size: small                 # maps to instance class + storage in the Composition
  environment: production     # selects the Composition; drives backup and HA defaults
  compositionRef:
    name: postgres-aws-multi-az

Four fields. The Composition behind them resolves to a subnet group, a parameter group, an instance, an IAM role, a secret written to the namespace and a scrape config. That ratio, five lines in and forty resources out, is the whole point.

The cost is real. You now operate a control plane cluster as tier-zero infrastructure: if it is down, nobody provisions anything. Composition authoring has a steep learning curve, and debugging one means tracing status conditions through several layers of generated resources. Provider coverage lags the cloud APIs. Budget an engineer who genuinely knows it, not a rotation.

Terraform or OpenTofu modules behind a pipeline

The conservative choice, and for many organisations the correct one. Developers open a pull request against a repo containing a small module invocation; the pipeline plans, gates, and applies. Your platform is a module library plus a pipeline plus a review policy.

Everyone already knows the tooling, plans are reviewable, and the blast radius of a bad change is visible before it lands. If your team's muscle is Terraform and your CI platform is already settled, this gets you a working self-service flow in weeks rather than quarters.

What you give up is reconciliation. Terraform converges when the pipeline runs and not otherwise, so drift persists until someone notices, and state becomes an operational object you protect, lock and occasionally repair by hand. The request queue also tends to reappear in a new costume: if every module change needs platform-team review, you have rebuilt the ticket system in HCL.

A bespoke API

Sometimes justified. If your provisioning logic has to span systems with no common model — a mainframe, a colo, three clouds and an internal billing system — a purpose-built service that owns the workflow can be the only thing that fits.

Be honest about what you are signing up for: authentication, authorisation, an API contract, versioning, a state store, a reconciliation loop, an upgrade story and an on-call rotation, serving an audience of maybe two hundred engineers. Most teams choosing this path underestimate it by a factor of three. If you cannot name the specific capability that Crossplane and Terraform both fail to provide, you do not need it.

The decision rule. If your workloads run predominantly on Kubernetes and you have an engineer who can own composition authoring, use Crossplane. If your estate is mixed, your depth is in HCL, or you need something working this quarter, use Terraform or OpenTofu modules with a pipeline, plus Argo CD for the application layer. Build a bespoke API only when you can write down the requirement neither satisfies. A commercial platform orchestrator such as Humanitec is worth pricing when you want the workload-to-infrastructure mapping as a product rather than as something you maintain.

The catalogue and software templates

A service catalogue is the platform's index of what exists, who owns it, what it depends on and where its runbook lives. Backstage is the default — CNCF incubating since March 2022, and the de facto schema even for teams running something else, since Port and Cortex both ingest catalog-info.yaml. Templates are the other half: a software template scaffolds a repository already wired into the pipeline, ownership metadata, base image, health endpoints, dashboards and alert routing.

# backstage/templates/http-service/template.yaml (excerpt)
spec:
  parameters:
    - title: Service
      required: [name, owner, tier]
      properties:
        tier:
          type: string
          enum: [tier-1, tier-2, tier-3]   # drives SLO targets and pager policy
  steps:
    - id: scaffold
      action: fetch:template
      input:
        url: ./skeleton
        values: { name: '${{ parameters.name }}', tier: '${{ parameters.tier }}' }
    - id: register
      action: catalog:register        # catalogue entry created at scaffold time, not later
      input:
        catalogInfoPath: /catalog-info.yaml

Two failure modes here, both common enough to plan around.

The first is the golden path that covers one service shape. You model the stateless HTTP service in Go because that is what the platform team knows, ship it, and then discover the organisation is mostly shipping Python data jobs, event consumers and a growing pile of model-serving workloads with GPU scheduling requirements of their own. Adoption stalls, and the teams you missed build their own thing permanently. Before writing a template, count what has actually been created in the last year and cover the two most common shapes.

The second is catalogue rot. A hand-maintained catalogue is wrong within a quarter: services get renamed, teams reorganise, ownership moves and nobody updates the entry. The fix is structural: metadata lives in the repo, registration happens at scaffold time, and drift gets flagged by a scheduled reconciler that opens a pull request rather than a ticket. If catalogue accuracy depends on someone remembering, it is already stale.

Environments, and the ephemeral ones

Provisioning is where the control plane earns its keep. The baseline: a team gets a namespace, its dependencies, its secrets and its ingress from one declaration, with no human in the path.

Ephemeral environments, one per pull request and torn down on merge, are the highest-value capability most platforms skip and the easiest to get wrong financially. They work when the workload is stateless and its dependencies are cheap to clone. They get expensive fast when every PR environment provisions its own managed database, and they become a support burden when seeding realistic data is manual.

Three rules keep them viable: enforce a hard TTL with a controller rather than a convention; share expensive backing services across PR environments using per-PR logical isolation instead of per-PR instances; and make teardown a property of the environment's lifecycle, not a pipeline step that gets skipped when a job fails.

Pull request environment created by the control plane and reclaimed on merge or timeout

Interfaces: Git first, CLI second, portal last

The order matters, and most teams get it backwards because the portal is what demos well.

Git should be the primary interface. It gives you review, audit, rollback and access control that already exist and that auditors already accept. A manifest in a repo, reconciled by a controller, needs no additional plumbing to be safe.

A CLI comes second, for what Git is bad at: scaffolding, one-off queries, local development loops, and "what is the state of my thing right now". A good platform CLI mostly generates and validates the files that then go into Git.

The portal comes last, on top of a control plane already working through Git and the CLI. Built first, it becomes a form writing to a system nobody else can write to, which makes the portal itself the abstraction and forces every capability to be re-implemented inside it. Built last, it is a discovery surface — what exists, who owns it, what its scorecard says, where the docs are — which is what developers actually want from it. The adoption argument for portals is covered elsewhere in our platform engineering writing; the architectural argument is simply that thin clients over a shared control plane are cheap and thick ones are not.

Policy, enforced where it cannot be skipped

Policy as code belongs in admission control, not only in the pipeline. CI-only enforcement means anything applied directly to the cluster — an incident fix at 3am, a controller with broad RBAC, a kubectl apply from a laptop — walks straight past your rules.

Kyverno and OPA Gatekeeper are both reasonable. Kyverno, a CNCF graduated project since March 2026, expresses policies as Kubernetes resources in YAML, so anyone who writes manifests can read them. Gatekeeper uses Rego: more expressive, with a real learning curve. Choose it when your policies need logic YAML matching cannot express.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-ownership
spec:
  validationFailureAction: Enforce   # Audit first for a release, then flip to Enforce
  rules:
    - name: owner-label
      match:
        any: [{ resources: { kinds: [Deployment, StatefulSet] } }]
      validate:
        message: "Set app.kubernetes.io/owner to a team in the catalogue."
        pattern:
          metadata:
            labels:
              app.kubernetes.io/owner: "?*"

Run everything in audit mode for a release cycle before enforcing. Shipping a new policy straight to Enforce is the fastest way to have the platform team blamed for a production outage in week one.

Secrets get the same treatment: developers should never hold credentials for the secret store. The pattern that holds up is a declarative reference in the repo, resolved at runtime by External Secrets Operator or the provider's CSI driver, with the platform owning the trust between workload identity and the store. If onboarding explains how to log into Vault, the abstraction has already leaked.

The abstraction problem

Every platform abstraction is a bet that you can hide complexity without hiding capability. You will lose that bet somewhere; the design question is what happens when you do.

The failure mode is specific. A team needs something your abstraction does not express — a sidecar, an unusual scheduling constraint, a storage class you never modelled, a node group configuration your Kubernetes layer doesn't surface. They file a request. You are mid-quarter. They wait two weeks, then fork the module, and now there are two platforms: yours, and the one the strongest team maintains in private. Once that fork exists it never merges back, and every subsequent team copies the fork because it has more features.

Prevent it with an escape hatch designed in from the start, not bolted on after the first fork.

  • Passthrough fields. Let a high-level resource carry an explicit block of low-level overrides that the composition merges rather than validates. Ugly, and far better than a fork.

  • Graduated exit. Make it possible to take the generated output — the rendered manifests, the underlying Terraform — and own it directly, while keeping the shared CI, policy and observability layers. Leaving the abstraction should not mean leaving the platform.

  • A named path for exceptions. Someone on the platform team owns "this doesn't fit" requests with a stated response time. The absence of that path is what makes forking rational.

Treat every escape-hatch use as a product signal. Three teams reaching for the same override means you found a missing feature, not three difficult teams.

A one-way-door abstraction forcing a permanent fork compared with a graduated escape hatch

What to build first

Ignore the reference architecture for the first ninety days and build in this order.

  1. Pick the two service shapes that actually get created, from last year's repo history, and write templates for them. Nothing else until a real team ships one.

  2. Stand up the control plane for one resource type — usually a database or a namespace with ingress — end to end, through Git. One resource done properly teaches you more than five done shallowly.

  3. Wire the catalogue at scaffold time so it is correct by construction from the first service.

  4. Put policy in audit mode, publish what will eventually be enforced, and give teams a release cycle to comply.

  5. Add the portal when the control plane is boring, and only as a read surface plus a template launcher.

The checkpoint at day ninety is not how many capabilities exist. It is whether a team outside the platform group has shipped something to production on the paved road without asking you a question. If not, you have built infrastructure rather than an internal developer platform, and more features will not fix it.

Advertisement

Follow DevOps Society on LinkedIn

Practical infrastructure engineering in your feed.

Follow

Written by

DevOpsSociety Editorial Team

Editorial Team

The DevOpsSociety Editorial Team covers DevOps, cloud infrastructure, Kubernetes, AI infrastructure, platform engineering, cybersecurity, FinOps, and modern engineering practices. We publish practical insights, technical guides, architecture analysis, and research for engineers and technology leaders.

More from DevOpsSociety →
The Infrastructure Briefing

Get the infrastructure briefing.

Practical DevOps, cloud, AI infrastructure and engineering insights, delivered weekly. Read by engineers and engineering leaders.

No spam. Unsubscribe anytime.

Internal Developer Platform: Architecture, Tools, and Best Practices, DevOps Society