Skip to content

Advertisement

DevOps Society
DevOpsCase Study

Nubank Cut Its Disaster Recovery Test From Four Hours to 40 Minutes by Treating Resilience as a Product

Nubank went from week-long coordination across 70 engineers to a 40-minute automated disaster recovery test that terminated 900 EKS instances and produced audit-ready evidence. How it is built.

Published your local timeupdated

Nubank Cut Its Disaster Recovery Test From Four Hours to 40 Minutes by Treating Resilience as a Product

Disaster recovery tests at banks tend to be big events. They take weeks to schedule, need a room full of teams, and produce a pile of screenshots for the regulator. Nubank, which serves more than 140 million customers in Brazil, Mexico and Colombia, has turned that event into a routine job.

The details are in an AWS Cloud Operations blog post from 28 October. AI gets the headline, but the more interesting part is how the test was redesigned before any agent got involved.

What did DR testing look like before?

Nubank runs thousands of microservices on Amazon EKS and tracks more than 300 business flows it treats as critical. Each test cycle took over four hours and pulled in product, platform and risk teams. Lining up schedules across seven or more teams took weeks. Faults were injected by hand, evidence was gathered by hand, and investigating the results took another one to two weeks. Regulators in each country set the frequency, type and scope of tests.

Engineering Director Vitor Puente describes the shift as going from coordinating 70-plus engineers to producing audit-ready documentation automatically.

How does the automated test work?

Nubank calls it One Step Execution. A single run does this:

  1. Argo Workflows orchestrates the test and its documentation.

  2. k6 generates synthetic traffic that mimics customers using the critical flows.

  3. AWS Fault Injection Service injects the fault, in parallel with the traffic.

  4. FIS rolls the fault back automatically on a time limit or when safety metrics are breached.

  5. The platform tracks how the affected systems and flows recover.

  6. A Grafana plugin captures dashboard screenshots as evidence.

  7. AWS DevOps Agent writes the report, pulling metrics, logs and alerts through MCP connections.

Each test starts from a "skill", a structured prompt that names the business flow, the fault, the time window and what good recovery looks like, such as traffic moving to a healthy zone or nodes being replaced in time. The agent then checks the signals against those expectations and flags gaps.

What were the results?

  • Test cycle time fell from about 4.3 hours to about 40 minutes.

  • Engineering effort fell 43% across nine tests in 2026, and cost per cycle fell 91%.

  • Coverage grew from about 40 business flows to more than 160.

  • One run terminated more than 900 EKS instances under synthetic traffic and produced recovery evidence for regulators.

Nubank ran one scenario in 2025 and runs three now: an availability zone going down, a whole cluster going down (replaying a 2024 incident and testing Karpenter), and data destruction in DynamoDB. Six are planned.

What can other teams copy?

  1. Make scenarios reusable. Nubank treats resilience as a product with canonical scenarios that work across countries and accounts. Write a scenario once, run it everywhere.

  2. Generate real traffic during the test. A fault with no users hitting the system proves little. Synthetic traffic on critical flows is what makes the evidence credible.

  3. Automate the rollback before the fault. Time limits and metric-based stop conditions are what make it safe to run often.

  4. Define "recovered" in writing. The skill file is a recovery spec. Even without an agent, writing down what good recovery looks like turns a test into a pass or fail.

  5. Let machines collect evidence. Screenshots and reports are where most of the human hours went. That is the easiest place to start.

The post does not describe human approval controls beyond the automatic rollback, and Nubank's next steps include scheduled runs with on-call teams notified before each fault. Anyone copying the model should decide early who signs off on a test that terminates hundreds of nodes.

Quick answers

How did Nubank speed up disaster recovery testing?

Nubank automated its DR tests with Argo Workflows, k6 synthetic traffic, AWS Fault Injection Service and AWS DevOps Agent, cutting a test cycle from about 4.3 hours to about 40 minutes.

What does AWS DevOps Agent do in Nubank's DR tests?

It reads metrics, logs and alerts through MCP connections, checks recovery against the expectations defined for each test, flags gaps and writes the report used as evidence.

What DR scenarios does Nubank test?

An availability zone becoming unavailable, a whole EKS cluster becoming unavailable, and data destruction in Amazon DynamoDB, with six scenarios planned.

Advertisement

Follow DevOps Society on LinkedIn

Practical infrastructure engineering in your feed.

Follow

Written by

DevOpsSociety Editorial Team

Editorial Team

The DevOpsSociety Editorial Team covers DevOps, cloud infrastructure, Kubernetes, AI infrastructure, platform engineering, cybersecurity, FinOps, and modern engineering practices. We publish practical insights, technical guides, architecture analysis, and research for engineers and technology leaders.

More from DevOpsSociety →
The Infrastructure Briefing

Get the infrastructure briefing.

Practical DevOps, cloud, AI infrastructure and engineering insights, delivered weekly. Read by engineers and engineering leaders.

No spam. Unsubscribe anytime.

Nubank Cut Its Disaster Recovery Test From Four Hours to 40 Minutes by Treating Resilience as a Product, DevOps Society