A Zero-Downtime Kubernetes Migration Runbook That Survives Contact With Production

Migrations do not fail on Kubernetes, they fail on the cutover

I have been brought into a few Kubernetes migrations after a previous attempt failed, and the previous attempts had a shared structure: a maintenance window, a DNS change, and hope. When something went wrong at 03:40, the only available action was a full rollback under pressure — the exact moment when people make the worst decisions of the project.

The alternative is not a better maintenance window. It is not having one. You run both environments live, you move traffic between them in increments small enough that a mistake is invisible to customers, and every increment can be undone by changing a number. That is the whole idea behind the zero-downtime Kubernetes migration engagement on this site, and it is what turns a terrifying event into a dull afternoon.

This is the runbook, in the order I actually run it.

Phase 0: decide what you are not migrating

Before anything is codified, cut scope. The most common way a migration overruns is by trying to modernise everything at once — new cluster, new CI, new observability stack, new secrets management, new service mesh, all live at the same time. When something breaks you now have six suspects.

Write down explicitly:

• Services in scope, in the order they will move. Start with the least critical stateless service that still gets real traffic. Not a toy — real traffic is what proves the pattern.
• Services out of scope for now, especially databases. Moving compute to Kubernetes while the database stays exactly where it is, is a legitimate and much safer end state for phase one.
• Changes explicitly deferred: language runtime upgrades, framework bumps, refactors. The container that runs in Kubernetes should be as close as you can make it to the process that runs on the VM today.

Everything you defer here is a variable you do not have to control during the cutover.

Phase 1: make the target reproducible before it is important

Build the cluster and its networking in code from the beginning, not “for real later”. A cluster that was clicked together in a console cannot be rebuilt when you need it rebuilt, which is always at the worst time.

# The whole environment described once, reviewable in a pull request.
module "cluster" {
  source = "./modules/eks"

  name               = "prod"
  kubernetes_version = "1.31"
  vpc_id             = module.network.vpc_id
  private_subnets    = module.network.private_subnets

  # Same instance families as the VMs being replaced, so the first
  # comparison is workload-to-workload, not hardware-to-hardware.
  node_groups = {
    general = { instance_types = ["m6i.xlarge"], min = 3, max = 12 }
  }
}

Two things belong in this phase and are usually postponed to everyone’s regret. First, GitOps from day one — Argo CD or Flux, so that cluster state matches git and every change during the migration is a reviewable, revertible merge rather than an kubectl apply someone ran from a laptop and cannot remember. Second, observability before workloads: metrics, logs and traces flowing from the new cluster into the same dashboards you already watch, with the same names. Comparing two environments during a traffic shift is impossible if their telemetry does not line up.

Phase 2: run one service in parallel, serving nothing

Deploy the first in-scope service to the new cluster and give it zero production traffic. It should be:

• Reachable through the new ingress path, on a hostname only you know.
• Connected to the same databases, caches and queues as the current production instance. This is the part people balk at and it is essential — if the new environment talks to a copy of the data, you are not testing the thing you will cut over to.
• Emitting metrics under the same names, tagged with an environment label so you can graph old and new side by side.

Then drive real load at it. Replay production traffic if you can, synthesise it if you cannot, and let it run overnight. What you are looking for is not “does it respond” — it will — but the second-order differences: connection pool behaviour, DNS resolution inside the cluster, TLS chains, egress IPs, file descriptor limits, timezone and locale in the container, and whether anything was quietly depending on a file on the VM’s local disk.

That last one catches people constantly. Grep for absolute paths before you go looking for exotic causes.

Phase 3: the network path is the actual project

Compute is easy. The network path is where the outage lives. Work through each of these deliberately.

Weighted routing. You need a layer that can send a controllable fraction of traffic to each environment. In descending order of how much I like them mid-migration: an external load balancer or CDN with weighted target groups; an ingress controller with weight annotations; DNS weighting.

DNS is last for a reason. TTLs are advisory, resolvers cache aggressively, and clients pin. DNS-based shifting means you cannot roll back in seconds, only in “however long the worst-behaved resolver takes”, which defeats the point. Use DNS to move to the new front door before the migration, then do the actual shifting behind it.

# nginx ingress: canary by weight. The canary Ingress carries the annotations;
# the stable Ingress is untouched, which is what makes rollback a one-line change.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "5"
spec:
  ingressClassName: nginx
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: api
                port:
                  number: 8080

If you are standing this up new rather than reusing an existing ingress, prefer a Gateway API HTTPRoute with weighted backendRefs — weighted splitting is part of the specification there rather than an annotation pair, and the community ingress-nginx controller has been unmaintained since March 2026. Ingress vs Gateway API covers the choice.

Egress identity. If partners, payment providers, or a customer’s firewall allowlist your current outbound IPs, the new cluster’s egress must be allowlisted before any traffic shifts. Give the cluster a NAT gateway with static IPs and get them approved early — this has weeks of lead time at some organisations and it is invisible until it fails.

Internal service discovery. Services on the VMs that call each other by hostname need to reach their counterparts regardless of which side they are on. During parallel running, point both environments at the same stable endpoints rather than at each other’s internals.

Private access. If your existing operational access is a VPN appliance with a public listener, the migration is a good moment to fix that. Replacing it with a zero-trust mesh removed a public attack surface in an afternoon in the VPN to mesh engagement, and it means cluster access does not depend on a box nobody has patched.

Phase 4: state, sessions and the things that do not move

Stateless services shift trivially. Everything below needs a decision before the first percent of traffic moves.

Sessions. If sessions live in application memory, a user whose next request lands on the other environment gets logged out. Move sessions to a shared store (Redis, the database, or signed cookies) before migrating, as a separate change with its own release. Do not do it during the shift.

Local disk. Uploads, caches, and generated files written to a local path do not exist on the other side. Move them to object storage first, again as a separate change.

Sticky routing. If sessions genuinely cannot be shared in time, use consistent hashing at the routing layer so a given user stays on one side. It works, and it slows down your ability to roll back, because reverting weight does not immediately move pinned users.

The database. My strong preference is to not move it in the same project. Keep the database where it is, point both environments at it, and migrate compute. If the database must move too, that is a separate project with its own replication, cutover and rollback plan, executed either well before or well after the compute migration — never during.

Cron and background jobs. These are the most-forgotten source of double execution. When both environments are live, a scheduled job may run twice: once on each side. Disable schedulers on the old side, or gate every job behind a leader lock, before the first traffic shift.

Phase 5: agree the rollback triggers before you need them

Write these down, in a document everyone on the call can see, before any traffic moves. Concretely:

• Error rate: shift back if 5xx rate on the new side exceeds the old side’s baseline by more than an agreed margin, sustained for more than an agreed window.
• Latency: shift back if p99 on the new side exceeds the old side’s p99 by more than an agreed margin, sustained.
• Anything customer-visible: shift back immediately, discuss afterwards.
• Anyone on the call is uncomfortable: shift back. Cost of a rollback here is a few minutes, and the trust it builds is what gets you permission for the next increment.

The point is not the specific thresholds — they differ per system. It is that a threshold decided calmly in advance is a decision, and a threshold argued about during an incident is a fight.

Phase 6: turn the dial

Now the boring part, which is how you know it is working.

1. 1% for at least one full business cycle. Look at error rate, p99, and the specific business metric that would notice — checkout completions, messages sent, jobs enqueued. One percent is enough to expose configuration errors and not enough to hurt.
2. 5%, hold for a peak period. This is where connection-pool and concurrency differences show up.
3. 25%, hold overnight. Batch jobs, nightly crons and slow leaks appear here.
4. 50%, hold a full day. Both sides carrying real load reveals shared-dependency limits: database connection counts, third-party rate limits, queue throughput.
5. 100%, but leave the old environment running and warm.
6. Hold at 100% for at least a week before decommissioning anything. The whole value of the parallel run is that the previous environment remains a one-line rollback until you are genuinely confident.

At each step, the question is not “is it working” but “is it identical”. Graph old and new on the same axes. Differences that look like noise at 1% become incidents at 50%.

Phase 7: decommission on purpose

The migration is not finished when traffic is at 100%. It is finished when the old environment is gone and nothing quietly depended on it. Before you delete anything:

• Confirm no traffic has reached the old side for the hold period. Check access logs, not assumptions.
• Check for egress from the old environment, not just ingress to it. Cron jobs, webhook senders, and batch exports are invisible in an ingress graph.
• Snapshot the old machines and keep the snapshots for a defined period.
• Remove the old side from monitoring last, so a surprise request still alerts someone.
• Delete the parallel-run scaffolding — canary Ingress objects, environment labels, temporary firewall rules — so the next person does not inherit a half-migrated-looking system.

Then write the pattern down. The real deliverable of a first migration is not one service on Kubernetes; it is a repeatable procedure the team can run themselves for the next twelve services, which was the most durable outcome of the migration case study above.

The one-page version

1. Cut scope. Defer every change that is not “same workload, new platform”.
2. Terraform the cluster, GitOps from day one, observability before workloads.
3. Run one service in parallel against the same data, with zero traffic, under synthetic load.
4. Solve the network path: weighted routing that is not DNS, egress IPs allowlisted, service discovery stable.
5. Fix sessions, local disk, and duplicate cron execution before any traffic moves.
6. Agree rollback triggers in writing.
7. Shift 1 → 5 → 25 → 50 → 100%, holding through a peak and an overnight at each stage.
8. Hold at 100% for a week with the old environment warm.
9. Decommission deliberately, checking egress as well as ingress.

None of this is clever. It is just ordered so that at every point, the worst thing that can happen is you turn a number back down. If you want this planned against your actual stack, or an extra pair of hands on the cutover itself, that is DevOps and platform engineering work, and the routing layer is API gateways and networking — where the ingress 502 vs 503 vs 504 guide covers what those errors mean when a shift goes wrong.