Incident replay — a couple of characters pruned the world (Skyscanner)

The real incident

Skyscanner, 24 August 2021. A change described internally as a no-op edited a configuration file at the root of the "cells" tree — the file that carries regional configuration to every cell, everywhere. Two characters went missing: the template's {{ }} delimiters.

Source: Skyscanner — How a couple of characters brought down our site

Without the delimiters the templating system produced nothing usable. That file was rolled out globally and immediately — the very property that made it useful for regional config. Argo CD then did its one job: it compared desired state (now: no valid namespaces, no services) with live state, and reconciled the difference by deleting 478 services across every namespace, availability zone and region on earth. skyscanner.net and the mobile apps were down for four and a half hours; full recovery across all regions took two more days.

The Cells Architecture existed precisely to contain regional failure. It could not contain a central config failure, because every cell was faithfully obeying the same corrupted instruction. Automation does not distinguish "you asked for nothing" from "you asked me to delete everything."

This cluster, right now

kubelings runs the same shape, three cells small: a reconciler script renders one Service per region from cells-template, then applies it with --prune, so anything it manages but no longer renders is removed.

The regions line already lost its delimiters, and the reconciler has already run:

kubectl -n kubelings get svc -l app.kubernetes.io/managed-by=cells   # nothing left
kubectl -n kubelings get configmap cells-config -o jsonpath='{.data.regions}{"\n"}'
kubectl -n kubelings get configmap cells-reconciler -o jsonpath='{.data.run\.sh}'

Running a reconcile is how the "agent" is invoked here — it reads the script out of the ConfigMap, so your edits take effect on the next run:

kubectl -n kubelings get configmap cells-reconciler -o jsonpath='{.data.run\.sh}' > /tmp/run.sh
NS=kubelings CONFIG_CM=cells-config sh /tmp/run.sh

 ┌───────────────────┐
 │config loses {{ }} │
 │                   │
 └───────────────────┘
         │            
rolled out globally   
         │            
         ▼            
 ┌────────────────┐   
 │render is empty │   
 │                │   
 └────────────────┘   
         │            
   apply --prune      
         │            
         ▼            
  ┌─────────────────┐ 
  │all cells pruned │ 
  │                 │ 
  └─────────────────┘ 

Your task

Bring the site back, then make this class of change survivable:

1. Restore the region list in cells-config — the three cells are eu, us, ap — and reconcile so web-eu, web-us, web-ap exist again.
2. Make the reconciler fail closed: patch run.sh so a render missing its regions refuses to apply at all rather than pruning. Refuse before the apply, not after.
3. Leave the reconciler's --prune in place. Pruning is not the bug — pruning an empty render is.

The check proves the guard by running the reconciler twice: once with the good config (must succeed, cells intact) and once against a frozen copy of the broken one, cells-config-broken (must fail, cells intact). A guard that refuses everything is not a guard — it's an outage with better manners.

kubectl -n kubelings patch configmap cells-config --type=merge \
  -p '{"data":{"regions":"eu us ap"}}'

kubectl -n kubelings edit configmap cells-reconciler

# fail closed: an empty/short render is a bug, never a deletion order
want=$(printf '%s\n' $regions | grep -c . || true)
have=$(grep -c '^kind: Service' "$rendered" || true)
if [ "$want" -lt 3 ] || [ "$have" -lt "$want" ]; then
  echo "refusing to reconcile: rendered $have Services for $want regions" >&2
  exit 1
fi

kubectl -n kubelings get configmap cells-reconciler -o jsonpath='{.data.run\.sh}' > /tmp/run.sh
NS=kubelings CONFIG_CM=cells-config        sh /tmp/run.sh   # must succeed
NS=kubelings CONFIG_CM=cells-config-broken sh /tmp/run.sh   # must refuse, delete nothing
kubectl -n kubelings get svc -l app.kubernetes.io/managed-by=cells

# Which Argo apps can delete for you today?
kubectl get applications -A -o json | jq -r '.items[]
  | select(.spec.syncPolicy.automated.prune == true)
  | "\(.metadata.name)  allowEmpty=\(.spec.syncPolicy.automated.allowEmpty // false)"'