Introduction: What If You Could Test Outages Like Airplanes?

Before a new airplane ever carries a passenger, it spends a long time in a wind tunnel. Engineers learn how it behaves under turbulence, stress, and edge conditions—safely, predictably, and cheaply.

Software systems deserve the same treatment.

Chaos engineering gave us the mindset: deliberately experiment on systems to build confidence in their ability to withstand real‑world failures. But in practice, many teams jump straight from theory into running chaos experiments in staging or even production—often without rehearsed playbooks, clear roles, or tested user interfaces.

There’s a missing layer between “we should be more resilient” and “let’s inject failures into production.” That layer is analog incident story wind tunnels: low‑tech, paper‑based simulations that let you prototype outages before real users feel the turbulence.

From Chaos Engineering to Analog Wind Tunnels

Chaos engineering focuses on:

• Exposing weaknesses before they become disasters
• Practicing response under realistic conditions
• Building confidence that the system can handle failure

Analog incident wind tunnels bring these goals one level earlier—before code is shipped, before dashboards are built, before on‑call rotations are live.

Instead of starting with live systems, you start with:

• Hand‑drawn screens for dashboards and tools
• Paper workflows for escalation and communication
• Story-based outages written on index cards or sticky notes

By staying analog and low‑fidelity, you can:

• Prototype how incidents should work
• Reveal design flaws in your tools and processes
• Iterate cheaply and quickly

Only then do you automate, codify, and operationalize.

What Is an “Analog Incident Story Wind Tunnel”?

Think of it as a tabletop exercise crossed with a UX paper-prototyping session, focused specifically on outages and incident response.

You simulate a failure scenario from detection to resolution using only paper artifacts:

• Rough sketches of monitoring dashboards
• Fake chat windows and ticketing systems
• Paper runbooks and playbooks
• Index cards representing alerts, customer reports, and system states

The team “plays through” the outage end-to-end, moving paper around the table like air moving over a model wing.

The goal is not to test the technology itself. It’s to test:

• Interfaces – What do people see and click first?
• Workflows – Who talks to whom, in what order?
• Decision-making – How do we choose next actions under uncertainty?
• Communication – What do we say to customers, and when?

You’re building a story of the incident—step by step, interaction by interaction—and using paper as your modeling clay.

Why Paper? The Power of Low-Fidelity Simulations

Paper feels almost too simple in a world of distributed tracing and real-time observability. But that simplicity is what makes it powerful.

1. It’s incredibly cheap and fast
With hand‑drawn designs, you can:

• Sketch a new dashboard layout in minutes
• Redesign an escalation flow with one arrow change
• Throw away bad ideas without sunk-cost pain

No back-end changes. No tickets. No deployment windows.

2. It lowers the psychological stakes
People are more willing to criticize a messy sketch than a polished UI. That makes it easier to:

• Question assumptions: “Why is this alert here?”
• Redesign flows: “This should page us earlier.”
• Explore alternatives: “What if this went to a runbook instead?”

3. It keeps focus on the human system
Most incidents are not purely technical; they’re socio-technical. The system includes:

• The code and infrastructure
• The people responding
• The tools they use
• The communication patterns they follow

Paper prototypes center the humans and workflows, not just the machines.

How to Build Your Own Incident Story Wind Tunnel

You don’t need much to start:

• Printer paper or sticky notes
• Markers and pens
• Index cards
• A whiteboard or table

Then follow a simple structure.

1. Define the Scenario

Pick a realistic outage or near-miss:

• “Primary database is slowly degrading and eventually becomes unavailable.”
• “Authentication service latency spikes; users can’t log in intermittently.”
• “A bad configuration rollout takes down part of the API.”

Write the scenario on a card. Add a few beats—time-based events:

• T+0: Silent failure begins
• T+5: Error rate increases
• T+10: Users start complaining on social media
• T+15: On‑call gets an alert

These become the “wind gusts” in your tunnel.

2. Sketch the Interfaces and Artifacts

On separate sheets, hand‑draw:

• Monitoring dashboards
• Alert notification screens
• Ticketing or incident command tools
• System diagrams
• Runbook pages

Don’t aim for beauty; aim for clarity. You want just enough fidelity that someone can point and say:

“At this moment, I would click here and look at this graph.”

3. Assign Roles

Bring a small cross-functional group:

• On‑call engineer
• Incident commander (or whoever usually leads)
• Product/feature owner
• Maybe a customer support or comms representative

Assign each person their real-world role. One facilitator acts as:

• The “system” (revealing new events)
• The narrator (keeping time moving)
• The scribe (capturing insights)

4. Run the Simulation

Walk through the scenario in real time or slightly accelerated.

The facilitator reveals events:

• New alert card: “High error rate on /checkout”
• User report card: “Can’t place order; page keeps spinning.”
• Monitoring card: “CPU OK, DB connections at 90%.”

Participants respond using only the paper tools you’ve provided:

• They “open” dashboards by pulling the right sheet
• They “page” people by moving escalation cards
• They draft customer updates on sticky notes

Your job as a group is to treat the paper as if it’s real and see where it fails you.

5. Capture Gaps and Frictions

Every time someone says:

• “I wish I could see X here.”
• “Where would I find who owns this service?”
• “We’d probably waste time checking Y first.”

…pause. Write that down. That’s turbulence.

Examples of what you might uncover:

• No clear owner for a critical dependency
• Dashboards that optimize for normal operation, not triage
• Confusing handoffs between engineering and support
• Missing decision points: “At what error rate do we pull the feature flag?”

These are exactly the weaknesses you want to surface before you introduce real failures.

What Paper Reveals That Tools Alone Don’t

Paper incident wind tunnels are especially good at surfacing gaps in three areas.

1. Monitoring: Are We Seeing the Right Things?

Questions that often arise:

• Would we detect this failure early enough?
• Which metrics matter first during triage?
• Do we have a single place that tells us the story of user impact?

If your hand‑drawn dashboard requires a dozen scribbled annotations to be useful, that’s a design signal: you need better observability for incidents, not just normal operations.

2. Communication: Who Knows What, When?

You can quickly see whether your communication patterns are resilient:

• How long before support knows there’s an incident?
• How do customers get updates: status page, email, in-product banner?
• Who is authorized to post externally, and based on which triggers?

Playing this out on paper makes vague assumptions painfully obvious.

3. Decision-Making: Can We Make Good Calls Under Pressure?

In a real incident, hesitation and confusion cost you minutes. Paper simulations let you test:

• Do we have clear criteria for rollback vs. roll forward?
• Do runbooks contain actual decisions, not just commands?
• Who has final say when engineers disagree on next steps?

Because the stakes are low, people are more willing to challenge fuzzy decision boundaries and refine them.

From Paper to Push-Button: Toward Repeatable Incident Response

One of the core goals of mature incident management is to make response:

As close to a repeatable, push‑button workflow as possible.

Not robotic or inflexible—but predictable and codified where it matters.

Analog wind tunnels are a safe place to iterate toward that goal:

1. Start with stories
   Tell the story of an outage from first symptom to final resolution.

2. Prototype the workflow on paper
   Draw the screens, the alerts, the decisions, the communications.

3. Refine through multiple passes
   Each run uncovers friction you can smooth away.

4. Then automate the stable parts
   Once the path feels right on paper, encode it:

   • Alert routing rules
   • Default dashboards
   • ChatOps commands
   • Standard status page templates

Instead of automating whatever happens to exist today, you’re automating a designed, tested incident experience.

Getting Started: A Simple First Exercise

If this feels abstract, try this lightweight starter:

1. Book 90 minutes with 3–5 people who have been in a recent incident.
2. Pick one memorable incident and reconstruct it:
   • Timeline on sticky notes
   • Key decisions on index cards
   • Interfaces sketched from memory
3. Now ask: “If this happened again tomorrow, how do we wish it would go?”
4. Redraw the ideal version on paper:
   • Fewer steps
   • Clearer alerts
   • Cleaner communication flow
5. Compare the real vs. ideal side by side. The gap is your roadmap.

Do this a few times, and you’ve effectively built a small analog wind tunnel lab.

Conclusion: Safer Skies Through Paper Turbulence

Resilient systems aren’t just about better code or more metrics. They’re about teams that:

• Anticipate failures
• Practice response
• Continuously refine how they work under pressure

An analog incident story wind tunnel is a low-cost, high‑leverage way to do exactly that—before you run chaos experiments in production, before customers are angry, before your team is sleep‑deprived at 3 a.m.

By embracing hand‑drawn, low‑fidelity simulations, you:

• De‑risk new incident processes and tools
• Uncover gaps in monitoring, communication, and decisions
• Move steadily toward repeatable, push‑button incident response

Turbulence will always be part of running complex systems. The question is whether you and your users first encounter it in the wild—or safely, in your own paper wind tunnel.

If your team runs on-call, incidents, or reliability, grab some markers and start building that wind tunnel today—on paper, where it’s safe to crash as many times as you need.