Introduction

Most teams practice incident response in one of two ways: not at all, or with a carefully scripted tabletop exercise that feels nothing like a real 03:00 a.m. page.

But what if you could feel the chaos instead—hear it clack, bounce, and collide in front of you? What if on-call triage practice was as visceral and unpredictable as a ball careening around a pinball table?

Enter the Paper Incident Story Pinball Table: a low-tech, tactile “chaos machine” designed to simulate production failures in a safe, hands-on way. It’s part chaos engineering, part tabletop exercise, part game—built to help SREs and product teams get better at on-call triage, together.

This post walks through the idea, how to design one, and how to use it to build real resiliency and shared ownership of reliability.

Why a Tactile Chaos Machine?

Digital systems fail in messy, nonlinear ways. Logs explode, dashboards light up, alerts cascade, dependencies wobble. But most training is linear: slide decks, diagrams, predictable scenarios.

A physical chaos machine—like a pinball table—introduces unstructured, embodied randomness:

• You can see incidents appear as balls, cards, or tokens.
• You can hear the collisions as they hit different components.
• You can feel the time pressure as multiple "incidents" are in play.

This mimics real-world chaos the way tools like Android’s Monkey generate random UI events, or chaos engineering tools randomly kill processes. The difference is that the chaos is visible and social. Everyone around the table shares the same view of reality, which is exactly what good incident command tries to achieve.

The physicality forces teams to deal with competing priorities, imperfect information, and collisions of events—just like real production.

The Pinball Metaphor: Collisions, Not Just Bugs

Most new on-call engineers imagine incident response as “find the bug, fix the bug.” But real incidents are usually about managing collisions:

• Multiple alerts for one root cause
• Dependent services failing in sequence
• Conflicting priorities (availability vs. deploy velocity vs. cost)
• Different teams and tools all coming into play at once

A pinball table makes this concrete:

• Balls represent events: alerts, failures, user reports.
• Targets represent systems, services, or components (API, DB, cache, payment gateway, etc.).
• Bumpers represent monitoring, redundancy, or auto-healing mechanisms.
• Lanes or paths represent incident workflows or escalation routes.

Your job during the exercise is not just to "catch the ball" but to manage and route the collisions: decide what gets priority, where to focus, what to ignore, and how to keep the system from spiraling.

Designing the Paper Incident Story Pinball Table

You don’t need real electronics or a physical pinball cabinet. You can build a “paper pinball table” on a large whiteboard or poster board.

Core Components

1. System Map as Playfield
   Draw your architecture like a pinball playfield:

   • Boxes for services (web, API, DB, queue, cache, third-party integrations)
   • Arrows for dependencies
   • Special zones for things like “CI/CD”, “Feature Flags”, “Traffic Router”

2. Incident Tokens (The Balls)
   Represent incidents with physical tokens:

   • Colored magnets or sticky notes
   • Poker chips, marbles, or printed “incident cards”
   • Each token has a type (latency, error spike, data inconsistency, 3rd-party outage, etc.)

3. Chaos Triggers
   Ways to introduce randomness:

   • A die roll or card draw every 2–3 minutes to spawn a new incident
   • A chaos deck: small cards labeled “Kill API pod”, “Slow DB by 10x”, “Drop 20% of external calls”, etc.
   • A simple timeline (“at T+5, something fails in storage”) where the exact manifestation is random

4. Controls and Flippers
   Instead of mechanical flippers, you have actions:

   • Scale a service up/down
   • Roll back a deployment
   • Toggle a feature flag
   • Reroute traffic
   • Throttle non-critical functionality These should be written as explicit cards or options the team can choose.

5. Scoring and Outcomes
   Score based on:

   • User impact avoided or minimized
   • Recovery time (time to restore steady state)
   • Clarity of communication and documentation

This gives you a lightweight but expressive system where a handful of random events can turn into a realistic, multi-incident story.

Borrowing from Chaos Engineering and Kubernetes

To make scenarios feel like real production, take cues from Kubernetes-style chaos experiments:

• Random pod termination → Incident token appears at a service with reduced capacity.
• Network partition → Certain edges in your diagram “go dark” or become flaky.
• Resource starvation → Throttle a component (e.g., DB max connections halved).
• Bad rollout → Mark a service box as “degraded” until a rollback card is played.

You can implement this by:

• Having chaos cards that say things like:
  • “Kill 2 API replicas; error rate spikes by 20%.”
  • “Network between API and cache is intermittently failing.”
  • “New deployment silently breaks authentication for 5% of users.”
• Combining chaos cards with monitoring clues:
  • When a chaos card is drawn, hand players related “alert” stickies: API_5XX_HIGH, LATENCY_DB, ALERT_3P_PROVIDER_ERRORS, etc.

The goal isn’t to be perfectly accurate to your infrastructure. It’s to capture the shape of failures and the kinds of tradeoffs and hypotheses SREs and product teams actually face.

Practicing On-Call Triage with Physical Randomness

With the pinball table set up, you can now run on-call drills. A simple format:

1. Define Roles

   • Incident commander
   • Communications lead (status updates)
   • Primary responder (on-call engineer)
   • Observers/note-takers

2. Start the Chaos Machine

   • At time zero, introduce 1–2 incident tokens on the board.
   • Every couple of minutes, roll a die or draw a chaos card to spawn new events.

3. Simulate Observability

   • For every incident, provide:
     • 1–3 alerts
     • A “dashboard snapshot” (pre-printed charts) or verbal metrics
   • Let participants ask for more data at the cost of time.

4. Make Decisions Under Pressure Participants must:

   • Decide which incident to prioritize.
   • Choose actions (scale, rollback, toggle flags, rate limit, etc.).
   • Explain their reasoning out loud.

5. Advance Time and System State

   • After each decision, you update the board:
     • Some incidents resolve.
     • Some get worse.
     • New ones may appear.

The physical randomness ensures that no two runs are the same. Practicing triage becomes about pattern recognition and decision quality, not memorizing a scripted scenario.

Using Recovery Time as a Core Metric

A key advantage of this approach is that you can bake recovery time directly into the exercise.

For each incident:

• Note the start time (when the first alert or token hits the board).
• Note the recovery time (when the system is back to a defined “steady state”).

You can define steady state in simple terms:

• No user-facing errors above baseline
• Latency back within SLO
• No critical alerts firing

Then, after the exercise:

• Compare recovery times across scenarios.
• Ask: What decisions shortened or lengthened recovery?
• Identify single points of failure in knowledge (e.g., “only Alex knew how to rollback this service”).

Over time, you can track team performance:

• Mean time to acknowledge (MTTA) in the exercise
• Mean time to recovery (MTTR)
• Number of incidents where communication broke down

These are the same metrics you care about in production—but now you can practice moving them in a low-stakes environment.

Simple Case Management: Paper Tickets for Paper Incidents

To reinforce good habits, pair the pinball exercise with a lightweight case-management system:

• Every incident token on the board maps to a paper incident ticket.
• Each ticket captures:
  • Time detected
  • Symptoms (alerts, user reports)
  • Suspected cause(s)
  • Actions taken (with timestamps)
  • Resolution / mitigation
  • Follow-up items or learnings

During the exercise, someone (often an observer) plays the role of incident scribe.

Benefits:

• Teams practice writing concise, useful timelines.
• You create material for post-incident reviews immediately.
• You normalize the idea that documentation is part of incident response, not optional homework.

After the drill, run a short retro:

• What went well?
• Where did we lose time or get confused?
• Which playbooks or runbooks need to exist but don’t?
• Which alerts were noisy or unhelpful?

The paper trail from the exercise becomes a training asset and a design input for improving your real systems.

Embedding SREs with Product Teams for Shared Ownership

The pinball table works best when SREs don’t run it for product teams, but with them.

• SREs bring knowledge of reliability, observability, and chaos patterns.
• Product engineers bring knowledge of business flows, edge cases, and user impact.

Run these drills while SREs are embedded with product teams:

• Co-design the system map so both groups agree on how things really work.
• Co-create chaos cards that reflect real past incidents.
• Rotate roles so product engineers sometimes act as incident commander.

This builds shared ownership of reliability:

• Reliability is seen as a property of the product, not just the platform.
• Product teams learn how their features behave under stress.
• SREs gain context on product priorities and user-critical paths.

Over time, the line between “SRE incident” and “product incident” starts to blur—which is exactly what you want.

Conclusion

The Paper Incident Story Pinball Table is intentionally simple: paper, markers, tokens, maybe a die. But within that simplicity is a powerful idea:

• Use physical chaos to mimic real operational chaos.
• Practice on-call triage as a team sport, not a solo heroics test.
• Focus on recovery time, communication, and decision-making under uncertainty.
• Bring chaos engineering principles into an accessible, low-tech format.

Most importantly, the pinball metaphor reminds engineers that incident response is not just about hunting a single bug. It’s about managing a fast-moving collision of events, alerts, failures, and people—and guiding the whole system back to stability.

If your current incident training feels flat or theoretical, try building your own paper chaos machine. You might be surprised how much your team can learn from a handful of index cards and a pinball mindset.