Introduction: When the Screens Go Dark

Most resilience conversations start with tools: better monitoring, richer dashboards, smarter alerts. But the moments that truly test an organization aren’t when everything is humming along — they’re when the tools themselves fail.

Think of a railroad in the early 20th century. Trains moved safely long before real‑time digital control systems existed. They relied on signals, flags, paper timetables, and simple, shared rituals that everyone understood. Those systems weren’t perfect, but they were robust in ways our hyper‑connected tools sometimes are not.

This is the idea behind the Paper Rail Signal Lab: design low‑tech, human‑driven early warning rituals that work even when your most sophisticated high‑tech systems are down.

In this post, we’ll explore how to:

• Create paper-based checklists, forms, and signals that still work in a blackout
• Define “normal” in plain human terms (no dashboards required)
• Borrow SRE’s golden signals for human monitoring
• Script clear decision paths from observation to action
• Run low‑tech tabletop drills
• Design explicitly for degraded modes
• Continuously refine your rituals from real incidents

Why You Need Low‑Tech Rituals

Modern operations are deeply entangled with their tools. If monitoring, chat, ticketing, or SCADA go down, you don’t just lose visibility — you lose coordination.

Low‑tech outage rituals are your backup nervous system. They:

• Work when power, network, or VPN are unreliable
• Reduce panic by providing a familiar script
• Help new responders plug in quickly
• Create shared situational awareness without fancy dashboards

The goal is not to abandon high tech, but to design a graceful fallback for when it’s unavailable or lying to you.

1. Designing Simple, Low‑Tech Rituals

Start by assuming the worst:

• Monitoring tools: unreliable or offline
• Central chat: split or unavailable
• Ticketing: inaccessible
• Documentation: trapped behind SSO

Now ask: In that world, what do we still need people to do in the first 30–90 minutes?

Typical answers:

• Notice something is wrong
• Share that information
• Triage what’s impacted
• Decide who acts, and how
• Capture key decisions for later review

Design paper-based artifacts to support each step:

• Outage trigger card – A one-page sheet: “If you see X, do Y” with contact numbers and escalation steps.
• Manual incident log – A printed form to record times, observations, actions, and decisions.
• Impact checklist – A short list of critical services and customers, with checkboxes: “Impacted? Yes/No/Unsure.”
• Role reminder cards – Simple role descriptions (Incident Lead, Comms, Scribe, Tech Lead) with bullet lists of responsibilities.

Keep each artifact:

• Short (preferably 1 page)
• Legible (large fonts, clear headings)
• Self-contained (no need to look things up elsewhere)

Print them. Put them where people actually work: next to phones, in on‑call rooms, near physical consoles, at reception.

2. Define Normal: Human‑Readable Baselines

You can’t recognize abnormal if you’ve never defined normal.

Create normal-behavior baselines for your most critical components in plain language:

• “During business hours, we usually have ~50–80 logins per minute; more than 200 for 5 minutes is unusual.”
• “Night batch jobs finish by 03:00; if they’re still running at 04:00, treat as degraded.”
• “Warehouse pick queue rarely exceeds 120 orders; above 300 for 15+ minutes requires investigation.”

Document these baselines on a Normal State Reference Sheet for each key system:

• Typical volumes (requests, orders, jobs)
• Normal response/processing times
• Usual error types and their expected frequency
• Known seasonal/daily spikes

This sheet should be understandable by someone who cannot see a dashboard. Use ranges and qualitative language, not just numbers:

“If calls are taking more than twice as long and customers are frequently mentioning timeouts, conditions are not normal.”

These baselines become the foundation for your low‑tech early warning signals.

3. Human Golden Signals: What to Watch Manually

SRE teams often talk about golden signals: latency, traffic, errors, saturation. You can adapt this idea for humans during outages.

Define a small set of indicators that people can observe or count manually. For example:

• Response time (human version):
  • How long customers wait on the phone
  • How long a page or transaction seems to take from a user’s perspective
• Error patterns:
  • Number of error reports per 10 calls
  • Recurring phrases in support tickets (“stuck,” “spinning,” “timeout”)
• Queue length:
  • Orders waiting in backlog
  • Open support cases in a visible board
  • Trucks waiting at a dock

Create a Paper Golden Signals Card:

• List 3–5 indicators per system
• Explain how to measure them by hand (e.g., “Count how many error calls you receive in 10 minutes”)
• Define simple thresholds: Green / Yellow / Red

Example:

Login Service – Manual Golden Signals
• Error calls > 5 in 10 minutes → YELLOW
• Error calls > 15 in 10 minutes → RED
• More than 30-second delay for 3+ users in a row → YELLOW
• Widespread inability to log in → RED

These are your paper rail signals: simple, visible states everyone can understand.

4. Script the Path from Observation to Action

Rituals fail when people see a problem but don’t know what they’re allowed or expected to do.

Use decision-analysis style thinking to write small decision trees that connect:

1. A specific signal (e.g., “Queue length is above 300 for 15 minutes”)
2. A clear condition (YELLOW vs RED)
3. A defined action (who does what, when)

A simple format:

If [signal] is [YELLOW/RED] for [duration]
Then [role] does [action]
And [who else] is informed by [channel]

Example:

If login errors are RED for 10+ minutes
Then First Responder calls Incident Lead by phone
And Incident Lead starts manual incident log and activates call bridge

Print these as Decision Cards and keep them with your golden signals cards. The goal is not to anticipate every scenario, but to make the first few moves obvious and safe.

5. Run Low‑Tech, Tabletop-Style Drills

Rituals become real only when people practice them.

Run regular tabletop exercises where you deliberately:

• Disallow use of normal tools: “Monitoring is down; Slack is down; ticketing is slow.”
• Hand out printed golden signals, decision cards, and logs.
• Present a scenario: “Customers are reporting failures to place orders.”
• Walk through the first 60–90 minutes entirely on paper and voice.

During the drill, observe:

• Where people hesitate (“Who should I call?” “Do I log this?”)
• Which cards people actually reach for
• Where the ritual feels too slow, too complex, or unclear

Afterwards, hold a brief hot wash:

• What helped? What got ignored? What was missing?
• Did roles feel clear?
• Did we get from signal → decision → action quickly enough?

Use these insights to refine the artifacts and the ritual itself.

6. Design Explicitly for Degraded Modes

Degraded mode is not an afterthought; it’s a design target.

Make conscious choices about how information will flow when:

• SCADA or monitoring is intermittent or frozen
• Chat is unavailable or fragmented across tools
• Ticketing or incident platforms are unreachable

Concrete tactics:

• Whiteboards as central status displays: one per location, with a simple column layout (Time / Observation / Action / Owner).
• Phone trees: printed call lists with primary and backup numbers, plus rules for when to escalate.
• Printed playbooks: slim binders or folders with the most important cards, contact lists, and procedures.
• Physical tokens: something as simple as a colored magnet or paper card that signals who currently has Incident Lead or Comms roles.

The goal is to ensure that information, authority, and responsibility still flow even when your usual channels don’t.

7. Continuous Refinement from Real Incidents

Your first version of these rituals will be wrong in important ways. That’s expected.

Treat your paper rituals like living code:

• After every real incident or drill, update the cards while memories are fresh.
• Remove steps that never get used and add the shortcuts people naturally invented.
• Replace vague thresholds with better ones based on observed data.
• Adjust roles to match how people actually collaborate under stress.

Keep version numbers and dates on your printed materials. When you update, reprint and redistribute. Old versions should be visibly retired to avoid confusion.

Over time, you’ll develop an ecosystem of simple, durable practices that feel natural to your teams and hold up under pressure.

Conclusion: Building Your Own Paper Rail Signal Lab

High‑tech systems are wonderful — until they’re not. When visibility drops and coordination tools vanish, what’s left is people, paper, and shared understanding.

By:

• Designing low‑tech rituals and artifacts
• Defining human-readable normal baselines
• Adapting golden signals for manual observation
• Scripting clear paths from signal to action
• Practicing via low‑tech tabletop drills
• Designing explicitly for degraded modes
• Continuously refining based on reality

…you create a Paper Rail Signal Lab inside your organization: a place where resilience is designed, tested, and improved, independent of any one tool.

When the screens go dark, you won’t be guessing what to do. You’ll be following a well-practiced ritual — one that keeps the trains moving, safely, until the lights come back on.