ONOn-Call NocheSimulated environment

Case 001 · Priority One

The Checkout
That Never Returned

A playable SRE detective experience. Follow the telemetry, separate coincidence from cause, and restore checkout before the trail goes cold.

Open the case file

Live simulation

Evidence, not assumptions.

Incident active
01Brief02Investigate03Diagnose04Remediate05Verify06Debrief
CONFIDENTIAL · SIMULATION

Opening briefing · 22:14 UTC

Checkout is failing.
The pods look green.

At 22:04, traffic surged. Since then, checkout success has fallen from 99.6% to 57.8%, and P95 latency climbed from 420 ms to 8.4 seconds. Inventory appears healthy. Database connection errors are present in checkout logs.

ObjectiveFind the root cause
ConstraintMinimize customer risk
Potential suspectsDeployment · Inventory · Database
All telemetry is deterministic and simulated. No live production system is connected.

Behind the build

A game on the surface.
A repeatable demo motion underneath.

This prototype turns incident response into a guided customer-evaluation asset. The browser runs deterministic fixtures; a production implementation would replace them with live, permissioned telemetry.

Conceptual architecture
Browser experienceCase UI · Scorecard
Scenario engineTyped fixture · Rules
OpenTelemetryMetrics · Logs · Traces
DynatraceDavis AI · Grail
Kubernetes cluster
StorefrontWeb
CheckoutAPI
PostgreSQLData
InventoryAPI
Terraformprovisions repeatable cluster, observability, identity, and environment configuration

Live implementation

How the pieces fit

  • OpenTelemetrycollects vendor-neutral signals and preserves trace context.
  • Dynatracecorrelates service health, dependencies, SLOs, and business impact.
  • Kuberneteshosts services; health and scaling signals inform recovery.
  • Terraformmakes demo and pilot environments repeatable and reviewable.

Prototype boundaries

What is simulated

  • In this sitedeterministic metrics, traces, logs, events, topology, scoring, and assistant guidance.
  • In productionauthenticated APIs, real OpenTelemetry pipelines, Dynatrace entities, Kubernetes state, and approval workflows.
  • Safe by designno credentials, customer data, cluster access, or autonomous production changes.

Recruiter-friendly skills map

Incident investigationSRE workflows
Telemetry viewsDynatrace & observability
Deployment manifestsKubernetes
Repeatable provisioningTerraform / IaC
Demo guide + templateTechnical enablement
Reusable scenarios + scorecardScalable sales motion
Evidence assistantAgentic systems
CASE / 002

Extend the system

Add the next incident as data, not a redesign.

Create a typed scenario fixture with timeline, evidence, suspects, remediation options, and scoring keys. The investigation shell remains reusable.

app/scenario.ts → Scenario

Solutions Engineer mode

From technical proof
to customer value.

10
minute path
01

SRE leader

DiscoverHow do you prioritize signal quality and reduce mean time to recovery?

DemonstrateShow causal correlation, SLO impact, and a repeatable investigation path.

02

Platform engineer

DiscoverWhere do teams lose context across metrics, logs, traces, and cluster state?

DemonstrateShow OpenTelemetry portability, service topology, and Kubernetes evidence.

03

VP of Engineering

DiscoverWhat is the business cost of slow, inconsistent incident response?

DemonstrateConnect checkout recovery to revenue protection and operational learning.

Suggested path

  1. Frame business impact57.8% checkout success and the “green pods” trap.
  2. Follow the evidenceCorrelate pool saturation, trace wait time, and logs.
  3. Choose recoveryDiscuss blast radius, database headroom, and approval.
  4. Prove valueVerify recovery, review score, and export the report.

Pilot success criteria

  • Reduce time to isolate a cross-signal incident
  • Trace one critical journey end to end
  • Define actionable SLOs and ownership
  • Deploy repeatable telemetry through IaC
  • Produce an agreed runbook and scorecard

Objections & responses

“We already have dashboards.”

Dashboards show data. This motion tests whether teams can connect impact, cause, action, and recovery across signals.

“We can’t expose production data.”

Start with deterministic fixtures, then use scoped read-only access and sanitized pilot services.

“Another agent feels risky.”

The assistant is evidence-grounded, cannot reveal unseen data, and leaves remediation approval with the operator.