Skip to content

Chapter 28: Portfolio Observability and Operations

One Surface, Many Services

One system's evidence was chapter 17; ten systems on one surface is this chapter, with freshness as a first-class reading.


1. One Surface, Never a Parallel Truth

The three structures produce services; at a portfolio, the operator's question changes from is this service working? to which service is not, and what does that mean for the whole? That question needs one surface — and the surface must not invent its own truth.

One surface, consuming each service's own signals. The moment the portfolio collects separately, it has created a second measurement of the same thing, and the two will disagree during exactly the incident when the disagreement matters most.

🎯 What You'll Learn

  • Why the surface consumes each service's own signals rather than collecting its own
  • What "degraded" must look like, before an incident asks
  • The post-mortem that promotes a prevention into an artefact

The Surface

Column What it answers
service which one
signal what it says, and from where
freshness how old the reading is
owner who is accountable
escalation who is called, and when

Freshness is first-class. A stale reading presented as current is the most expensive lie a status surface can tell, because it is believed by the person making a decision.

✅ What to Check

  • Does any portfolio surface compute a value its services also compute? That is the parallel truth.
  • Does every row carry a freshness time?

2. Budgets, Degradation and Two Kinds of Health

Error budgets per service. A budget says how much unreliability a service may spend — and spending it is a decision, not a surprise. The budget is what makes "where do we spend reliability effort?" answerable.

"Degraded" must be defined before the incident. If the only states are up and down, every partial failure is an argument. Define what degraded looks like per service: which functions work, which do not, and what a caller sees.

Health Signals The decision it drives
system availability · latency · saturation reliability work, capacity
feature gate escapes · rework · freshness design, review, the next iteration
portfolio which services are degraded, and the dependency order where the operator looks first

Portfolio health is not the sum of feature health. A portfolio can be healthy while a feature is dead — and the surface must say which one is being reported.

✅ What to Check

  • Does each service have a budget, or only an alert?
  • Is "degraded" defined, or is it the word used when nobody knows?

3. Incidents Sized for One Operator

A single operator cannot run a large incident process. The process must be sized to one person and one agent:

Step The question The artefact
severity how bad, by whom judged a severity table, written
first response what to check first the surface's order
comms who must be told, and how a message template
escalation who takes over the escalation path

The post-mortem promotes a prevention into an artefact — a test, a check, a play, a signal, or a rule. A post-mortem whose outcome is "we will be more careful" has produced a sentence, not a control.

DR across services belongs here too: which services come back first, and what can wait. The order is a decision made before the incident, because during it there is no time to agree.

✅ What to Check

  • Does your last post-mortem name an artefact that now enforces the prevention?
  • Is the DR order written, or discovered?

4. The Copy Behind the Surface

A surface that reports ten services is still reporting ten places where data can be lost. The portfolio rule is one sentence: every system that holds first-party data has a copy and a proof — and neither half is optional.

One copy, per schema. The database is shared, so the copy is taken at the suite level and scoped per schema: one schedule, one manifest, one place where "is it fresh?" is answered. A copy per repository would fragment the record and give each service its own idea of what a backup is.

A freshness signal, on the same surface. The copy is a reading like any other: its age in hours, beside the health of the services it protects. A backup that stops is invisible in a log and obvious on a surface — and "invisible until the incident" is the whole failure mode this section exists to prevent.

A restore that replaces the project, not the schema. This is the fact that changes the decision: a restore puts an older copy in place of the live database, so every write since that copy is gone — across every schema that shares the project, not only the one that broke. A copy is therefore recovered by a person, with a written case (chapter 25's roll back · roll forward · neither), never by a repair command.

Two numbers, stated. The recovery point objective — how much data the copy may lose — and the recovery time objective — how long the restore takes — belong in the same table as the service's budget, because they are the same kind of decision. A number nobody has measured is not an objective; it is chapter 18's drill that turns it into one, and the drill is run per schema so each owner learns their own cost.

One honest limit. The copy does not replace the provider's own recovery (point-in-time recovery, where it is enabled), and the provider's recovery does not replace the copy: one lives in the same account as the data, and the other is the copy you can still read when the account is the thing that failed.

5. The AI Delta

An agent on call cannot triage what it cannot perceive: the status surface is its eyes, so a signal it cannot read is a signal that does not exist for the responder.

Freshness becomes first-class for the same reason — an agent will act on a stale reading without hesitating, where a human might notice.

💰 Cost Check: a parallel status surface costs the disagreement during the incident. An undefined "degraded" costs an argument at the worst moment. A post-mortem without an artefact costs the same incident twice. The surface, the budget and the prevention catalogue cost a day and pay on every incident after.

flowchart LR
  S1["service A signal"] --> P["one portfolio surface"]
  S2["service B signal"] --> P
  A["agent reads it"] --> P
  P --> F{"freshness"}
  F -->|"fresh"| D["a decision"]
  F -->|"stale"| X["STALE — never green"]
  P --> L["incident → prevention artefact"]

What this shows: the surface consumes, never computes; and the freshness check is what makes it safe for a machine to read.

The Delegation Line

The agent detects, drafts the incident record and assembles the timeline. The human declares severity, comms and the prevention. The system refuses a green reading from a stale signal, and refuses to close an incident without a promoted prevention.


6. Build the Delivery

Deliverables

Artefact Shape Where it lives
status surface service · signal · freshness · owner · escalation one page, generated
error budgets per service, with the spend that triggers a decision docs/budgets.yaml
incident playbook severity · first response · comms · escalation the Playbook
prevention catalogue every post-mortem's prevention, and the artefact docs/prevention.md
DR order what returns first, and what may wait the Playbook

Decisions

Choice Default What changes if you choose otherwise
the surface one, reading each service's own signals a parallel truth discovered under pressure
budgets or alerts first budgets an alert with no threshold to argue about
"degraded" defined per service, before the incident an argument during it
the prevention an artefact a sentence everyone forgets

✅ Gates

  • A signal older than its freshness window reads STALE, never green.
  • An incident without a promoted prevention is open.
  • A surface that computes a value its service owns is a finding.

Runbook

  1. Build the surface from existing signals. Checkpoint: every row has a freshness time.
  2. Set budgets. Checkpoint: spending one triggers a named decision.
  3. Write the playbook and define "degraded". Checkpoint: a caller sees what degraded means.
  4. Run one incident end to end. Checkpoint: severity, comms and escalation are all exercised.
  5. Land its prevention as an artefact. Checkpoint: the prevention fails on the original defect.

🎯 Proof

  • A stale signal that says so.
  • An incident whose prevention fails on the original defect.
  • A surface that regenerates without a meeting.

7. 🔑 Chapter Summary

One surface, consuming what the services already say, with freshness as a first-class reading.

  1. The surface consumes, never computes — a parallel truth disagrees exactly when it matters.
  2. Budgets make reliability a decision; "degraded" must be defined before the incident.
  3. System, feature and portfolio health are different — say which one is being reported.
  4. An incident sized for one operator, and a post-mortem that promotes a prevention into an artefact.
  5. Every system that holds first-party data has a copy and a proof — one suite copy, a per-schema restore, a freshness signal on the surface, and two numbers nobody has to guess.
  6. The agent reads the surface; freshness is what makes that safe.